52 Engine Yard incidents · maggio 2016 — official updates, affected components, duration and resolution details.
Stiamo attualmente indagando su questo problema.
Questo incidente è stato risolto.
Tradotto automaticamente dall'aggiornamento ufficiale dell'incidente.
Abbiamo identificato un problema con i certificati SSL su alcuni endpoint. Alcuni utenti possono vedere errori di certificazione. Questo è più probabile su Firefox, dispositivi Apple, proxy aziendali e rigorose integrazioni API. Altri clienti potrebbero non essere interessati. Stiamo sostituendo i certificati interessati e aggiorneremo questa pagina quando il lavoro è completo. I tuoi dati non sono interessati.
Questo incidente è stato risolto.
Tradotto automaticamente dall'aggiornamento ufficiale dell'incidente.
AWS has confirmed increased error rates and latencies affecting multiple services in the US-EAST-1 region. The incident is currently classified as "Degraded" by AWS. You may experience slowness, timeouts, or trouble accessing some parts of platform and services. We're closely monitoring the situation and will keep you updated as AWS provides more information.
AWS has confirmed that the outage in the US-EAST-1 region has now been resolved, and all services are operational.
We are currently investigating an issue causing 403 Forbidden callback errors, which are resulting in failed Chef runs and broken deployments. Our team is actively working to identify the root cause and restore full functionality.
We have successfully implemented a fix for the issue that caused errors, and our systems are now operating normally. All impacted services are fully restored, and we have verified that the issue is no longer reproducible. We will continue to closely monitor system performance and provide another update within the next 24 hours
The incident has been resolved.
We are currently investigating an issue related to the inability to login to https://login.engineyard.com/login.
Corrective actions were applied and the login to EngineYard Cloud should once again properly work.
We are currently investigating an issue with the eybackup process. Our team is actively working to identify the root cause and implement a fix.
We have identified the issue, and our team is working on a final fix.
Our team has investigated and fixed the issue. If backups are still failing, please perform a Chef run and test again—this should resolve the issue.
This incident has been resolved.
We are currently investigating the issue where some customers are unable to make deployments with the “Failed to boot in” messages.
A fix has been implemented and we are monitoring the situation.
The server provisioning failure issue has been resolved.
We have been notified that certain UI functionalities, such as accessing backups, are currently degraded. We have identified the root of this issue and are working to have it addressed. In the meantime, please reference this article to download your backups through a different method: https://support.engineyard.com/article/45200-view-and-download-database-backups
Our infrastructure team has implemented a fix. We continue to monitor the situation closely and will provide further updates as soon as possible. Thank you for your patience.
The backup-access issue using the UI has been successfully resolved
We're actively addressing a disruption affecting specific accounts, which is currently impacting deployment, change application, and SSH access. We've identified the cause and are working to implement a permanent fix.
Dear Valued Customers, We prioritize transparency and wish to keep you thoroughly informed during the ongoing service disruption. Please find the most recent update about our system's status below: - Resolution Progress: Our technical experts have pinpointed the main cause of the issue. We are now working in close collaboration with Amazon Web Services (AWS) to expedite the restoration of our services. - Anticipated Resolution Timeframe: We are hopeful that the service interruption will be brief. Our dedicated team is tirelessly working to reinstate our systems as swiftly as possible. We acknowledge the significant effect this outage has on your operations and deeply regret any inconvenience it may cause. We are grateful for your patience and understanding during this period. Be assured that we are diligently working to resume full services promptly and will continue to provide you with regular updates. Thank you for your ongoing trust in the EngineYard service.
Monitoring - We have addressed the underlying causes of the outage and are working on restoring functionality. We aim to progressively restore all service throughout the rest of the day, US hours. We will continue to provide updates as the situation develops. As always, thank you for your continued patience.
Our systems are now operational and we expect no further issues. Regardless, to any customers who believe they are still affected by this incident please reach out to Support for immediate assistance.
We have identified an issue affecting the access to EYK clusters. We have pinpointed the likely cause of the problem and are actively working towards a lasting solution. We will continue to update about our progress through this channel.
Dear Valued Customers, We are committed to keeping you fully informed during this service interruption. Here is the latest update regarding our system status: - Current System Status: The EYK backend system is currently down. EYK clusters do present availability issues due to this. For EYC customers, on certain accounts, functionalities like deploying, applying changes, and accessing the instances via SSH continue to be unavailable. Work is ongoing to identify and fix the issues. We are aware of the inconvenience this causes and are working diligently to resolve it. - Progress on Resolution: Our technical team has successfully identified the root cause of the issue. We are now collaborating closely with Amazon Web Services (AWS) to restore service as quickly as possible. - Expected Timeline: We are optimistic that the delay in service will not be prolonged. Our team is fully engaged and working around the clock to bring our systems back online. - Next Scheduled Update: We will provide our next update on January 10th at 10:00 AM EST. Please look out for our communication at that time for the latest information. We understand the critical impact this disruption has on your operations and sincerely apologize for any inconvenience caused. Your patience and understanding during this time are greatly appreciated. Rest assured, we are fully committed to restoring full service as soon as possible and will keep you updated every step of the way. Thank you for your continued trust in the EngineYard product.
We have addressed the underlying causes of the outage and are working on restoring functionality. We aim to progressively restore all service during the rest of the day, US hours. We will continue to provide updates as the situation develops. As always, thank you for your continued patience.
The issue has been resolved.
On certain accounts, functionalities like deploying, applying changes, and accessing the instances via SSH are currently unavailable. Work is ongoing to identify and fix the issues.
We have identified the probable issue and we are working toward a permanent resolution. We will keep informing our customers via this channel.
We have identified an issue affecting the access to EYK clusters. We have pinpointed the likely cause of the problem and are actively working towards a lasting solution. We will continue to update about our progress through this channel.
Dear Valued Customers, We are committed to keeping you fully informed during this service interruption. Here is the latest update regarding our system status: - Current System Status: The EYK backend system is currently down. EYK clusters do present availability issues due to this. For EYC customers, on certain accounts, functionalities like deploying, applying changes, and accessing the instances via SSH continue to be unavailable. Work is ongoing to identify and fix the issues. We are aware of the inconvenience this causes and are working diligently to resolve it. - Progress on Resolution: Our technical team has successfully identified the root cause of the issue. We are now collaborating closely with Amazon Web Services (AWS) to restore service as quickly as possible. - Expected Timeline: We are optimistic that the delay in service will not be prolonged. Our team is fully engaged and working around the clock to bring our systems back online. - Next Scheduled Update: We will provide our next update on January 10th at 10:00 AM EST. Please look out for our communication at that time for the latest information. We understand the critical impact this disruption has on your operations and sincerely apologize for any inconvenience caused. Your patience and understanding during this time are greatly appreciated. Rest assured, we are fully committed to restoring full service as soon as possible and will keep you updated every step of the way. Thank you for your continued trust in the EngineYard product.
We have addressed the underlying causes of the outage and are working on restoring functionality. We aim to progressively restore all service during the rest of the day, US hours. We will continue to provide updates as the situation develops. As always, thank you for your continued patience.
The issue has been confirmed to be resolved.
An issue has been reported preventing deployment in the latest version of the Engine Yard v7 stack. We recommend that you refrain from upgrading when possible. We are working on resolving this matter with the highest priority and will provide updates as they become available.
The latest release causing the issue is withdrawn while we are looking for a fix to the observed behavior. We are working on resolving this matter with the highest priority and will provide updates as they become available.
We are happy to confirm that the affected version has been removed. If you upgraded to it while it was available, please reach out to Customer Support so that our agents may roll back your environment to a working version. Thank you for your patience and understanding.
We Identified a problem on our backend when it comes to run Chef to configure instances. Staff on duty is currently working on this matter.
A fix has been implemented and we are monitoring the status of Chef runs across the platform.
The deployed fix addressed the issue, allowing Chef runs to happen smoothly once again. As such we're marking the incident as Resolved. Feel free to open a ticket with Engine Yard Support for any questions, comments, or concerns that may arise.
We're seeing instance failures in one AZ in the N. Virginia AWS region. Customer environments and applications may be impacted by this at the current time. Engine Yard's own cloud.engineyard.com application has been impacted and at this time we are working on restoring service to this application. AWS have yet to official report any issues, but please monitor https://status.aws.amazon.com/ for updates.
Functionality has now been restored to https://cloud.engineyard.com. AWS are seeing connectivity issues in one AZ in the US-East-1 (N. Virginia) region, which is causing instance reachability issues impacting both application functionality and instance control operations. Please open a ticket if you are having issues, and the issue can be monitored via https://status.aws.amazon.com/
AWS have stabilised the impacted AZ and are restoring impacted instances. We are seeing the majority of impacted instances fully functional again. If you see any further issues please contact Support.
AWS has resolved the incident and we are not experiencing issues anymore. Please contact Support for any queries.
AWS are reporting internet connectivity issues in both US-West regions (N. California and N. Oregon), which may be impacting the reachablility of customer's applications and environments hosted in those regions. Amazon are currently investigating and working towards resolution, and the latest updates on the status of the issue can be viewed at https://status.aws.amazon.com/.
We are continuing to work on a fix for this issue.
AWS have now resolved the internet connectivity issues in the two US-West regions and normal service has been resumed.
Amazon are reporting increased API error rates in the US-East-1 (North Virginia) region. Please hold off from making any infrastructure level changes to environments hosted in this region (instance restart, creation and termination) at this time. If you see any issues with your environments or applications please open a ticket.
Amazon have mitigated the issue causing the API errors and therefore the region is functioning as normal again and all Engine Yard infrastructure related operations should now run without issue.
We are currently investigating issues with signing into our support portal https://support.cloud.engineyard.com/, as well as the failure to submit tickets via the "File a Ticket" functionality in the Engine Yard dashboard. On-going tickets can still be updated by replying to their corresponding email threads, but should you wish to raise a new ticket please contact us via the #engineyard IRC channel on irc.libera.chat (https://web.libera.chat/?nick=EYGuest%7C?#engineyard) or call us (+1-866-518-9273).
The issue causing the sign-in issues has now been identified and resolved, so all support portal functionality is now restored.
The expiry of the Let's Encrypt DST Root CA X3 certificate is causing failures for calls to servers running Let's Encrypt SSLs from client instances running the stable-v5 stack or lower. For more information and details of how to fix, please see this guide .
A fix has now been released on v5 and v6 stacks to resolve this issue, further information can be found at the following link https://support.cloud.engineyard.com/hc/en-us/articles/4407649023515-Let-s-Encrypt-X3-root-certificate-expiration
Due to an issue with SSL validations, the EY Support Portal may not be accessible. For urgent issues please call EY Support phone line. Our staff on duty has identified the issue and is investigating a solution.
A fix on the integration between EY Cloud and Zendesk has been put in place. The EY Support Portal should be accessible again. Staff on duty continues to monitor the issue.
Integration between EYCloud and Zendesk is working properly. Monitoring, as well as customers, have not alerted of new occurrences of this case. This issue is deemed resolved.
The issue has been identified and a fix is being implemented.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
**Incident Timeline:** * 18:15 UTC: First customer reports issues with Chef _Apply_ runs on instances. * 18:50 UTC: Investigation into the issue links the issue to failures with Portage \(Gentoo package manager\), specifically the failure of customer instances to access the platform’s Portage server. * 20:15 UTC: Further testing has shown the issue to be wide-reaching and impacting all stacks \(stable v1 to v5\) aside from \(the Ubuntu based\) stable-v6. As such an Incident is created to inform customers. * 21:00 UTC: The Portage server is identified and restarted, though remains inaccessible. * 21:30 UTC: Portage server access is gained and direct investigation of connection failures from platform instances is performed. * 21:50 UTC: Flushing of IPTables firewall rules on Portage server restores connectivity of customer instances to Portage. * 22:00 UTC: Re-application of dynamically generated IPTables firewall rules is found to not impact connectivity. * 22:15 UTC: Source of issue is tracked to a change in the format of the AWS IP Range list published shortly before the latest automatic update to the dynamically generated IPTables firewall rules on the Portage server. This change was found to be reverted in later published lists. * 22:30 UTC: Incident is declared as Resolved. **Incident Root Causes:** * Engine Yard instances running stacks stable-v1 to stable-v5 run the Gentoo OS, which utilises Portage as its package manager. Engine Yard curates its own packages through a dedicated Portage server. * Upon a Chef _Apply_ run instances synchronise the local Portage Tree from the Portage server. * For security reasons the Portage servers is fire-walled from the wider internet, but needs to still allow access from AWS IP addresses. * To keep this fire-walling up to date, the Portage server runs a regular task to download the [latest published AWS IP Ranges list](https://docs.aws.amazon.com/general/latest/gr/aws-ip-ranges.html) and utilise this to dynamically generate firewall rules, granting these ranges access. * In the \(UTC\) evening of 1st April Amazon published a new IP list, which contained an additional field, not previously included in the list. * This additional field led to a failure of the list to be parsed by the firewall rule generation script, leading to the Portage server no longer granting access to the AWS IP ranges, thus blocking access to Portage from customer instances. **Incident Impact:** Customers on stacks stable-v1 to stable-v5 saw Chef _Apply_ run failures on creation of new instances or configuration of existing instances when such runs required updating of the Portage Tree or installation of packages. This failure applied to all regions. The incident did impact the deployment or running of customer applications. **Incident Corrective Actions:** We have reached out to Amazon regarding the changes to the published IP Range list, with regards to why the list was published in such a state and if such changes will be published again in future. Engine Yard will be undertaking a review of the dynamic fire-walling script, in order to prevent future parsing failures resulting in unwanted firewall lockdowns. We shall also be reviewing and improving internal documentation and knowledge sharing in order to improve investigation and resolution of any future Platform issues.
