50 Divio incidents · October 2020 — official updates, affected components, duration and resolution details.
We are currently experiencing an infrastructure issue on our AWS eu-central Region.
The situation seems to have normalized and we continue to monitor the situation.
This incident has been resolved.
We are currently experiencing an infrastructure issue on our AWS eu-central Region.
The situation seems to have normalized and we continue to monitor the situation.
We are continuing to monitor for any further issues.
This incident has been resolved.
We are currently experiencing intermittent disruptions related to our DNS provider (DNSimple), which is investigating infrastructure instability on their end. This may affect domain resolution for some customers. Our team is monitoring the situation and will provide updates as DNSimple works to resolve the issue.
DNSimple has restored a significant portion of their infrastructure, with nameservers recovering and DNS services coming back online. Domain resolution is stabilizing for our customers. We continue to monitor the situation and will post a final update once DNSimple confirms full resolution.
DNSimple has confirmed that all systems are operational and we can confirm from our side that domain resolution is working normally for all customers.
Cloudflare is currently experiencing global network issues which are affecting part of Divio’s services, including access to the Divio Control Panel. We are closely monitoring the issues.
Update - We have confirmed that in addition to the Divio Control Panel access, the Cloudflare issues are also intermittently affecting media and asset delivery for our clients. The core cause remains the ongoing global network instability at Cloudflare. We continue to monitor the situation closely.
Service Impact Restored: We have confirmed that the intermittent issues affecting the Divio Control Panel and Media/Asset Delivery are restored.
We're experiencing an elevated level of errors related to application scaling and deployments. The team is investigating and monitoring the situation.
We are continuing to investigate this issue.
We've identified an internal issue affecting application deployments and scaling. We recommend avoiding deployments or scaling until the issue is resolved. The team is working on a solution and will share updates as they become available.
The root cause of the issue has been identified. We are currently implementing a fix which is designed to resolve the problem across all affected regions upon deployment.
UK AWS is back to normal
CH region is back to normal
EU AWS region is back to normal
US AWS region is back to normal
The fix has been successfully deployed across all regions and we can now report that all systems are back to normal.
The deployments and service provisioning in the Swiss region (Exoscale) are currently not available due to a hardware issue. Exoscale and Divio are working together to resolve the issue. Running applications are not affected.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We're currently investigating an issue affecting the Divio Control Panel, which is temporarily unavailable. This does not impact any running applications.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We are investigating an issue with the control panel. The service is currently unavailable.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We are seeing delays in deployments. We are currently investigating the issue
The issue has been identified and a fix is being implemented
This incident has been resolved
We have to make a configuration change to the PostgreSQL 15 databases in the EU region that will result in intermittent connectivity during a 10 minutes window. We are sorry for the inconvenience.
The configuration change has completed successfully.
This incident has been resolved.
We've observed a surge in deployments on our infrastructure, leading to longer queue times for all customers. We are actively monitoring the situation and will update the status as soon as workers become available and the queue normalizes.
This incident has been resolved.
We are currently investigating an issue making deployments on the Swiss region impossible. We have found the root cause and are actively working on it. Existing applications are not impacted.
A fix has been implemented and deployments and backups are working again. We still experience an issue during creation of new buckets in the region and continue to investigate.
Some users might experience a problem to access objects in a bucket in this region. We continue to work on a full resolution of the issue.
We are currently investigating issues with our infrastructure provider Exoscale which relates to general management and bucket management.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
On the first of July 2024, a **Monday**, several clients hosting their application on the Exoscale Switzerland regions contacted support to report they had issues connecting to their S3 storage. We started investigating right away and discovered their access keys were invalid \(InvalidAccessKey exceptions\). We promptly checked all the S3 storages hosted on Exoscale, confirmed that the api keys of 50% of them have been deleted, and reached out to Exoscale. They pointed out the deleted API keys were all using a legacy mode and were scheduled for deletion after a deprecation period. Exoscale warned about the upcoming deletions via email, but due to a misconfiguration in the email notifications settings on our Exoscale organizations we never received it. Now that we identified the root cause, the second step was to provision new, non-legacy API keys for the problematic buckets. Exoscale deprecated at the same time part of their APIs used by our orchestrator, so we spent the day migrating our codebase to use the new APIs endpoints. On **Tuesday**, we were able to recreate all the api keys and update the application configurations. By redeploying the application, the S3 storage connection was restored. On **Wednesday**, after checking the daily nightly backups, we noticed some of the backups on the fixed storages were failing - some objects were not accessible to the new API keys \(head\_object: 403 Forbidden\). This was only the case on storages that were first provisioned years ago. This specific issue was related to how the storages were managed back then, when Exoscale features were less advanced. For those storages, the legacy API keys belonged to a sub-organization created specifically for each storage \(usr- \). As the new API keys belonged to the parent organization, the objects missing proper ACLs \(granting full control to the parent organization\) could neither be read nor written. We first tried to set bucket ownership controls on the problematic buckets to force all objects to belong to the parent organization, but this action disables the ACLs set at the object level, effectively making all objects private. Most applications only store publicly readable assets in their storage. We thus reverted the ownership controls and looked for a better solution. The solution was sketched out and thoroughly tested on **Thursday**, and applied to all storages on **Friday at noon**, effectively ending the outage. It consisted of the following steps, to be carried on each storage: 1. Find all the files missing the ACLs granting full control to the organization the new API key belong to, 2. Create an API key on the sub-organisation, 3. Use the API key created in 2 to patch the ACLs of the objects listed in 1. We took all the possible measures to ensure this won’t happen again and ensured the notifications were now properly configured. On the positive side, the solution to the outage allowed us to strengthen the security of the storages and improve the overall management of permissions on Exoscale buckets. We are sorry for the inconvenience and would like to thank all the Exoscale customers for their patience and their help in the handling of this incident.
We are currently experiencing issues in the AWS EU region; we are investigating.
Websites are reachable again after less than a minute. Our initial analysis did not expose any issues with the infrastructure. We are currently monitoring the situation.
This incident has been resolved.
We are currently investigating an issue with a part of our europe / germany open cloud.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We are currently investigating an issue with our control panel connectivity. Client applications are not effected.
The issue has been identified and a fix is being implemented.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We are currently investigating an issue with our control panel connectivity. Client applications are not effected.
We are continuing to investigate this issue.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We are currently investigating an issue with the ingress tier on our German region, leading to multiple websites experiencing downtime.
We identified the issue being a DDoS attack against one of the hosted properties. We have implemented a mitigation action while we continue the investigation.
We diverted traffic for the affected properties. All other properties should be operating nominally. We continue to monitor the situation.
This incident has been resolved.
We got reports that users from Switzerland experience slowness on our control panel at control.divio.com The issue is presumably related to an issue at Cloudflare and we continue to investigate the issue.
This incident has been resolved.
The issue has been identified and a fix is being implemented.
This incident has been resolved.
Our infrastructure provider in the CH region has an ongoing incident. https://exoscalestatus.com/incidents/79994
Exoscale has implemented a solution and we are monitoring the results.
This incident has been resolved.