Cloudrepo Status

We have not observed a single 502 across our systems for the past two hours, while close to peak load. We will consider this issue resolved as there is no current customer impact. We will continue to monitor and evaluate internally in order to prevent any future disruption.

The incident has been confirmed as resolved.

At approximately 21:00 UTC on 5/2/22 - customers began to see intermittent 502s as well as slow down in response times. These rates increased through the night and were resolved at approximately 13:39 on 5/3/22. Root cause was determined to be an influx of requests from an outside source. We have scaled the system accordingly and will continue to monitor the status closely. Going forward we will continue to 1) investigate the root cause of the issue as well and 2) investigate as to why our alerting system failed to notify the support team. This is our first customer impacting outage in over 3 years and we'd like to sincerely apologize to any of our partners who were impacted.

During a server upgrade, there was approximately 1-2 minutes of API downtime. This was unintentional and we are looking into why this happened so we will avoid future outages in the future as we upgrade hardware.

Customer Impact: Access to our storage APIs (publishing/reading packages) was returning 500 errors for some partners. This is a repeat of the May 9th outage - please refer to the incident summary for more details. Resolution: After we were alerted of this issue we were able to restore functionality to all partners. Duration: Approximately 45 minutes at approximately 11:00 CST and 20 minutes around 18:00 CST. Future Mitigation: To prevent this from happening again, we will be implementing several changes: 1) Improve our monitoring to detect 500 errors as soon as they occur. 2) Increased the size of our cluster to give us more headroom in our connection pools. 3) Continue to investigate root cause and fix anything that may be holding on to connections.

Customer Impact: Access to our storage APIs (publishing/reading packages) was returning 500 errors for some partners. Root Cause: Our servers exhausted their connections to the storage layer and our monitoring system did not alert us to this degraded state - a partner alerted us instead. Resolution: After we were alerted of this issue we were able to restore functionality to all partners. Duration: Approximately two hours Future Mitigation: To prevent this from happening again, we will be implementing several changes: 1) Improve our monitoring to detect 500 errors as soon as they occur. 2) Increased the size of our cluster to give us more headroom in our connection pools. 3) Continue to investigate root cause and fix anything that may be holding on to connections.