45 Gemfury incidents · Cʼhwevrer 2020 — official updates, affected components, duration and resolution details.
We are currently investigating this issue.
Bots/crawlers are overwhelming the service. Adjusting throttling.
Throttling adjustments are in place. Errors subsiding. Monitoring.
Errors have returned to normal levels. Fix confirmed. Resolving.
We're seeing unusual traffic to our repositories that are causing elevated errors and timeouts. We've investigating possible ways to mitigate this.
The number of open connections to each server was reaching our configured limit. We have raised the limit. Monitoring.
Configuration & scaling improvements seem to be absorbing the traffic spikes more comfortably now. Resolving.
We are again seeing errors & delays for uploads. Investigating.
Looks like some of the misconfiguration issues persisted from yesterday. A few of the queues were not processed, which prevented account updates from propagating to repositories. We have made a fix, and are now processing the backlog.
Jobs processing normally, and outstanding work has been completed. Resolving.
We are investigating delays in processing of uploads and other asynchronous work.
Production-specific misconfiguration caused our async workers to enter a crash-restart loop. We've fixed the issue, and we will be looking into how to catch these issues earlier in the release process. Resolving.
We are currently investigating this issue.
We had a temporary 10x traffic increase on one of our backend services, which lead to elevated timeouts and errors. Error rates have now returned to normal while we continue to look for the root cause of that traffic spike.
Resolving the incident to continue the investigation and fix offline.
Investigating elevated error rates
New code with an inefficient query in a critical path deployed. We have rolled it back - errors returning to normal.
Operations returned to normal. We will address the performance issue through our development cycle.
We are investigating elevated error and timeout rates.
Unusual traffic spike seems to have temporarily overwhelmed one of our API endpoints. That spike has subsided, error/timeout rates are back to normal. We are monitoring while we continue to investigate.
We've tracked the issue down to a frequent DB update and applied a fix to reduce it. Resolving the incident while we will keep monitoring the fix.
We are reopening the incident due new availability issues related to AWS outage.
We are continuing to monitor the upstream issue.
Upstream issue resolved. If you’re still having availability issues, don’t hesitate to contact us directly.
**Summary** First, we want to thank everyone for your patience as we worked through this incident. After restoring platform availability, we noticed that some services \(e.g. uploads\) did not immediately recover. This postmortem explains what happened, what we found, and what we’re doing to prevent similar issues in the future. **What Happened** The platform experienced a temporary availability outage that affected several services. While core systems came back online as expected, a subset of services continued to experience availability issues after the upstream outage. **Root Cause** The affected services were still using stale DNS cache entries from during the outage, preventing them from reaching other internal components. While DNS caching is technically correct, our monitoring should have detected the failures, and prompted corrective action. However, our existing checks were only performing shallow availability tests and did not verify internal reachability, leading to a false positive. **Resolution** Once identified, clearing the DNS caches restored full connectivity and service availability. To prevent recurrence, we’ve extended our monitoring to include internal reachability probes, ensuring that similar issues are surfaced automatically in the future. **Closing** We understand how disruptive downtime can be, and we’re committed to learning from each incident to make our platform more resilient. Thank you again for your patience and continued trust.
As many others, we have been affected by the AWS outage. Particularly, our Git/Build service could not accept pushes and trigger new builds. Currently, upstream errors and our errors have subsided, we are monitoring toward a resolution. Note: We were unable to update this status page because this service was also affected.
Upstream issue has been resolved. We are back to normal. Resolving.
We are seeing elevated error rates. Investigating.
We've found that the issue was triggered by a recent configuration update. We've rolled back the changes, and the error rates have returned to normal levels. Checking further into the changes that caused the failure.
It appears a misplaced space in a configuration update of caused the configured library to throw exceptions for some of the requests. We've fixed the configuration and will be patching the library to ensure this is handled more gracefully. Resolving.
We are seeing increased timeouts for proxied requests to the public npm registry. We are tracking the upstream issue with npmjs.org.
Upstream incident has been resolved, and no problems otherwise. Resolving.
We are currently investigating this issue.
Our asynchronous workers are not running due to platform issues, and thus cannot process uploads. We are waiting for updates from our platform provider.
We were able to start some of the workers. Starting to work through the async task backlog.
Although some platform instability persists, we have processed most of the asynchronous task backlog (uploads, emails, etc.). Most functionality has been restored. We will continue monitoring the status of the platform until resolution. If you're still experiencing delays or issues with your account, please contact [email protected] with full details.
We are continuing to monitor for any further issues.
We are resolving this as we are no longer seeing customer-facing errors. We'll continue to monitor both our and the platform's status to address any issues that may arise.
We've deployed an update that caused elevated errors due to a caching error that didn't surface during testing. We've rolled back the update, implemented a fix, and rolled out the fixed update. Monitoring.
The fix was effective. Service fully restored.
Investigating custom domains and uploads outage.
A routine infrastructure update misconfigured our load balancers. We've updated the load balancers and updated DNS settings. It may take time for changes to propagate to clients. Monitoring.
Request and error rates returning to normal levels. We will continue to monitor. Resolving.
Investigating build failures
Shortage of resources prevented build jobs from being scheduled. Scaling up nodes seems to have fixed the issue. Monitoring.
Back to normal. Resolving.
We are investigating an issue with failing builds via `git push`
Clock drift resulted in authentication failure between Git server and builder nodes. Correcting the time resolved the sporadic failures. We will investigate why nodes don't automatically synchronize their clocks. Resolving the immediate issue.
Investigating "git push" build issues
Payment issue with our cluster provider caused temporary API suspension. We've fixed the payment, and waiting for restoration of cluster access.
Cluster access restored. Git repo building issues have been resolved.
We are investigating elevated error rates
A cache instance has failed triggering automatic failover. Error rates returning to normal.
Everything returned to operating normally. Resolving.
Investigating latency & error spike
We've rolled back most recent deployment. Still debugging.
We've found a caching bug that was introduced by the latest deployment. It will not be present in future builds. Resolving.
Earlier today, we deployed a bug that exposed a legacy system to excessive traffic, overwhelming our core Redis instance and causing cascading failures in certain functionality. In the past, we occasionally used a legacy internal system to capture debug information for rare data states, aiding in tracking and reproducing customer issues. This information was stored in our core non-volatile Redis instance. While this approach had worked for rare conditions and low-traffic code paths, this incident occurred due to mistakenly adding such tracking to heavily-trafficked functionality. May 29th, 12:43 UTC - Deployment of a release with the tracking bug The release containing the tracking bug passed preflight and was deployed to production. Initially, everything appeared stable. However, the utilization of our non-volatile Redis, which usually hovers below 5%, slowly started to increase. Unfortunately, this increase went unnoticed. May 29th, 15:55 UTC - Redis storage reached maximum utilization When the non-volatile Redis storage reached 100% utilization, write operations began receiving "OOM command not allowed" error responses, resulting in 500 errors for certain user-facing APIs. Most read operations were successful, but not all. Regrettably, the tracking code was present in the API layer servicing the Dashboard and the CLI, causing errors for those read operations. Worse, the error rate remained low enough to not trigger any alarms. May 29th, 23:04 UTC - Redis instance cleaned to restore service Customer service noticed an increase in error reports and promptly notified engineering to investigate. Engineering quickly identified the issue and cleared excess data from Redis, restoring service. May 29th, 23:40 UTC - Fix deployed to remove the tracking bug We deployed a fix to remove the tracking bug. Further steps Later in the day, we removed the legacy tracking system and migrated that functionality to use standard error and metrics tracking. This step will prevent similar space issues in the future and consolidate our monitoring infrastructure. Moving forward, we will implement more alarms for storage utilization and introduce more fine-grained tracking of error rates.
