53 Python Package Index incidents · marts 2022 — official updates, affected components, duration and resolution details.
infrastrukturvedligeholdelse har en større indvirkning på tilgængeligheden af PyPI 's backends end forventet. Vi arbejder på at stabilisere.
vedligeholdelsen er færdig og alt er stabiliseret
Automatisk oversat fra den officielle hændelsesopdatering.
Flere python.org-tjenester blev påvirket af udløb af interne TLS-certifikater forårsaget af brug af Api 'er forældet af en nylig biblioteksopdatering. Vi har løst problemet og overvåger, efterhånden som tjenesterne kommer tilbage online efter at have fornyet deres interne certifikater.
Alle tjenester er genoprettet.
Automatisk oversat fra den officielle hændelsesopdatering.
Vi undersøger i øjeblikket dette spørgsmål.
Vi fortsætter med at undersøge dette spørgsmål.
Vi har løst problemet og overvåger ændringer.
Denne hændelse er blevet løst.
Automatisk oversat fra den officielle hændelsesopdatering.
Vi undersøger i øjeblikket dette spørgsmål.
Trafikken er faldet, og klyngen er stabil. En performance fix er blevet udarbejdet.
Et fix er blevet implementeret og hændelsen er lukket.
Automatisk oversat fra den officielle hændelsesopdatering.
We are currently investigating this issue
The root cause has been identified as an erroneous scaling operation that affected some application workloads. All services have been restored and are coming back online. We are monitoring to confirm full recovery.
The incident has been resolved.
The incident has been resolved. ## Incident Report: Cluster-wide service outage **Duration**: ~36 minutes \(20:04–20:40 UTC\) **Impact**: Some PSF-hosted services were unavailable, including [python.org](http://python.org), [us.pycon.org](http://us.pycon.org), PyPI stats, [bugs.python.org](http://bugs.python.org), and related services. What was unaffected was our other cluster that manages [PyPI.org](http://PyPI.org) among other services related to PyPI. **Root Cause**: During local development of kubernetes workloads locally there was an incorrect context switch to one of our production clusters. The scale-down commands ran against the production cluster instead of the local environment, iterating through all deployments and setting them to zero replicas. which created cascading failures. **Recovery**: Services were restored with the help of Ee Durbin by bringing up infrastructure in dependency order, original replica counts were recovered from Kubernetes event history. Action items: * Separate kubeconfig files for production vs local, rather than relying on context switching * Research adding admission control or policies to prevent bulk scale-to-zero operations * Document the infrastructure dependency chain and recovery runbook for future incidents Jacob Coffee, PSF Infrastructure Team
We are currently investigating this issue.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We are investigating why PyPI uploads are failing.
We identified a database lock causing contention and stopped all upload backends to release all locks and are monitoring for recovery.
This incident has been resolved.
We are currently observing a slowdown in background requests processing for events on PyPI.org This could lead to stale web pages and API responses as the events system is not able to purge the caches. Our team is continuing to investigate.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
The PyPI search index has degraded and is not returning results. A reindexing job has begun, and results will be available once it completes.
This incident has been resolved.
We are working to investigate a partial outage in the PyPI backends.
We have identified a large increase in CDN cache misses causing excess load on our backends. We are working to determine the reason for this increase.
We have identified a flood of requests from a cluster using the `python-requests/2.32.3` User-Agent, and attempting to hit all JSON project/releasese en masse. This causes excessive cache misses, which are overwhelming our backends. A temporary block has been put in place.
Our block is in place and systems are stable, but we are not satisfied with having to block such a generic user-agent. If you are aware of a system that began a mass crawl of PyPI's JSON release endpoints starting at around 22:05 UTC 2025-05-25, please contact [email protected].
We have removed the temporary block and are continuing to monitor.
We have not seen a return of the traffic that caused this incident after removing the temporary block. We will open a new incident if we need to take action again.
We're investigating elevated error rates for pypi.org
We've identified an issue with database contention arising from locking.
The locking process has been terminated and we're monitoring as things recover.
This incident is resolved. We're going to do some further investigation to understand what led to the lock contention.
We are working to resolve an outage impacting our /simple and /json apis.
We've increased capacity in the worker pools that service /simple and /json and are monitoring as things recover.
Errors have subsided, and we are monitoring as things continue to recover.
This incident has been resolved.
We are investigating issues with files from files.pythonhosted.org being served with corrupted data. A support ticket has been opened with our CDN provider and we are awaiting their response.
This issue appears to be isolated to a single edge node in our CDN provider's network. They have validated the issue and are working to resolve.
Our CDN provider has reported that the node in question has been removed from service. We are monitoring for continued reports.
We are continuing to monitor for any further issues.
This incident has been resolved.
We are currently investigating an infrastructure outage impacting python.org/downloads backends, docs.python.org backends, bugs.python.org backends, wiki.python.org backends, as well as other services.
We are continuing to investigate this issue.
Private networking in our datacenter for these services is currently intermittent, we have opened a ticket with the provider requesting assistance.
We are continuing to work on a fix for this issue.
We have begun seeing recovery across all services as of 14:44 UTC, we are monitoring for stability and ensuring all services are healthy.
This incident has been resolved.
An error serving /simple requests has caused timeouts for some requests. We have identified the issue and are working on a revert.
This incident has been resolved.
**What happened?** PyPI experienced an outage which resulted in serving 500-level responses for certain requests, including requests to a subset of the [Simple Repository API](https://peps.python.org/pep-0503/), from approximately 19:00:00 UTC to 20:15:00 UTC. As a result, some packages were uninstallable during this period. While such outages are not entirely uncommon and generally easy to recover from, this outage was more severe due to PyPI becoming un-deployable as a result of its own outage. **Impact** At its peak, PyPI served errors at approximately 25 responses per second and in total nearly 100K requests during this ~1hr 15m period resulted in an error. This represents approximately 0.04% of usual traffic to the affected services. **Background** PyPI is served by a Python web application named ‘Warehouse’, with several dependencies on Python packages that are hosted on PyPI. As part of the deployment pipeline, Warehouse and its dependencies are built into a series of container images that are ultimately deployed into a container orchestration service. At the time of the incident, PyPI contributors were working on adding new functionality, which included introducing a new service to the service layer used with PyPI’s web framework. While the actual details of this new functionality is unrelated, a misconfiguration of this new service resulted in views that depended on this service failing to successfully serve a response, which was missed by PyPI’s unit test suite and by manual testing of the feature branch. **Investigation** PyPI administrators were notified of a production error approximately 10 minutes after it was introduced, and immediately after a faulty request was served: * 2024-08-21T18:48:29 UTC - The feature branch is merged into the \`main\` branch of the Warehouse repository * 2024-08-21T18:55:42 UTC - The container image is built and deployed to production * 2024-08-21T18:58:37 UTC - PyPI admins are notified of an exception in production affecting a Simple Repository API view At this point, PyPI administrators began working on a fix. **Root Causes and Trigger** PyPI administrators immediately determined that the recent merge was at fault due to timing and clear signals from the error observed. As a result of PyPI’s container orchestration and deployment service not explicitly supporting rollbacks, issues like this are generally resolved by introducing a new pull request which reverts the problematic commit, which in turn is merged and [deployed.](http://deployed.In) In this instance, a revert commit was prepared, however the decision was made to wait for a forward fix to be prepared instead. This was partly because it was thought that this fix would be a trivial change and could be prepared as quickly as a revert, thus saving time overall. However, additionally, the introduction of the new feature included a database migration which had already been applied in production. This migration could not be reverted simply by reverting the commit, as deployment would fail when it would attempt to bring the database up to date with the migration history, and find that the current migration version was no longer present. This instead would require an additional migration to migrate the database forwards to a state that would be compatible with the reverted commit, which would potentially take more time than introducing a fix. The fix was prepared, merged and deployed, however it contained an additional issue that resulted in it not ultimately resolving the root cause. An additional pull request to rectify the issue with the forward fix was prepared. At this time, PyPI administrators noted that PyPI’s CI/CD suite was failing to build the containers required to run the test suite on new pull requests, due to the \`/simple\` detail page for one of PyPI’s dependencies being unresolvable due to the outage. As a result, the team decided to pursue a full revert of the feature instead, hoping to leverage our container layer cache to ensure that external requests to PyPI would not be necessary to build. An additional pull request to fully revert the faulty commit was prepared. While the container image build on the new PR was successful, a CI/CD check which did not use the container image, namely a check that ensures Warehouse’s dependency lock files are up-to-date, failed due to the outage. PyPI administrators attempted disabling the required status of these checks in branch protection for the repository, however the failed dependency check caused the remaining tests to be canceled before completion, which prevented the deployment pipeline from picking up on the new commit and triggering a deployment. At this time, PyPI administrators determined that overriding the build/deploy service to manually revert the deployed container image back to a known good image, bypassing the release phase that included migrations, was required. **Mitigation** Mitigation required bypassing PyPI’s build and deploy pipeline to manually re-deploy a previously built image, without running the release phase which included migrations. This allowed PyPI to successfully respond to previously failing requests, allowing the reverted pull request to build and be deployed, fully resolving the issue in production. * 2024-08-21T20:17:58 UTC - The revert was fully deployed to production and errors subsided Additionally, after the production incident was resolved, it was determined that although the final deployment to TestPyPI had been reported as successful, it had not succeeded due to the database migration failing to get a lock, highlighting that there was additionally a logic error in our deployment reporting when deployments fail. **Lessons Learned** * Until additional protections are in place, when an outage affects \`/simple\`, reverts must be immediate or they run the risk of requiring manual intervention. * It is clear that a lack of functional testing is a gap in assurances against production outages. **Things That Went Well** * Maintaining cached container image layers proved potentially useful for recovering from issues where new container images could not be built due to the outage. **Things That Went Poorly** * PyPI used to maintain a mirror of itself on a third-party service to prevent incidents like this, however this is no longer used due to maintenance of the mirror becoming an issue due to on-disk size. * The decision to not revert and fix forward instead proved to not work out. While the additional time to do the additional work to revert could have still resulted in a stuck merge, waiting for the forward fix, which had additional issues, resulted in enough time passing for one of PyPI’s dependencies to be affected by the outage. * A popular dependency, which was also a Warehouse sub-dependency, quickly fell out of cache. PyPI’s CDN is configured to serve stale responses while it revalidates for 5 minutes, and to serve stale responses while the backends are returning errors for 1 day, so the CDN should not have served an error for any of these pages. * The lack of a mirror and a production dependency on PyPI allowed for an ‘ouroboros’ style paradoxical outage that could not be resolved by a simple revert. * A subset of CI/CD checks in the critical path to deployment reach out to PyPI directly as part of their tests, and as a result they will likely fail in conditions like this outage. * Failing CI/CD checks that were not strictly required by branch protection rules were preventing the deployment pipeline from proceeding with the deployment due to setting \`cancel-in-progress: true\` on the parent workflow, causing the workflow to fail when they failed or were canceled. * The Warehouse codebase has extensive unit testing which depends on stubbing/mocking which masked the root cause. The codebase only recently gained the ability to perform functional testing, but the problematic PR did not add additional functional tests, and the limited functional testing that did exist missed this issue. **Where We Got Lucky** * At the time of the incident, all of PyPI’s administrators were online and able to contribute towards mitigating the outage, including the sole administrator that was capable of performing the manual rollback. **Action Items** * Determine why the sub-dependency fell out of cache more quickly than expected based on the CDN configuration, and how that could have been prevented. * Add functional testing to cover the Simple Repository API, with a focus on critical code paths that can cause similar outages. * Update the deployment pipeline to correctly report on deployment status. * Introduce rollback functionality for the deployment pipeline so that a rollback does not require manual intervention or specific knowledge. * Re-introduce a mirror \(either static or a caching proxy\) for PyPI that would not be affected by production outages. * Revisit the CI/CD dependency check and determine how it be hardened against a production outage to not block a deployment.
We are currently investigating an issue with PyPI uploads causing uploads to fail.
The issue has been identified and a fix is being implemented.
A fix has been implemented and we are monitoring the results.
Uploads are now being accepted properly.
We are currently investigating an issue with the PyPI backend due to a recent deploy.
This incident has been resolved.
We are performing upgrades to PyPI's database storage which may impact performance and availability of the service as they rollout.
This maintenance is complete.
We will be performing database maintenance including version upgrades and storage re-configuration. This will result in several brief outages for the service.
Underlying upgrades are complete, we are running database statistics updates to optimize performance.
We are in the final steps of the upgrade and should see performance and availability stabilize in the next 30 minutes.
All upgrade steps are complete and performance is stabilizing. A final maintenance task is taking place in the background which will minimally impact performance until it is complete.
All maintenance is complete and telemetry around database and application performance have stabilized. This incident is complete.
An outage in our infrastructure is impacting availability of us.pycon.org and python.org
A cascading failure due to an expired certificate led to this outage. The certificate has been expired and systems are coming back online.
This incident has been resolved.
