Python Package Index 장애 기록

1. 2026년 10월 2일 오전 1:15 UTCidentified

   인프라 유지 보수는 예상보다 PyPI의 백엔드의 가용성에 큰 영향을 미칩니다. 우리는 안정적으로 일하고 있습니다.

2. 2026년 10월 2일 오전 1:32 UTCresolved

   유지 보수는 완전하고 모든 것이 안정

공식 인시던트 업데이트를 자동 번역했습니다.

1. 2026년 10월 1일 오전 11:13 UTCmonitoring

   Multiple python.org 서비스는 최근 라이브러리 업데이트에 의해 deprecated APIs의 사용으로 인한 내부 TLS 인증서의 만료에 영향을 미쳤습니다. 우리는 문제를 해결하고 서비스로 모니터링은 내부 인증서를 갱신 한 후 온라인으로 돌아갑니다.

2. 2026년 10월 1일 오전 11:18 UTCresolved

   모든 서비스는 복원됩니다.

공식 인시던트 업데이트를 자동 번역했습니다.

1. 2026년 9월 28일 오후 12:42 UTCinvestigating

   우리는 현재이 문제를 조사하고 있습니다.

2. 2026년 9월 28일 오후 1:47 UTCinvestigating

   이 문제를 조사하고 있습니다.

3. 2026년 9월 28일 오후 1:48 UTCmonitoring

   우리는 문제를 해결하고 변화를 모니터링하고 있습니다.

4. 2026년 9월 29일 오전 1:24 UTCresolved

   이 사건은 해결되었습니다.

공식 인시던트 업데이트를 자동 번역했습니다.

1. 2026년 9월 23일 오전 11:02 UTCinvestigating

   우리는 현재이 문제를 조사하고 있습니다.

2. 2026년 9월 23일 오후 2:06 UTCmonitoring

   트래픽은 양면과 클러스터가 안정적입니다. 성능 수정이 초안되었습니다.

3. 2026년 9월 23일 오후 5:05 UTCresolved

   수정이 시행되고 사건이 닫힙니다.

공식 인시던트 업데이트를 자동 번역했습니다.

1. 2026년 2월 26일 오후 8:04 UTCinvestigating

   We are currently investigating this issue

2. 2026년 2월 26일 오후 8:40 UTCinvestigating

   The root cause has been identified as an erroneous scaling operation that affected some application workloads. All services have been restored and are coming back online. We are monitoring to confirm full recovery.

3. 2026년 2월 26일 오후 11:26 UTCresolved

   The incident has been resolved.

4. 2026년 2월 26일 오후 11:26 UTCpostmortem

   The incident has been resolved. ## Incident Report: Cluster-wide service outage **Duration**: ~36 minutes \(20:04–20:40 UTC\) **Impact**: Some PSF-hosted services were unavailable, including [python.org](http://python.org), [us.pycon.org](http://us.pycon.org), PyPI stats, [bugs.python.org](http://bugs.python.org), and related services. What was unaffected was our other cluster that manages [PyPI.org](http://PyPI.org) among other services related to PyPI. **Root Cause**: During local development of kubernetes workloads locally there was an incorrect context switch to one of our production clusters. The scale-down commands ran against the production cluster instead of the local environment, iterating through all deployments and setting them to zero replicas. which created cascading failures. **Recovery**: Services were restored with the help of Ee Durbin by bringing up infrastructure in dependency order, original replica counts were recovered from Kubernetes event history. Action items: * Separate kubeconfig files for production vs local, rather than relying on context switching * Research adding admission control or policies to prevent bulk scale-to-zero operations * Document the infrastructure dependency chain and recovery runbook for future incidents Jacob Coffee, PSF Infrastructure Team

1. 2026년 2월 19일 오후 10:46 UTCinvestigating

   We are currently investigating this issue.

2. 2026년 2월 19일 오후 10:56 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

3. 2026년 2월 19일 오후 11:59 UTCresolved

   This incident has been resolved.

1. 2026년 1월 7일 오후 10:42 UTCinvestigating

   We are investigating why PyPI uploads are failing.

2. 2026년 1월 7일 오후 10:59 UTCmonitoring

   We identified a database lock causing contention and stopped all upload backends to release all locks and are monitoring for recovery.

3. 2026년 1월 7일 오후 11:00 UTCresolved

   This incident has been resolved.

1. 2025년 11월 12일 오후 3:46 UTCinvestigating

   We are currently observing a slowdown in background requests processing for events on PyPI.org This could lead to stale web pages and API responses as the events system is not able to purge the caches. Our team is continuing to investigate.

2. 2025년 11월 12일 오후 3:50 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

3. 2025년 11월 12일 오후 4:26 UTCresolved

   This incident has been resolved.

1. 2025년 7월 29일 오후 2:41 UTCmonitoring

   The PyPI search index has degraded and is not returning results. A reindexing job has begun, and results will be available once it completes.

2. 2025년 7월 29일 오후 7:58 UTCresolved

   This incident has been resolved.

1. 2025년 5월 24일 오후 10:57 UTCinvestigating

   We are working to investigate a partial outage in the PyPI backends.

2. 2025년 5월 24일 오후 11:11 UTCidentified

   We have identified a large increase in CDN cache misses causing excess load on our backends. We are working to determine the reason for this increase.

3. 2025년 5월 24일 오후 11:37 UTCidentified

   We have identified a flood of requests from a cluster using the `python-requests/2.32.3` User-Agent, and attempting to hit all JSON project/releasese en masse. This causes excessive cache misses, which are overwhelming our backends. A temporary block has been put in place.

4. 2025년 5월 24일 오후 11:54 UTCmonitoring

   Our block is in place and systems are stable, but we are not satisfied with having to block such a generic user-agent. If you are aware of a system that began a mass crawl of PyPI's JSON release endpoints starting at around 22:05 UTC 2025-05-25, please contact [email protected].

5. 2025년 5월 25일 오후 12:16 UTCmonitoring

   We have removed the temporary block and are continuing to monitor.

6. 2025년 5월 27일 오후 12:53 UTCresolved

   We have not seen a return of the traffic that caused this incident after removing the temporary block. We will open a new incident if we need to take action again.

1. 2025년 4월 24일 오후 12:02 UTCinvestigating

   We're investigating elevated error rates for pypi.org

2. 2025년 4월 24일 오후 12:09 UTCidentified

   We've identified an issue with database contention arising from locking.

3. 2025년 4월 24일 오후 12:09 UTCmonitoring

   The locking process has been terminated and we're monitoring as things recover.

4. 2025년 4월 24일 오후 12:18 UTCresolved

   This incident is resolved. We're going to do some further investigation to understand what led to the lock contention.

1. 2025년 4월 8일 오전 10:23 UTCidentified

   We are working to resolve an outage impacting our /simple and /json apis.

2. 2025년 4월 8일 오전 10:26 UTCmonitoring

   We've increased capacity in the worker pools that service /simple and /json and are monitoring as things recover.

3. 2025년 4월 8일 오전 10:31 UTCmonitoring

   Errors have subsided, and we are monitoring as things continue to recover.

4. 2025년 4월 8일 오전 11:00 UTCresolved

   This incident has been resolved.

1. 2025년 3월 11일 오후 12:13 UTCinvestigating

   We are investigating issues with files from files.pythonhosted.org being served with corrupted data. A support ticket has been opened with our CDN provider and we are awaiting their response.

2. 2025년 3월 11일 오후 12:47 UTCidentified

   This issue appears to be isolated to a single edge node in our CDN provider's network. They have validated the issue and are working to resolve.

3. 2025년 3월 11일 오후 12:50 UTCmonitoring

   Our CDN provider has reported that the node in question has been removed from service. We are monitoring for continued reports.

4. 2025년 3월 11일 오후 1:05 UTCmonitoring

   We are continuing to monitor for any further issues.

5. 2025년 3월 11일 오후 1:39 UTCresolved

   This incident has been resolved.

1. 2024년 12월 13일 오후 1:44 UTCinvestigating

   We are currently investigating an infrastructure outage impacting python.org/downloads backends, docs.python.org backends, bugs.python.org backends, wiki.python.org backends, as well as other services.

2. 2024년 12월 13일 오후 1:59 UTCinvestigating

   We are continuing to investigate this issue.

3. 2024년 12월 13일 오후 2:00 UTCidentified

   Private networking in our datacenter for these services is currently intermittent, we have opened a ticket with the provider requesting assistance.

4. 2024년 12월 13일 오후 2:45 UTCidentified

   We are continuing to work on a fix for this issue.

5. 2024년 12월 13일 오후 2:58 UTCmonitoring

   We have begun seeing recovery across all services as of 14:44 UTC, we are monitoring for stability and ensuring all services are healthy.

6. 2024년 12월 13일 오후 3:09 UTCresolved

   This incident has been resolved.

1. 2024년 8월 21일 오후 7:53 UTCidentified

   An error serving /simple requests has caused timeouts for some requests. We have identified the issue and are working on a revert.

2. 2024년 8월 21일 오후 8:23 UTCresolved

   This incident has been resolved.

3. 2024년 8월 22일 오후 8:47 UTCpostmortem

   **What happened?** PyPI experienced an outage which resulted in serving 500-level responses for certain requests, including requests to a subset of the [Simple Repository API](https://peps.python.org/pep-0503/), from approximately 19:00:00 UTC to 20:15:00 UTC. As a result, some packages were uninstallable during this period. While such outages are not entirely uncommon and generally easy to recover from, this outage was more severe due to PyPI becoming un-deployable as a result of its own outage. **Impact** At its peak, PyPI served errors at approximately 25 responses per second and in total nearly 100K requests during this ~1hr 15m period resulted in an error. This represents approximately 0.04% of usual traffic to the affected services. **Background** PyPI is served by a Python web application named ‘Warehouse’, with several dependencies on Python packages that are hosted on PyPI. As part of the deployment pipeline, Warehouse and its dependencies are built into a series of container images that are ultimately deployed into a container orchestration service. At the time of the incident, PyPI contributors were working on adding new functionality, which included introducing a new service to the service layer used with PyPI’s web framework. While the actual details of this new functionality is unrelated, a misconfiguration of this new service resulted in views that depended on this service failing to successfully serve a response, which was missed by PyPI’s unit test suite and by manual testing of the feature branch. **Investigation** PyPI administrators were notified of a production error approximately 10 minutes after it was introduced, and immediately after a faulty request was served: * 2024-08-21T18:48:29 UTC - The feature branch is merged into the \`main\` branch of the Warehouse repository * 2024-08-21T18:55:42 UTC - The container image is built and deployed to production * 2024-08-21T18:58:37 UTC - PyPI admins are notified of an exception in production affecting a Simple Repository API view At this point, PyPI administrators began working on a fix. **Root Causes and Trigger** PyPI administrators immediately determined that the recent merge was at fault due to timing and clear signals from the error observed. As a result of PyPI’s container orchestration and deployment service not explicitly supporting rollbacks, issues like this are generally resolved by introducing a new pull request which reverts the problematic commit, which in turn is merged and [deployed.](http://deployed.In) In this instance, a revert commit was prepared, however the decision was made to wait for a forward fix to be prepared instead. This was partly because it was thought that this fix would be a trivial change and could be prepared as quickly as a revert, thus saving time overall. However, additionally, the introduction of the new feature included a database migration which had already been applied in production. This migration could not be reverted simply by reverting the commit, as deployment would fail when it would attempt to bring the database up to date with the migration history, and find that the current migration version was no longer present. This instead would require an additional migration to migrate the database forwards to a state that would be compatible with the reverted commit, which would potentially take more time than introducing a fix. The fix was prepared, merged and deployed, however it contained an additional issue that resulted in it not ultimately resolving the root cause. An additional pull request to rectify the issue with the forward fix was prepared. At this time, PyPI administrators noted that PyPI’s CI/CD suite was failing to build the containers required to run the test suite on new pull requests, due to the \`/simple\` detail page for one of PyPI’s dependencies being unresolvable due to the outage. As a result, the team decided to pursue a full revert of the feature instead, hoping to leverage our container layer cache to ensure that external requests to PyPI would not be necessary to build. An additional pull request to fully revert the faulty commit was prepared. While the container image build on the new PR was successful, a CI/CD check which did not use the container image, namely a check that ensures Warehouse’s dependency lock files are up-to-date, failed due to the outage. PyPI administrators attempted disabling the required status of these checks in branch protection for the repository, however the failed dependency check caused the remaining tests to be canceled before completion, which prevented the deployment pipeline from picking up on the new commit and triggering a deployment. At this time, PyPI administrators determined that overriding the build/deploy service to manually revert the deployed container image back to a known good image, bypassing the release phase that included migrations, was required. **Mitigation** Mitigation required bypassing PyPI’s build and deploy pipeline to manually re-deploy a previously built image, without running the release phase which included migrations. This allowed PyPI to successfully respond to previously failing requests, allowing the reverted pull request to build and be deployed, fully resolving the issue in production. * 2024-08-21T20:17:58 UTC - The revert was fully deployed to production and errors subsided Additionally, after the production incident was resolved, it was determined that although the final deployment to TestPyPI had been reported as successful, it had not succeeded due to the database migration failing to get a lock, highlighting that there was additionally a logic error in our deployment reporting when deployments fail. **Lessons Learned** * Until additional protections are in place, when an outage affects \`/simple\`, reverts must be immediate or they run the risk of requiring manual intervention. * It is clear that a lack of functional testing is a gap in assurances against production outages. **Things That Went Well** * Maintaining cached container image layers proved potentially useful for recovering from issues where new container images could not be built due to the outage. **Things That Went Poorly** * PyPI used to maintain a mirror of itself on a third-party service to prevent incidents like this, however this is no longer used due to maintenance of the mirror becoming an issue due to on-disk size. * The decision to not revert and fix forward instead proved to not work out. While the additional time to do the additional work to revert could have still resulted in a stuck merge, waiting for the forward fix, which had additional issues, resulted in enough time passing for one of PyPI’s dependencies to be affected by the outage. * A popular dependency, which was also a Warehouse sub-dependency, quickly fell out of cache. PyPI’s CDN is configured to serve stale responses while it revalidates for 5 minutes, and to serve stale responses while the backends are returning errors for 1 day, so the CDN should not have served an error for any of these pages. * The lack of a mirror and a production dependency on PyPI allowed for an ‘ouroboros’ style paradoxical outage that could not be resolved by a simple revert. * A subset of CI/CD checks in the critical path to deployment reach out to PyPI directly as part of their tests, and as a result they will likely fail in conditions like this outage. * Failing CI/CD checks that were not strictly required by branch protection rules were preventing the deployment pipeline from proceeding with the deployment due to setting \`cancel-in-progress: true\` on the parent workflow, causing the workflow to fail when they failed or were canceled. * The Warehouse codebase has extensive unit testing which depends on stubbing/mocking which masked the root cause. The codebase only recently gained the ability to perform functional testing, but the problematic PR did not add additional functional tests, and the limited functional testing that did exist missed this issue. **Where We Got Lucky** * At the time of the incident, all of PyPI’s administrators were online and able to contribute towards mitigating the outage, including the sole administrator that was capable of performing the manual rollback. **Action Items** * Determine why the sub-dependency fell out of cache more quickly than expected based on the CDN configuration, and how that could have been prevented. * Add functional testing to cover the Simple Repository API, with a focus on critical code paths that can cause similar outages. * Update the deployment pipeline to correctly report on deployment status. * Introduce rollback functionality for the deployment pipeline so that a rollback does not require manual intervention or specific knowledge. * Re-introduce a mirror \(either static or a caching proxy\) for PyPI that would not be affected by production outages. * Revisit the CI/CD dependency check and determine how it be hardened against a production outage to not block a deployment.

1. 2024년 8월 9일 오후 6:22 UTCinvestigating

   We are currently investigating an issue with PyPI uploads causing uploads to fail.

2. 2024년 8월 9일 오후 7:08 UTCidentified

   The issue has been identified and a fix is being implemented.

3. 2024년 8월 9일 오후 7:23 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

4. 2024년 8월 9일 오후 7:26 UTCresolved

   Uploads are now being accepted properly.

1. 2024년 7월 31일 오전 3:21 UTCinvestigating

   We are currently investigating an issue with the PyPI backend due to a recent deploy.

2. 2024년 7월 31일 오전 3:34 UTCresolved

   This incident has been resolved.

1. 2024년 7월 12일 오후 2:28 UTCidentified

   We are performing upgrades to PyPI's database storage which may impact performance and availability of the service as they rollout.

2. 2024년 7월 12일 오후 4:18 UTCresolved

   This maintenance is complete.

1. 2024년 6월 8일 오후 1:43 UTCidentified

   We will be performing database maintenance including version upgrades and storage re-configuration. This will result in several brief outages for the service.

2. 2024년 6월 8일 오후 1:58 UTCmonitoring

   Underlying upgrades are complete, we are running database statistics updates to optimize performance.

3. 2024년 6월 8일 오후 2:26 UTCmonitoring

   We are in the final steps of the upgrade and should see performance and availability stabilize in the next 30 minutes.

4. 2024년 6월 8일 오후 2:42 UTCmonitoring

   All upgrade steps are complete and performance is stabilizing. A final maintenance task is taking place in the background which will minimally impact performance until it is complete.

5. 2024년 6월 8일 오후 2:52 UTCresolved

   All maintenance is complete and telemetry around database and application performance have stabilized. This incident is complete.

1. 2024년 6월 5일 오후 3:38 UTCidentified

   An outage in our infrastructure is impacting availability of us.pycon.org and python.org

2. 2024년 6월 5일 오후 3:57 UTCmonitoring

   A cascading failure due to an expired certificate led to this outage. The certificate has been expired and systems are coming back online.

3. 2024년 6월 5일 오후 4:05 UTCresolved

   This incident has been resolved.