Python Package Index Status

The incident has been resolved. ## Incident Report: Cluster-wide service outage **Duration**: ~36 minutes \(20:04–20:40 UTC\) **Impact**: Some PSF-hosted services were unavailable, including [python.org](http://python.org), [us.pycon.org](http://us.pycon.org), PyPI stats, [bugs.python.org](http://bugs.python.org), and related services. What was unaffected was our other cluster that manages [PyPI.org](http://PyPI.org) among other services related to PyPI. **Root Cause**: During local development of kubernetes workloads locally there was an incorrect context switch to one of our production clusters. The scale-down commands ran against the production cluster instead of the local environment, iterating through all deployments and setting them to zero replicas. which created cascading failures. **Recovery**: Services were restored with the help of Ee Durbin by bringing up infrastructure in dependency order, original replica counts were recovered from Kubernetes event history. Action items: * Separate kubeconfig files for production vs local, rather than relying on context switching * Research adding admission control or policies to prevent bulk scale-to-zero operations * Document the infrastructure dependency chain and recovery runbook for future incidents Jacob Coffee, PSF Infrastructure Team

This incident has been resolved.

This incident has been resolved.

This incident has been resolved.

This incident has been resolved.

We have not seen a return of the traffic that caused this incident after removing the temporary block. We will open a new incident if we need to take action again.

This incident is resolved. We're going to do some further investigation to understand what led to the lock contention.

This incident has been resolved.

This incident has been resolved.

This incident has been resolved.

This maintenance is complete! All metrics have returned to baseline.

The scheduled maintenance has been completed.

The scheduled maintenance has been completed.

We have completed this maintenance! As far as we are aware no user impact was reported at any phase aside from the temporary upload suspension. If you do experience issues with files.pythonhosted.org in the coming weeks that may be related, you can report to https://github.com/pypi/warehouse/issues/13403.

While successful and without error, this database maintenance took a little over 200% of the time we expected and communicated. In analyzing the performance of the dump/restore process necessary for upgrading major versions of our database, we noted that one table in particular \(our largest\) took nearly four times longer than in initial testing of the procedure while all other tables performed as observed in tests. We identified two differences between our trial runs and the actual final migration: - PyPI’s codebase being in read-only mode - Upgrades to the database client utilities We have nothing to indicate that changing our codebase to stop database writes would have impacted performance, and are reasonably confident that using different versions of the dump/restore utilities may have impacted our estimates. We will consider other methods for this kind of maintenance in the future and apologize for the extended duration of this maintenance. After our previous major database upgrade in December 2021, we are satisfied with the minimized impact on installers in this maintenance window but have a goal of reducing the duration that PyPI needs to be in read-only in the future.

The scheduled maintenance has been completed.

The scheduled maintenance has been completed.

The scheduled maintenance is complete. Please report any issues you might experience to https://github.com/python/psf-infra-meta/issues

The scheduled maintenance has been completed.

The scheduled maintenance has been completed.