52 Pulumi incidents · september 2023 — official updates, affected components, duration and resolution details.
GitHub har rapporteret en aktiv, kritisk hændelse. Relateret funktionalitet i pulumi-konsollen er midlertidigt utilgængelig.
http: / / www.githubstatus.com / parties / zkxwbgr0cnmx
Denne hændelse er blevet løst.
Automatisk oversat fra den officielle hændelsesopdatering.
Vi undersøger et problem, hvor nogle Team Access Tokens er ude af stand til at autentificere til Pulumi Cloud API, som kan resultere i mislykkede implementeringer eller API fejl. Vi har identificeret årsagen og et fix er ved at blive udsendt. Vi vil give en opdatering, når fixet er live.
Vi fortsætter med at arbejde på en løsning på dette problem.
Et fix er blevet gennemført, og vi overvåger resultaterne.
Denne hændelse er blevet løst.
Automatisk oversat fra den officielle hændelsesopdatering.
We are aware of an issue affecting login access to Pulumi Cloud and are investigating.
We have taken mitigation steps and are seeing signs of recovery. We are continuing to monitor the situation.
We are continuing to monitor for any further issues.
This incident has been resolved.
We are aware of degraded performance on Pulumi APIs and are actively investigating the issue.
We are continuing to investigate this issue.
We have identified a likely cause and are working on steps to address the issue.
A fix has been rolled out and we are monitoring results
We are continuing to monitor for any further issues.
This incident has been resolved.
We are investigating an issue affecting customers who sign in to Pulumi Cloud with a Bitbucket identity or use Bitbucket-connected accounts. Beginning at approximately 06:00 UTC on June 11, requests to refresh Bitbucket authentication tokens began failing. Affected users may experience failed sign-ins with Bitbucket, as well as intermittent errors in the Pulumi Console and API after their current Bitbucket session token expires (these tokens expire every two hours). Integrations that rely on Bitbucket credentials may also be affected. Our investigation indicates the failures originate from Bitbucket's OAuth token service rather than Pulumi infrastructure, and we have engaged Atlassian support. Customers using GitHub, GitLab, email, or SSO identities are unaffected. Workaround: Customers whose Pulumi account has an additional sign-in method (GitHub, GitLab, email/password, or SSO) can continue to sign in using that method. We will post updates as the investigation progresses.
Atlassian has acknowledged an incident affecting Bitbucket OAuth login, which is the cause of the Bitbucket sign-in and authentication failures affecting Pulumi Cloud. Their incident is being tracked at: https://bitbucket.status.atlassian.com/incidents/h51sr145gsmx The issue is external to Pulumi, and no action is required from Pulumi customers. Until Atlassian resolves their incident, customers signing in with a Bitbucket identity may continue to experience failed sign-ins and intermittent Console and API errors after their current session token expires. Customers using GitHub, GitLab, email, or SSO identities remain unaffected. Workaround: Customers whose Pulumi account has an additional sign-in method (GitHub, GitLab, email/password, or SSO) can continue to sign in using that method.
Atlassian has deployed a fix for their incident affecting Bitbucket OAuth login, and we have confirmed that Bitbucket authentication requests from Pulumi Cloud are succeeding again. Error rates have returned to normal levels and sign-in with Bitbucket is working.
The incident has now been resolved and the service is operating normally for all customers.
Investigating elevated API error rates and response times, particularly with ESC endpoints
A fix has been deployed and error rates and response times have recovered. We are continuing to monitor.
The incident has been resolved.
Some Pulumi Cloud organizations experienced elevated errors when decrypting ESC environments. A subset of environment revisions were affected, and those revisions failed to decrypt 100% of the time during the window. Overall, 3.32% of decrypt-attempt requests across Pulumi Cloud failed due to this issue. At the start of the window, we rolled out a change that prevented secret ciphertext copied from an ESC environment from being decrypted in other environments. Customers who had previously copied ciphertext between environments were affected. Users saw a message: “decryption failed. If this secret's ciphertext was copied from a different environment, copy and paste the plaintext instead.” This change was intended as a security improvement, but the customer impact was higher than expected, so the change was rolled back. The ESC team has conducted an internal postmortem to understand the misjudged customer impact, improve our responsiveness to scenarios like this, and created a plan to launch this security improvement without customer disruption.
Users authenticating with Bitbucket are currently unable to sign in to Pulumi Cloud or use the Pulumi Cloud API. This affects all operations that rely on Bitbucket authentication, including CLI usage and API calls. Other authentication methods are not affected. We have identified the cause of the issue and are actively working on a fix. We will provide an update once the fix has been deployed.
We have started deploying a fix for this issue. We will provide an update once the deployment is complete.
The fix has been deployed and we are seeing recovery. We are continuing to monitor to confirm the issue is fully resolved.
This incident has been resolved.
We are aware of reduced response times across Pulumi Cloud APIs. We have identified the cause and are working on a fix
This incident has been resolved.
We are aware of issues with machine tokens having issues opening environments as part of running previews. We are currently investigating
We are continuing to investigate this issue.
We are working on a rollback that should resolve this incident. As this rollback is deploying, you may see error rates decrease for a time until the deploy is complete.
Rollback has deployed. We are continuing to monitor
This incident has been resolved.
We have received reports of errors when creating new Pulumi Neo tasks. We are investigating the issue.
We've identified and rolled back the change that caused the issue. The incident is now resolved.
Cloudflare is having an outage which is impacting certain regions ability to resolve artifacts via get.pulumi.com
Cloudflare has restored the impacted services.
These are affected by the global AWS outage
Customers in the eastern part of the United States may experience intermittent errors loading documentation and registry pages
We are continuing to work on a fix for this issue.
All impacted areas have recovered
We are currently experiencing issues with displaying some previews in Pulumi Cloud. Our team is actively investigating the problem and working toward a resolution. We will provide updates as soon as more information is available.
We’ve identified the root cause of the preview display issues in Pulumi Cloud and are actively rolling out a fix. Thanks for your patience — we’ll share another update once the rollout is complete.
The issue affecting previews in Pulumi Cloud has been resolved. Previews are functioning normally again. Please note that affected previews during the incident window (between 11pm UTC Sept 7, 2025 and 7pm UTC Sept 8, 2025) were not recoverable and will not be available in your history.
We are currently investigating a major issue with a release this morning. We are in the process of rolling this back.
We are noticing a significant drop in error rates and are continuing to monitor.
This incident has been resolved.
The issue has been identified and a fix is being implemented.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
As part of a programmed maintenance, old stacks might not load logs or outputs. We are actively working on resolving this issue
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
The issue has been identified and a fix is being implemented.
A fix has been implemented and is being deployed. We are monitoring rollout, ETA of approx 30 minutes.
This incident has been resolved.
The issue has been identified and a fix is being implemented.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We are currently tracking an issue where logs for updates aren't being saved and displayed
A fix is being prepared.
This incident has been resolved.
