Störungsverlauf von Loom

1. 8. Mai 2026 um 03:00 UTCinvestigating

   We are experiencing issues with multiple Atlassian products. Our teams are investigating further and more updates including will be shared within 1 hour.

2. 8. Mai 2026 um 04:10 UTCidentified

   We have identified that the root cause of the issue is related to an infrastructure outage from our public cloud provider. We are working closely with them to mitigate this issue. We will provide further updates when they become available.

3. 8. Mai 2026 um 04:10 UTCidentified

   Our teams continue to work on mitigating the infrastructure outage from our public cloud provider. We will provide further updates when they are available.

4. 8. Mai 2026 um 04:10 UTCidentified

   We are continuing to work with our public cloud provider to mitigate this issue. We are starting to see some recovery in regions outside of Eastern USA, however, users globally may still be experiencing issues with certain product features. These are listed at the bottom of each product page.

5. 8. Mai 2026 um 05:53 UTCmonitoring

   The underlying issue in public infrastructure which affected asynchronous event processing has been mitigated and all the affected services are recovering. We are now working on clearing the backlog of queued events, which means some actions (such as notifications, automation triggers, and data syncs) may be in degraded state. We will continue to monitor and provide updates as the backlog is cleared.

6. 8. Mai 2026 um 08:03 UTCmonitoring

   We continue to monitor the situation as services recover. We are currently in the process of clearing the backlog of queued events. We will provide a further update in approximately one hour.

7. 8. Mai 2026 um 11:12 UTCmonitoring

   Our services are now fully operational. We continue to replay any events that were missed during the incident, and are making good progress. We will provide a further update once replays are complete. If you experience any ongoing issues, please contact our support team. We apologise for the disruption and thank you for your patience.

8. 8. Mai 2026 um 19:45 UTCresolved

   On May 8, 2026, some customers utilizing Atlassian products experienced elevated error rates and degraded performance. The issue has now been resolved, and the service is operating normally for all affected customers.

9. 20. Mai 2026 um 06:57 UTCpostmortem

   All dates and times below are in UTC unless stated otherwise. ### Summary On May 8, 2026 between 00:22 and 06:08, one of our hosting providers suffered a significant incident in a specific availability zone in prod-east which led to Atlassian customers experiencing degraded performance and delays of background operations and automation execution. The incident started on May 8, 2026 at 00:22 and was detected within 4 minutes by automated monitoring systems. Our teams worked to restore core access by 06:08. Final cleanup of backlogged processes and minor issues progressed in stages from there was completed iteratively by 19:15. ### **IMPACT** The primary infrastructure affected in this incident was the event processing pipeline in the prod-east region, which distributes events between Atlassian services and underpins background operations such as automation execution, search indexing, notifications, permission synchronisation. * Between 00:22 and 06:08, an infrastructure incident in our hosting provider triggered an ingestion failure in our event processing pipeline. * At 02:50, event ingestion was failed over to an unaffected availability zone, progressively restoring live event flows. * At 06:08, reliability for new ingestion in prod-east recovered to 100%. The remaining work was to drain the accumulated cross-region backlog of messages, which completed by 17:00. * By 18:48, Automation had processed their backlog of events that were created while processing **Automation** Between 00:22 and 02:50, customers with automation rules triggered by events originating from the prod-east region experienced a significant reduction in rule executions. During this window, event-triggered automation rules were not firing because the events that trigger them were not being delivered. Rule authoring, saving, and rules triggered manually, by schedules, or by webhooks were not affected. At 02:50, the event processing infrastructure failed over to an unaffected availability zone, restoring delivery of live events to Automation and allowing new event-triggered rules to begin executing normally. However, events generated during the impact window still needed to be replayed before delayed automations could be processed. Beginning at 08:28, upstream services replayed their queued events in a coordinated sequence, and all replayed events were processed by 18:48. During the replay window, customers may have experienced automation rules executing later than expected, a small number of rules reaching daily processing limits due to compressed replay, and time-sensitive rules not completing as expected if internal timeout thresholds were exceeded. **Jira and Jira Service Management** Between 00:22 and 02:50, customers with tenants hosted in the prod-east region experienced disruption to Jira and Jira Service Management event-driven features like automation, along with a short period of elevated errors during infrastructure failover. Core Jira experiences, including issue view, boards, and project navigation, remained available throughout the incident. Jira event delivery was affected by the primary impact, preventing downstream services from receiving issue lifecycle events. This affected automation rules triggered by Jira events, AI agent orchestration in Jira, notifications for issue updates and transitions, search indexing for newly created or modified issues, and event-driven integrations between Jira and other Atlassian products. At 02:50, the event processing infrastructure failed over to an unaffected availability zone, restoring delivery of new events. All events generated during the impact window were retained in a recovery queue and required replaying. This began at 08:28 and completed at 12:00. During the replay window, customers may have experienced automation rules executing later than expected, delayed notifications arriving hours after the triggering action, temporary gaps in search results for content created or modified during the impact window, and AI agent workflows not completing as expected where internal timeout thresholds were exceeded. **Confluence** Between 00:22 and 02:50, customers with tenants hosted in the prod-east region experienced disruptions to event-driven services in Confluence. This resulted in delays to search indexing, notifications, automation rule execution, and permission synchronisation. The underlying event processing infrastructure failed over to an unaffected availability zone, after which live Confluence operations resumed normally. However, events generated during the impact window were queued for replay, and some background services remained delayed until that replay and related validation work completed. Between 10:14 and 17:00, a bulk replay of all the queued tenant replay tasks was completed to restore data consistency. During and immediately after the replay window, customers may have experienced search results not reflecting content created or modified during the outage, delayed or missing notifications for page and comment activity, automation rules firing later than expected, and brief delays in permission synchronisation for tenants relying on incremental identity sync. **Bitbucket and Pipelines** Between 00:22 and 06:08, customers using Bitbucket and Pipelines experienced failures and degraded functionality across event-driven workflows. Core Git operations, including push, pull, and clone, were not affected and continued to operate normally throughout the incident. Automatic pipeline triggers initiated by push or pull request events were unavailable during the impact window. Merge queues, custom merge checks, Forge-based triggers, workspace permission changes, and some workspace provisioning flows were also affected. Customers using merge queues were unable to merge pull requests, and some pipeline steps failed because queued work contributed to elevated concurrency limits. At approximately 03:57, Pipelines was reconfigured to consume events through an alternative path, restoring automatic pipeline triggering. Merge queues, custom merge checks, Forge triggers, and other affected workflows were progressively restored as the underlying event processing infrastructure recovered. All Bitbucket and Pipelines services were confirmed fully operational by 06:08. After recovery, queued events were reviewed and replayed where safe to restore data consistency for billing, audit logging, and other background processes. **Identity Services** Between 00:22 and 02:50, customers with tenants hosted in the prod-east region experienced delays in the propagation of identity and group membership changes to downstream Atlassian products. Core identity operations, including authentication, login, and direct group management actions, were not affected and continued to function normally throughout the incident. The impact was limited to asynchronous, event-driven operations that depend on the event processing pipeline. This included delays in delivering group membership and user profile changes to products such as Jira and Confluence, which affected downstream permission synchronisation and crowd sync flows. A small number of SCIM-based identity synchronisation and site provisioning workflows also experienced temporary delays. After the event processing infrastructure recovered, backed-up identity and group directory events were replayed where required, restoring downstream consistency for affected products. No identity data was lost. Group membership changes, user profile updates, and provisioning-related events that occurred during the impact window were retained and processed after recovery. ### **REMEDIAL ACTIONS PLAN & NEXT STEPS** We know outages impact your productivity. While our monitoring and recovery processes helped us respond quickly, this incident highlighted opportunities to further strengthen resilience for event-driven services. We are prioritizing improvements that will: * **Enhance failover coverage** so critical event processing can recover more smoothly during infrastructure disruptions. * **Strengthen recovery handling** so replayed events can be processed more quickly. We apologize to customers whose services were impacted during this incident; we are taking immediate steps to improve the platform’s performance and availability. Thanks, Atlassian Customer Support.

1. 29. April 2026 um 07:00 UTCinvestigating

   We are investigating degraded performance issues affecting Loom's core functionalities - authentication, recording, viewing and editing looms. We will provide more details within the next hour.

2. 29. April 2026 um 07:50 UTCresolved

   The Team has identified the root cause and the issue has now been resolved, and the impacted services are operating normally. Team will continue to monitor the impacted services.

1. 3. Februar 2026 um 10:24 UTCinvestigating

   Customers utilizing Confluence and Jira Mobile may experience disruptions in OAuth authentication flows. Forge installations and invocations might also be disrupted. Our team is actively investigating the same and we shall keep you informed of the progress within next 60 mins or sooner.

2. 3. Februar 2026 um 10:50 UTCresolved

   Team has identified the root cause and the issue has now been resolved, and the impacted services are operating normally. Team will continue to monitor the impacted services.

1. 19. November 2025 um 22:32 UTCidentified

   We are aware of some customers that are experiencing no audio when viewing their Loom recordings. Our team has identified this issue and is working urgently to restore the audio for these recordings. Please rest assured that the audio has been recorded correctly, and we expect this to be fully available when this incident is resolved. New recordings are no longer impacted by this issue. We will provide an update within two hours with our progress as the team continues to investigate.

2. 20. November 2025 um 00:33 UTCidentified

   Our team believes that we have identified the cause of the issue impacting audio during playback of videos for some customers. As mentioned in our previous updates, new recordings will be working as expected, however videos recorded between 19 Nov 16:51 UTC and 19 Nov 21:30 UTC are expected to be impacted by this issue. A fix for this issue to restore audio to these recordings is now being developed and tested, and we expect to have further update within 2 hours on progress for this resolution.

3. 20. November 2025 um 02:30 UTCidentified

   The fix to resolve issues on audio recordings impacted by this incident is now being prepared for deployment. Currently the expected duration of the rollout is 3 hours, at which point we expect the audio from the impacted recordings to be fully restored. We will provide further update at that time with the progress of the restoration and provide any additional information that may be required.

4. 20. November 2025 um 05:39 UTCresolved

   The rollout of the fix for impacted videos without audio has now completed. Customers should now be able to review existing videos without audio and see that it has been properly restored. If you are continuing to see any issues with playback of your videos, please reach out to Loom support for assistance. We apologize for any inconvenience caused by this issue.

1. 27. Oktober 2025 um 15:35 UTCinvestigating

   Wir untersuchen ein Problem mit Recording, Loom AI, Authentifizierung und Abrechnung, das einige Loom Cloud-Kunden betrifft. Wir werden innerhalb der nächsten Stunde weitere Details mitteilen.

2. 27. Oktober 2025 um 16:33 UTCinvestigating

   Wir untersuchen weiterhin das Problem mit Recording, Loom AI, Authentifizierung und Abrechnung, das einige Loom Cloud-Kunden betrifft. Wir untersuchen und werden Updates um 17:35 UTC oder früher veröffentlichen.

3. 27. Oktober 2025 um 17:34 UTCinvestigating

   Wir untersuchen weiterhin das Problem mit Recording, Loom AI, Authentifizierung und Abrechnung, das einige Loom Cloud-Kunden betrifft. Wir werden ein weiteres Update um 18:35 UTC oder früher veröffentlichen, wenn die Dinge voranschreiten.

4. 27. Oktober 2025 um 18:32 UTCinvestigating

   Wir untersuchen weiterhin aktiv das Problem mit Recording, Loom AI, Authentifizierung und Abrechnung, das einige Loom Cloud-Kunden betrifft. Wir schätzen Ihre Geduld. Wir werden ein weiteres Update um 19:35 UTC oder früher veröffentlichen, wenn die Dinge voranschreiten.

5. 27. Oktober 2025 um 19:06 UTCmonitoring

   Wir haben die Ursache des Problems mit Recording, Loom AI, Authentifizierung und Abrechnung identifiziert, was sich auf einige Loom Cloud-Kunden auswirkt. Wir haben Minderungsmaßnahmen ergriffen, um das Problem anzugehen, und beobachten nun genau. Wir werden ein weiteres Update um 20:00 UTC oder früher bereitstellen, wenn die Dinge voranschreiten.

6. 27. Oktober 2025 um 19:59 UTCmonitoring

   Wir implementieren aktiv weitere Maßnahmen, um das Problem mit Recording, Loom AI, Authentifizierung und Abrechnung für einige Loom Cloud-Kunden zu beheben. Wir werden weiterhin genau beobachten und ein weiteres Update um 21:00 UTC oder früher bereitstellen, wenn die Dinge voranschreiten.

7. 27. Oktober 2025 um 21:06 UTCmonitoring

   Wir haben weitere Abhilfemaßnahmen implementiert, um das Problem mit Recording, Loom AI, Authentifizierung und Abrechnung für einige Loom Cloud-Kunden zu beheben. Wir sehen gute Anzeichen für eine Erholung und alle Dienste sind in Betrieb. Wir werden weiterhin genau beobachten und ein weiteres Update um 22:00 UTC oder früher bereitstellen, wenn die Dinge voranschreiten.

8. 27. Oktober 2025 um 22:00 UTCresolved

   Am 27. Oktober 2025, zwischen 12:30 und 22:15 UTC, erlebten wir für einige Loom-Kunden eine verschlechterte Leistung, die sich auf die Dienste Recording, Loom AI, Authentication und Billing auswirkte. Wir haben einen Fix eingesetzt, um das Problem zu mildern und haben verifiziert, dass sich die Dienste erholt haben. Die Bedingungen, die das Problem verursachen, wurden behoben und wir arbeiten aktiv an einer dauerhaften Lösung. Das Problem wurde behoben und der Dienst funktioniert normal. Sobald wir unseren internen Incident-Review-Prozess abgeschlossen haben, werden wir postmortem eine detailliertere Darstellung dessen veröffentlichen, was schief gelaufen ist, zusammen mit den Schritten, die wir unternehmen, um zu verhindern, dass dies in Zukunft erneut geschieht. Anmerkung: Infolge dieses Vorfalls können einige Webstühle, die während des Zeitraums der verschlechterten Leistung aufgezeichnet wurden, in einem "Upload" -Zustand geblieben sein. Diese Videos sind nicht wiederherstellbar und wurden aus den Bibliotheken der Benutzer archiviert.

9. 21. November 2025 um 19:49 UTCpostmortem

   ### Zusammenfassung Am 27. Oktober 2025 zwischen 12:30 und 20:55 UTC wurde Loom für alle Kunden deutlich abgebaut. Die Benutzer erlebten auch kurze Degradationsperioden am 28. Oktober 2025 von 20:26 bis 20:33 UTC und am 30. Oktober 2025 von 19:22 bis 19:27 UTC. Während des Vorfalls traten Probleme mit intermittierender oder fehlgeschlagener Wiedergabe, Aufnahmen, Erstellung von Transkripten, Anmeldungsproblemen und allgemeinen Fehlern oder Langsamkeit im gesamten Dienst auf. Unsere Überwachung zeigt, dass zwischen 20% und 80% der Benutzerinteraktionen in diesen Zeiträumen fehlschlugen, wobei die Fehler in der späteren Hälfte des Vorfalls zunahmen. Wir wissen, dass Ausfälle Ihre Produktivität beeinträchtigen, und wir entschuldigen uns bei Kunden, die während dieses Vorfalls betroffen waren. Wir unternehmen sofortige Schritte, um die Zuverlässigkeit von Loom auf der Grundlage unserer Erkenntnisse aus diesem Vorfall zu verbessern. ### Analyse Ab dem 25. Oktober 2025 zeigte der primäre Datenbankcluster von Loom erste Anzeichen für ein Problem. Der Ressourcenverbrauch wuchs langsam und Replikationsslots hielten nicht Schritt. Ingenieure wurden am 26. Oktober 2025 alarmiert und begannen mit der Untersuchung. Wir haben zu diesem Zeitpunkt keine Anzeichen von Kundenauswirkungen gesehen, aber wir haben ein Problem mit einem wiederholten Hintergrundjob identifiziert, der erfolglos in einer Schleife läuft. Wir haben einen temporären Fix für dieses Problem verschickt und geplant, am nächsten Tag weiter zu untersuchen. Am Montag, den 27. Oktober 2025, tauchte das Problem mit der zunehmenden Nutzung deutlich auf und wir begannen, die Auswirkungen der Kunden zu beobachten. Ingenieure beobachteten das Problem genau und engagierten sich sofort. Es war klar, dass unsere datenbank nicht wie erwartet funktionierte, und in den nächsten stunden haben wir mehrere versuche zur verminderung unternommen. Insbesondere haben wir den Datenbankautor und zwei Leser skaliert, um den Ressourcendruck zu verringern. Dies half nicht, und die Auswirkungen der Benutzer blieben unverändert. In den nächsten Stunden haben wir mehrere Versuche unternommen, die normale Datenbankleistung wiederherzustellen. Wir haben die jüngsten Änderungen zurückgenommen, die Leistung einzelner Abfragen optimiert, neue Indizes hinzugefügt, Datenbankkonfigurationen aktualisiert und große System- und Anwendungstabellen vakuumiert. Diese Änderungen sorgten für eine kurzfristige Erleichterung, aber die Leistung verschlechterte sich immer wieder. Das fallenlassen unserer replikationsslots hat den ressourcendruck beseitigt, aber die gesamtleistung der abfrage nicht verbessert. Wir beobachteten dann, dass identische Abfragen auf dem Schreiber viel schneller geplant und ausgeführt wurden als auf unseren Leseinstanzen, so dass wir beschlossen, den gesamten Abfrageverkehr auf die einzelne Schreibinstanz zu verschieben. Diese Änderung milderte sofort die Kundenauswirkungen. Während diese Aktion das unmittelbare Problem löste, war es langfristig kein stabiler Zustand für unseren Dienst. In den nächsten Tagen haben wir mehrere Versuche unternommen, zu unserer typischen Single-Writer-, Dual-Reader-Architektur zurückzukehren. Jedes Mal sahen wir die gleichen Probleme wieder auftauchen, was zu kurzen Degradationsperioden führte. Zwischen jedem dieser Versuche haben wir weitere Optimierungen für unsere Datenbanknutzung vorgenommen, aber wir sahen immer noch nicht die Leistung, die wir beim Hinzufügen unserer Leseinstanzen erwartet hatten. Dies war besonders herausfordernd, da unsere Dienste wie erwartet mit einem einzigen Leser funktionierten und wir diese Verschlechterung nur beobachteten, wenn wir einen zweiten Leser zum Cluster hinzufügten. Am 31. Oktober 2025 versuchten wir das Failover mit einer kleineren Instanzgröße, ähnlich unserer ursprünglichen Architektur vor dem vertikalen Skalierungsversuch. Dies gelang, indem die Loom-Architektur wieder normalisiert und der Vorfall vollständig abgeschwächt wurde. ### Wurzelursache Im Großen und Ganzen kann dieser Vorfall in zwei Teile unterteilt werden. Erstens wurde die anfängliche Verschlechterung durch mehrere Loom-Änderungen in den Monaten vor dem Vorfall verursacht. Zweitens resultierte die erweiterte Minderung unerwartet aus unserem Versuch, die Skalierung der Instanzgrößen zu mildern. Der erste Teil dieses Vorfalls wurde durch eine komplexe Interaktion mehrerer alter und neuerer Änderungen an Loom-Diensten verursacht: * Seit Jahren hat Loom einen Hintergrundjob, der verschachtelte Datenbanktransaktionen verwendet, ein seltenes Muster in unserer Codebasis. Eine Elterntransaktion wird für den gesamten Job geöffnet, mit Kindtransaktionen für jede Charge von Arbeit. Dieser Job war zuverlässig, wird aber im Falle eines Misserfolgs mehrmals wiederholt. Dieser Job ist relativ kleinvolumig und in der Regel schnell, kann aber in seltenen Fällen erhebliche Datenbankarbeit erfordern. * In den letzten Monaten, um kommende Features und Migrationen zu unterstützen, haben die Basisauslastung und -aktivität in der Loom-Datenbank deutlich zugenommen. * Am 14. Oktober 2025 wurde auch zur Unterstützung einer bevorstehenden Funktion eine umfassende Änderung vorgenommen, wie die Loom-App Datenbanktransaktionen behandelt. Diese Änderung sollte nur in unseren Test- und Staging-Umgebungen wirksam werden, aber in speziellen Fällen im Zusammenhang mit verschachtelten Transaktionen wurde sie auch in der Produktion angewendet. * Am 25. Oktober 2025 lösten die kombinierten Veränderungen eine langsame, zunehmende Verschlechterung unserer Datenbank aus. Mehrere große Hintergrundjobs werden kombiniert und sind fehlgeschlagen, was dazu führt, dass Wiederholungen eine permanente offene Transaktion in der Datenbank beibehalten. Dies verschlechterte Replikationsschlitze und Autovakuum, was zu einer Aufblähung der Tabelle in der Datenbank führte. Am 27. Oktober 2025 begann diese Verschlechterung unsere Benutzer zu beeinflussen. * Nach dem Entfernen der ersten Trigger verschlechterte sich die Datenbank weiter und erforderte erhebliche manuelle Eingriffe, um sich zu stabilisieren. Die Wiederherstellung war kompliziert, weil ein Versuch der Minderung (vertikale Instanz-Skalierung) ein neues Problem auslöste. Der zweite Teil dieses Vorfalls wurde durch unseren früheren Minderungsversuch verursacht, unsere Instanzgrößen zu skalieren, um die erhöhte Belastung aus dem ersten Teil zu bewältigen: * Zu Beginn des Vorfalls haben wir als Reaktion auf die erhöhte Nutzung von Datenbankressourcen versucht, unsere Instanzen vertikal auf größere Größen zu skalieren. Wir betrachten dies im Allgemeinen als eine sichere und oft wirksame Minderung. * Diese Änderung hat die Verschlechterung der Datenbank unerwartet verschlimmert. Die Art der Degradation, die wir von den größeren Instanzen beobachteten, ähnelte der Art der Degradation, die wir bereits im ersten Teil des Vorfalls beobachtet hatten. Nach dem Vorfall haben wir direkt von unserem Cloud-Anbieter erfahren, dass die größeren Instanzgrößen eine separate Speicherarchitektur verwenden, die bei unserer Arbeitslast deutlich schlechter abschneidet. * Dieser Leistungsunterschied zwischen den Instanztypen zeigt sich besonders beim Öffnen neuer Verbindungen. Im Rahmen unserer Untersuchung fanden wir auch ein Problem mit unserem Verbindungspooler, das zu einer übermäßigen Verbindungsabwanderung führt, wenn wir auf einen DNS-Load-Balanced-Reader-Endpunkt zeigen. Dies erklärt, warum wir nur Probleme beim Hinzufügen eines zweiten Lesers zum Cluster beobachtet haben und nicht bei einer Single-Writer-Single-Reader-Konfiguration. ### **Abhilfen und Verbesserungen** Im Rahmen unserer Untersuchungs- und Minderungsbemühungen während des Vorfalls haben wir bereits mehrere wichtige Maßnahmen abgeschlossen, um ähnliche Probleme anzugehen, darunter: * Optimiertes Transaktionshandling innerhalb unserer Anwendung. * Verbesserte Leistung von hochvolumigen und leistungsschwachen Abfragen. * Verbesserte Überwachung und Beobachtbarkeit des Vakuumstatus und der Aufblähung des Tisches. Verbesserungen des internen Tools für ein einfacheres Debuggen * In-Produkt-Banner, um Benutzer bei Vorfällen zu alarmieren. Darüber hinaus priorisieren wir die folgenden Verbesserungsmaßnahmen: * Weitere Verbesserungen bei der Transaktionsbearbeitung und -überwachung. * Optimierung unserer Verbindungspooling-Konfiguration. * Datenbankindex-Nutzungsoptimierungen. * Verbesserung der Datenbankkonfiguration. * Verbesserungen der Autovakuumparameter. * Verbesserung der Art und Weise, wie wir die Auswirkungen von Kunden messen und überwachen. Vielen Dank, Atlassian

Automatisch aus der offiziellen Störungsmeldung übersetzt.

1. 20. Oktober 2025 um 07:56 UTCinvestigating

   We have noticed that Atlassian Cloud services are impacted and our teams are actively investigating the same. We shall keep you informed of the progress every hour.

2. 20. Oktober 2025 um 08:24 UTCinvestigating

   Atlassian Cloud services are impacted and we are aware that our customers might not be able to create support tickets. Our teams are actively investigating the same. We shall keep you informed of the progress every hour.

3. 20. Oktober 2025 um 09:24 UTCinvestigating

   We are experiencing an outage due to some issue at the end of our public cloud provider. We are working closely with them to get this resolved or mitigated as quickly as possible. ETA of the same is not know at the moment. We shall continue to share updates every hour, if not sooner.

4. 20. Oktober 2025 um 09:58 UTCidentified

   We understand that our public cloud provider has identified the cause of the issue. We are starting to see some recovery and is working towards mitigation. We appreciate your patience. We shall continue to share updates every hour, if not sooner.

5. 20. Oktober 2025 um 11:07 UTCidentified

   We continue to work with our public cloud provider towards mitigating the issue at the earliest. We appreciate your patience. We shall continue to share updates every hour, if not sooner.

6. 20. Oktober 2025 um 12:06 UTCidentified

   Atlassian team is actively engaged and continues to work with our public cloud provider to mitigate this issue at the earliest. We are starting to see partial operations succeed. We appreciate your patience. We shall continue to share updates every hour, if not sooner.

7. 20. Oktober 2025 um 13:58 UTCidentified

   Our public cloud provider is working to mitigate this issue quickly. We are seeing some early positive indicators and are continuing to monitor. We appreciate your patience and will continue to provide updates every hour or sooner.

8. 20. Oktober 2025 um 14:03 UTCidentified

   We understand your pain and mitigating or fixing this issue is of utmost importance. Our public cloud provider is actively working to mitigate this issue on priority. We have been seeing partial operational success. We appreciate your patience and will continue to provide updates every hour or sooner.

9. 20. Oktober 2025 um 16:36 UTCidentified

   Update - Thank you for your continued patience. We understand the impact this issue is having on your operations and want to assure you that resolving this matter is our highest priority. Our public cloud provider is still actively working to mitigate this issue with urgency and while we do not have a definitive ETA at this time, we remain committed to full resolution and deeply appreciate your patience as we work through this situation. We will be providing hourly updates on this issue.

10. 20. Oktober 2025 um 17:27 UTCidentified

    Update - We understand the impact this issue is having on your operations and want to assure you that resolving this matter is our highest priority. Our public cloud provider is actively working to mitigate this issue with urgency. While we do not have a definitive ETA at this time, we remain committed to full resolution and deeply appreciate your patience as we work through this situation. We will continue to provide updates every hour or sooner as new information becomes available.

11. 20. Oktober 2025 um 17:43 UTCidentified

    We are currently aware of an ongoing incident impacting Atlassian Cloud services due to an outage with our public cloud provider, AWS. We understand the impact this issue is having on your operations and want to assure you that resolving this matter is our highest priority and we are closely monitoring the health of AWS services. While we do not have a definitive ETA at this time, we remain committed to full resolution and deeply appreciate your patience as we work through this situation. We will continue to provide updates every hour or sooner as new information becomes available.

12. 20. Oktober 2025 um 18:38 UTCmonitoring

    There have been no changes since our last update. We will provide our next updated by 9:00PM UTC or sooner as new information becomes available. We are currently aware of an ongoing incident impacting Atlassian Cloud services due to an outage with our public cloud provider, AWS. We understand the impact this issue is having on your operations and want to assure you that resolving this matter is our highest priority and we are closely monitoring the health of AWS services. While we do not have a definitive ETA at this time, we remain committed to full resolution and deeply appreciate your patience as we work through this situation.

13. 20. Oktober 2025 um 21:26 UTCmonitoring

    Monitoring - We've started seeing continued product experience improvement. While we still have a backlog of event processing, we are seeing improvements in systems operational capabilities across all products. We estimate a significant improvement with the next few hours and will continue to monitor the health of AWS services and the effects on Atlassian customers. We appreciate your continued patience and remain committed to full resolution as we work through this situation. We will post our next update in two hours.

14. 20. Oktober 2025 um 22:58 UTCmonitoring

    Our teams are continuing to monitor the recovery of systems across Atlassian products. This update is to inform that the Atlassian Support portal is fully operational at this time for customers that wish to contact support.

15. 21. Oktober 2025 um 00:03 UTCmonitoring

    Our team is now seeing recovery across all impacted Atlassian products. We are continuing to monitor for individual products that may still be processing backlogged items now that services are restored. The Atlassian Support portal is currently still displaying a message directing customers to our temporary support channel. Please note that our support portal is currently fully functional for those attempting to raise requests. We are continuing to look into this alert to remove this message. We will provide further update on our recovery status in one hour.

16. 21. Oktober 2025 um 01:07 UTCmonitoring

    We continue to see recovery progressing across all impacted products as backlogged items continue to be processed. The Atlassian Support portal is currently displaying a message directing customers to our temporary support channel. Please note that our support portal is currently fully functional for those attempting to raise requests. We are continuing to look into this alert to remove this message. We will provide further update on our recovery status in two hours.

17. 21. Oktober 2025 um 03:24 UTCmonitoring

    The issue relating to the Atlassian Support portal displaying a message to customers to use our temporary support channel has now been resolved. The Atlassian Support portal is fully functional for any ongoing support issues. With regards to other Atlassian products, we continue to see recovery continuing across all impacted products and our teams are continuing to monitor as the recovery continues. We will provide further update on our recovery status within two hours.

18. 21. Oktober 2025 um 05:24 UTCresolved

    Our team is now able to see full recovery across the vast majority of Atlassian products. We are aware of some ongoing issues with specific components such as migrations and JSM virtual service agents, and our team is continuing to investigate with urgency. We apologise for the inconvenience that this incident has caused and we will provide further information when the Post Incident Investigation has been completed.

19. 24. Oktober 2025 um 17:53 UTCpostmortem

    ### Postmortem publish date: Nov 19th, 2025 ### Summary All dates and times below are in UTC unless stated otherwise. Customers utilizing Atlassian products experienced elevated error rates and degraded performance between Oct 20, 2025 06:48 and Oct 21, 2025 04:05. The service disruptions were triggered due to an [AWS DynamoDB outage](https://aws.amazon.com/message/101925/#:~:text=1%3A50%20PM.-,DynamoDB,-Between%2011%3A48) and further affected by subsequent failures in [AWS EC2](https://aws.amazon.com/message/101925/#:~:text=service%20disruption%20event.-,Amazon%20EC2,-Between%2011%3A48) and [AWS Network Load Balancer](https://aws.amazon.com/message/101925/#:~:text=service%20disruption%20event.-,Amazon%20EC2,-Between%2011%3A48) within the us-east-1 region. The incident started at Oct 20, 2025 06:48 and was detected within six minutes by our automated monitoring systems. Our teams worked to restore all core services by Oct 21, 2025 04:05. Final cleanup of backlogged processes and minor issues was completed on Oct 22, 2025. We recognize the critical role our products play in your daily operations, and we offer our sincere apologies for any impact this incident had on your teams. We are taking immediate steps to enhance the reliability and performance of our services, so that you continue to receive the standard of service you have come to trust. ### IMPACT Before examining product-level impacts, it's helpful to understand Atlassian's service topology and internal dependencies. Products such as Jira and Confluence are deployed across multiple AWS regions. The data for each tenant is stored and processed exclusively within its designated host region. This design is intentional and represents the desired operational state, as it limits the impact of any regional outage strictly to tenants in-region, in this case us-east-1. While in-scope application data is pinned to the region selected by the customer, there are times when systems need to call other internal services that may be based in a different region. If a problem occurs in the main region where these services operate, systems are designed to automatically fail over to a backup region, usually within three minutes. However, if unexpected issues arise during this failover, it can take longer to restore services. In rare cases, this could affect customers in more than one region. It’s important to note that all in-scope application data for supported products is pinned according to a customer’s chosen region. **Jira** Between Oct 20, 2025 06:48 and Oct 20, 2025 20:00, customers with tenants hosted in the us-east-1 region experienced increased error rates when accessing core entities such as Issues, Boards, and Backlogs. This disruption was caused by AWS's inability to allocate AWS EC2 instances and elevated errors in AWS Network Load Balancer \(NLB\). During this window, users may also have observed intermittent timeouts, slow page loads, and failures when performing operations like creating or updating issues, loading board views, and executing workflow transitions. Between Oct 20, 2025 08:36 and Oct 20, 2025 09:23, customers across all regions experienced elevated failure rates when attempting to load Jira pages. This disruption was caused by the regional frontend service entering an unhealthy state during this specific time interval. Normally, the frontend service connects to the primary AWS DynamoDB instance located in the us-east-1 to retrieve the most recent configuration data necessary for proper operation. Additionally, the service is designed with a fallback mechanism that references static configuration data in the event that the primary database becomes inaccessible. Unfortunately, a latent bug existed in the local fallback path. When the frontend service nodes restarted, they were unable to load critical operational configuration data from primary or fallback sources, leading to the observed failures experienced by customers. Between Oct 20, 2025 06:48 and Oct 21, 2025 06:30, customers experienced significant delays and missing Jira in-app notifications across all regions. The notification ingestion service, which is hosted exclusively in us-east-1, exhibited an increased failure rate when processing notification messages due to AWS EC2 and NLB issues. This issue resulted in notifications being delayed - and in some cases, not delivered at all - to users worldwide. **Jira Service Management \(JSM\)** JSM was impacted similarly to Jira above, with the same timeframes and for the same reasons. Between Oct 20, 2025 08:36 and Oct 20, 2025 09:23, customers across all regions experienced significantly elevated failure rates when attempting to load JSM pages. This affected all JSM experiences including the Help Centre, Portal, Queues, Work Items, Operations, and Alerts. **Confluence** Between Oct 20, 2025 06:48 and Oct 21, 2025 02:45, customers using Confluence in the us-east-1 region experienced elevated failure rates when performing common operations such as editing pages or adding comments. The primary cause of this service degradation was the system's inability to auto-scale due to AWS EC2 issues to manage peak traffic load effectively. Though the AWS outage ended at Oct 20, 21:09, a subset of customers continued to experience failures as some Confluence web server nodes across multiple clusters remained in an unhealthy state. This was ultimately mitigated by recycling the affected nodes. To protect our systems while AWS recovered, we made a deliberate decision to enable node termination protection. This action successfully preserved our server capacity but, as a trade-off, it extended the time required for a full recovery once AWS services were restored. **Automation** Between Oct 20, 2025 06:55 and Oct 20, 2025 23:59, automation customers whose rules are processed in us-east-1 experienced delays of up to 23 hours in rule execution. During this window, some events triggering rule executions were processed out of order because they arrived later during backlog processing. This caused potential inconsistencies in workflow executions, as rules were run in the order events were received, not when the action causing the event occurred. Additionally, some rule actions failed because they depend on first-party and third-party systems, which were also affected by the AWS outage. Customers can see most of these failures in their audit logs; however, a few updates were not logged due to the nature of the outage. By Oct 21, 2025 5:30, the backlog of rule runs in us-east-1 was cleared. Although most of these delayed rules were successfully handled, there were some additional replays of events to ensure completeness. Our investigation confirmed that a few events may never have triggered their associated rules due to the outage. Between Oct 20, 2025 06:55 and Oct 20, 2025 11:20, all non-us-east-1 regional automation services experienced delays of up to 4 hours in rule execution. This was caused by an upstream service that was unable to deliver events as expected. The delivery service encountered a failure due to a cross-region dependency call to a service hosted in the us-east-1 region. Because of this dependency issue, the delivery service was unable to successfully deliver events throughout this time frame, resulting in customer-defined rules not being executed in a timely manner. **Bitbucket and Pipelines** Between Oct 20, 2025 06:48 and Oct 20, 2025 09:33, Bitbucket experienced intermittent unavailability across core services. During this period, users faced increased error rates and latency when signing in, navigating repositories, and performing essential actions such as creating, updating, or approving pull requests. The primary cause was an AWS DynamoDB outage that impacted downstream services. Between Oct 20, 2025 06:48 and Oct 20, 2025 22:46, numerous Bitbucket Pipeline steps failed to start, stalled mid-execution, or experienced significant queueing delays. Impact varied, with partial recoveries followed by degradation as downstream components re-synchronized. The primary cause was an AWS DynamoDB outage, compounded by instability in AWS EC2 instance availability and AWS Network Load Balancers. Furthermore, Bitbucket Pipelines continued to experience a low but persistent rate of step timeouts and scheduling errors due to AWS bare-metal capacity shortages in select availability zones. Atlassian coordinated with AWS to provision additional bare-metal hosts and addressed a significant backlog of pending pods, successfully restoring services by 01:30 on Oct 21, 2025. **Trello** Between Oct 20, 2025 06:48 and Oct 20, 2025 15:25, users of Trello experienced widespread service degradation and intermittent failures due to upstream AWS issues affecting multiple components, including AWS DynamoDB and subsequent AWS EC2 capacity constraints. During this period, customers reported elevated error rates when loading boards, opening cards, adding comments or attachments. **Login** Between Oct 20, 2025 06:48 and Oct 20, 2025 09:30, a small subset of users experienced failures when attempting to initiate new login sessions using SAML tokens. This resulted in an inability for those users to access Atlassian products during that time period. However, users who already had valid active sessions were not affected by this issue and continued to have uninterrupted access. The issue impacted all regions globally because regional identity services relied on a write replica located in the us-east-1 region to synchronize profile data. When the primary region became unavailable, the failover to a secondary database in another region failed, which delayed recovery. This failover defect has since been addressed. **Statuspage** Between Oct 20, 2025 06:48 and Oct 20, 2025 09:30, Statuspage customers who were not already logged in to the management portal were unable to log in to create or update incident statuses. This impact was restricted only to users who were not already logged in at the time. The root cause was the same as described in the Login section above, and it was resolved by the same remediation steps. ### REMEDIAL ACTION PLAN & NEXT STEPS We have completed the following critical actions designed to help prevent cross-region impact from similar issues: * Resolved the code defect in the fallback option to ensure that Jira Frontend Services in other regions remain unaffected during a region-wide outage. * Fixed the issue that prevented timely failover of the identity service which impacted new login sessions. * Resolved the code defect so that delivery services in unaffected regions remain operational during region-wide outages. Additionally, we are prioritizing the following improvement actions: * Implement mitigation strategies to strengthen resilience against region-wide outages in the notification ingestion service. Although disruptions to our cloud services are sometimes unavoidable during outages of the underlying cloud provider, we continuously evaluate and improve test coverage to strengthen resilience of our cloud services against these issues. We recognize the critical importance of our products to your daily operations and overall productivity, and we extend our sincere apologies for any disruptions this incident may have caused your teams. If you were impacted and require additional details for internal post-incident reviews, please reach out to your Atlassian support representative with affected timeframes and tenant identifiers so we can correlate logs and provide guidance. Thanks, Atlassian Customer Support

1. 12. September 2025 um 20:43 UTCmonitoring

   From 5:03 AM ET to 4:15 PM ET, some customers were unable to record Zoom meetings using Loom Meeting Recordings. Attempts made during this time may appear in your library but will not be playable, as no recording was captured. Unfortunately, these recordings cannot be recovered. We have released a fix and we are monitoring recovery.

2. 12. September 2025 um 22:14 UTCresolved

   The issue has been resolved and the service is operating normally.

1. 8. September 2025 um 21:46 UTCresolved

   From 5:20AM to 3:45PM UTC today some customers using the Loom Chrome Extension experienced issues with recording. Looms recorded during this time may show up in the library but will not be playable. Unfortunately we are unable to recover these videos. If you are still experiencing this issue please make sure you are recording with extension version 5.5.121.

1. 15. August 2025 um 19:35 UTCidentified

   We are investigating reports of issues for Loom customers. Including names showing as IDs. We will provide more details once we identify the cause.

2. 15. August 2025 um 19:50 UTCmonitoring

   We have identified the cause of the issues and we are monitoring recovery.

3. 15. August 2025 um 19:58 UTCresolved

   Between 18:45 and 19:35 UTC we experienced degraded functionality for Loom. The issue has been resolved and the service is operating normally.

1. 14. April 2025 um 19:07 UTCidentified

   Customers are unable to choose a monthly plan when upgrading or purchasing from loom.com. We are investigating this issue.

2. 14. April 2025 um 19:30 UTCmonitoring

   A fix has been implemented and we are monitoring the results

3. 14. April 2025 um 19:34 UTCresolved

   This issue is now resolved and customers can checkout as expected.