Opsgenie 장애 기록

1. 2026년 9월 14일 오후 4:46 UTCinvestigating

   Opsgenie 및 Jira Service Management에 영향을 미치는 서비스 중단 보고서를 적극적으로 조사하고 있습니다. 우리는 더 많은 정보로 업데이트를 공유 할 것입니다.

2. 2026년 9월 14일 오후 5:16 UTCidentified

   우리는 문제의 가능성이 원인을 식별하고, 우리의 팀은 크게 완화에 노력하고 있습니다. 우리는 더 많은 정보로 추가 업데이트를 계속 공유 할 것입니다.

3. 2026년 9월 14일 오후 5:27 UTCmonitoring

   이 문제는 해결되었습니다, 그리고 서비스는 일반적으로 모든 영향을받는 고객을 위해 운영됩니다. 우리는 안정성을 확인하기 위해 긴밀히 모니터링 할 것입니다.

4. 2026년 9월 14일 오후 5:43 UTCresolved

   9월 14, 2026일, Opsgenie 및 Jira Service Management는 중단을 경험하고, 영향을 받은 사용자는 사용자 인터페이스에서 경고를 볼 수 있는 알림 및 비활성화를 보았습니다. 문제는 이제 해결되었습니다. 서비스는 일반적으로 모든 영향을받는 고객을 위해 운영됩니다.

5. 2026년 9월 18일 오후 5:02 UTCpostmortem

   ### 요약 9월 14, 2026일, 15:30과 17:10 UTC 사이, Opsgenie, Jira Service Management를 사용하여 Atlassian 고객은 경고 알림 및 Ops 기능을 사용할 때 간헐적 인 실패를 수신하는 데 어려움을 겪었습니다. 이 문제는 핵심 서비스에서 긴 실행 트랜잭션에 의해 트리거되었습니다. 이는 풀 배기에 주도하고 응답하지 않는 후속 요청이 발생했습니다. 긴 실행 거래는 9 월 9, 2026 및 9 월 11, 2026 UTC 사이에 릴리스 된 구성 변경에 의해 발생했습니다. 이 팀은 9 월 14, 2026 UTC까지 시스템의 변화의 영향을 적극적으로 모니터링하고 영향을 미칩니다. 그러나, 9 월 14 일 미국 근무 시간 동안 트래픽 증가, 2026, 이러한 예상치 못한 긴 실행 트랜잭션 발생. 사건은 자동화된 감시 체계에 의해 1 분 안에 검출되고, 가득 차있는 서비스 안정화는 9월 14, 2026에 17:10 UTC에 변화를 돌고 후에 발생했습니다. # # # # IMPACT를 사건은 주로 Opsgenie, Jira 서비스 관리, 그리고 미국 지역에서 호스팅되는 고객을 위한 Compass의 맞은편에 영향을 미칩니다. 사건 중, 경고 생성 및 알림 배달에 대한 엔드 투 엔드 흐름은 평균 지연을 경험 22 분. 경고 또는 알림 페이로드는 사건 중 떨어졌습니다. 모든 누락 된 이벤트는 성공적으로 처리되고 복구 된 서비스로 전달되었습니다. 또한 관련 Ops API 엔드포인트 및 웹 UI는 높은 대기시간과 간헐적인 오류를 경험했습니다. 중단은 15:30 UTC에서 시작하여 17:10 UTC (총 시간: 1 시간 40 분)에 의해 해결되었습니다. 엔지니어링 팀은 백업 재난화 파이프라인을 통해 즉시 경고하고 지연없이 완화를 시작했습니다. 그러나, 중단은 내부 알림 및 조정이 교차 기능 팀 (고객 지원 및 사용 통신 포함) 사이의 흐름을 영향, 외부 상태 페이지 업데이트 게시 지연으로 결과. ### ROOT 사용 이 사건은 새로 출시 된 구성 변경이 모든 페이지 부하에 중복 다운 스트림 서비스 호출을 만들 때 트리거되었습니다. 트래픽 스파이크를 유발합니다. 9 월 9, 2026 및 9 월 11, 2026 사이에 발표 된 구성 변경은 9 월 14, 2026까지 영향을 위해 적극적으로 모니터링되었습니다. 그러나 미국 근무 시간 동안 교통 스파이크는 예상치 못한 문제 발생. 다운스트림 서비스는 속도 제한 요청을 시작하며 충분한 백오프가 열린 작업자 스레드가 열리지 않고 공격적인 리트리 전략을 시작으로, 풀 배기 및 타임아웃으로 이어지는 중요한 교통 수단입니다. # # # # 이벤트 PLAN 및 다음 단계 이 사건은 inter-service rate limiting를 트리거 한 기능 플래그를 비활성화하여 mitigated. Atlassian 제품에 대한 신뢰할 수있는 액세스를 이해합니다. 우리는 재발을 방지하기 위해 다음 조치를 우선합니다. - ** 서비스 제공** - 내부 서비스 사이의 속도 제한 구성을 검토하고 조정합니다. - 긴 실행 트랜잭션을 방지하기 위해 리트리 전략을 최적화하고 문제가있는 지점에 격리 된 영향을 유지합니다. - ** 건축 고립 ** - 이 핵심 서비스의 아키텍처를 정의하여 다른 거래 사이에 더 고립을 가지므로 하나의 거래 유형의 분해는 관련 중요한 흐름에 영향을 미치지 않습니다. - ** Proactive Alerting를 개선 ** - 스레드 풀 포화에 대한 초기 경고를 추가하고 최종 사용자에 영향을하기 전에 간 서비스 속도 제한 응답을 높였습니다. - 고객 지원 및 사건 통신 경로에 대한 백업 알림 프로세스 구현 우리는이 사건에 영향을 미치는 고객에게 사과합니다. 우리는 플랫폼의 성능과 가용성을 개선하기 위해 즉각적인 조치를 취합니다. 감사합니다, Atlassian 고객 지원

공식 인시던트 업데이트를 자동 번역했습니다.

1. 2026년 8월 27일 오전 10:26 UTCidentified

   2026년 8월 19일, Opsgenie 독립 고객을 위한 Microsoft 팀 알림의 납품에 영향을 미치는 문제점을 발견했습니다. 뿌리 원인은 확인되었습니다 우리의 팀은 우리의 파트너와 협력하여이 문제를 해결합니다. 이 문제는 Jira Service Management 고객에 영향을 미치지 않습니다. 우리는 24 시간에 다음 업데이트를 제공 할 것입니다.

2. 2026년 8월 28일 오전 10:38 UTCidentified

   우리의 팀은 Opsgenie 독립 고객을 위한 마이크로소프트 팀 통보 납품 문제점을 해결하는 우리의 파트너와 함께 일합니다. 우리는 물자 갱신이 있는 경우에 72 시간 또는 더 빨리에 있는 다음 갱신을 제공할 것입니다.

3. 2026년 8월 31일 오전 10:29 UTCidentified

   우리의 팀은 우리의 파트너와 함께, Opsgenie 독립 고객을 위한 마이크로소프트 팀 통보 납품 문제점을 해결하기 위하여 계속합니다. 우리는 24 시간에 다음 업데이트를 제공 할 것입니다, 또는 더 빨리 중요한 업데이트가 있다면.

4. 2026년 9월 2일 오전 10:17 UTCidentified

   우리의 팀은 Opsgenie 독립 고객을 위한 마이크로소프트 팀 통보 납품 문제점을 해결하는 우리의 파트너와 함께 일합니다. 우리는 24 시간에 다음 업데이트를 제공 할 것입니다, 또는 더 빨리 중요한 업데이트가 있다면.

5. 2026년 9월 3일 오전 6:06 UTCresolved

   8 월 19 일, 우리는 Opsgenie 독립 고객을 위한 Microsoft 팀 알림의 납품에 영향을 미치는 문제점을 발견했습니다. 루트 원인은 외부 파트너에 기인하는 운영 오류에 특성화되었으며, 그들은 서비스를 복원하기 위해 정확한 조치를 취했습니다. 문제가 발생하면 통합을 복원하려면 다음 단계를 따르십시오. https://support.atlassian.com/opsgenie/docs/restore-the-opsgenie-microsoft-teams-v2-integration/. 추가 지원이 필요하면 support.atlassian.com/contact를 통해 Atlassian Support에 도달하십시오.

공식 인시던트 업데이트를 자동 번역했습니다.

1. 2026년 7월 21일 오전 9:48 UTCidentified

   우리는 문제를 해결하기 위해 우리의 알림 제공 업체와 함께 일하고 있습니다. 우리는 24 시간에 또 다른 업데이트를 제공하거나 더 많은 정보를 사용할 수있을 때.

2. 2026년 7월 22일 오전 6:59 UTCidentified

   Jira Service Management 및 Opsgenie 고객에 대한 음성 알림의 작은 수는 인도 기반 모바일 번호를 사용하여 실패 할 수 있습니다. 고객은 음성 통화 로그가 실패하고 예상치 못한 호출을받지 못할 수 있습니다. 우리의 제3자 통지 공급자는 음성 통보 납품을 회복하기 위하여 적극적으로 작동되고 우리는 일단 문제점이 해결된 갱신을 제공할 것입니다. 더 많은 지원 또는 정보가 필요하면 지원 팀에 연락 주시기 바랍니다.

3. 2026년 7월 24일 오후 1:07 UTCresolved

   우리의 알림 공급자는 일부 인도 기반 모바일 번호에 음성 통화 실패를 일으키는 문제의 뿌리를 확인했습니다. 우리는이 불편을 이해하고 우리의 공급자가 서비스를 복원하는 동안 당신의 인내력을 평가 할 수 있습니다. 음성 알림에 문제가 있는 경우, 지원팀에 문의하시기 바랍니다.

공식 인시던트 업데이트를 자동 번역했습니다.

1. 2025년 10월 20일 오전 7:56 UTCinvestigating

   We have noticed that Atlassian Cloud services are impacted and our teams are actively investigating the same. We shall keep you informed of the progress every hour.

2. 2025년 10월 20일 오전 8:24 UTCinvestigating

   Atlassian Cloud services are impacted and we are aware that our customers might not be able to create support tickets. Our teams are actively investigating the same. We shall keep you informed of the progress every hour.

3. 2025년 10월 20일 오전 9:24 UTCinvestigating

   We are experiencing an outage due to some issue at the end of our public cloud provider. We are working closely with them to get this resolved or mitigated as quickly as possible. ETA of the same is not know at the moment. We shall continue to share updates every hour, if not sooner.

4. 2025년 10월 20일 오전 9:58 UTCidentified

   We understand that our public cloud provider has identified the cause of the issue. We are starting to see some recovery and is working towards mitigation. We appreciate your patience. We shall continue to share updates every hour, if not sooner.

5. 2025년 10월 20일 오전 11:07 UTCidentified

   We continue to work with our public cloud provider towards mitigating the issue at the earliest. We appreciate your patience. We shall continue to share updates every hour, if not sooner.

6. 2025년 10월 20일 오후 12:06 UTCidentified

   Atlassian team is actively engaged and continues to work with our public cloud provider to mitigate this issue at the earliest. We are starting to see partial operations succeed. We appreciate your patience. We shall continue to share updates every hour, if not sooner.

7. 2025년 10월 20일 오후 1:58 UTCidentified

   Our public cloud provider is working to mitigate this issue quickly. We are seeing some early positive indicators and are continuing to monitor. We appreciate your patience and will continue to provide updates every hour or sooner.

8. 2025년 10월 20일 오후 2:02 UTCidentified

   We understand your pain and mitigating or fixing this issue is of utmost importance. Our public cloud provider is actively working to mitigate this issue on priority. We have been seeing partial operational success. We appreciate your patience and will continue to provide updates every hour or sooner.

9. 2025년 10월 20일 오후 4:36 UTCidentified

   Update - Thank you for your continued patience. We understand the impact this issue is having on your operations and want to assure you that resolving this matter is our highest priority. Our public cloud provider is still actively working to mitigate this issue with urgency and while we do not have a definitive ETA at this time, we remain committed to full resolution and deeply appreciate your patience as we work through this situation. We will be providing hourly updates on this issue.

10. 2025년 10월 20일 오후 5:27 UTCidentified

    Update - We understand the impact this issue is having on your operations and want to assure you that resolving this matter is our highest priority. Our public cloud provider is actively working to mitigate this issue with urgency. While we do not have a definitive ETA at this time, we remain committed to full resolution and deeply appreciate your patience as we work through this situation. We will continue to provide updates every hour or sooner as new information becomes available.

11. 2025년 10월 20일 오후 5:43 UTCidentified

    We are currently aware of an ongoing incident impacting Atlassian Cloud services due to an outage with our public cloud provider, AWS. We understand the impact this issue is having on your operations and want to assure you that resolving this matter is our highest priority and we are closely monitoring the health of AWS services. While we do not have a definitive ETA at this time, we remain committed to full resolution and deeply appreciate your patience as we work through this situation. We will continue to provide updates every hour or sooner as new information becomes available.

12. 2025년 10월 20일 오후 6:38 UTCmonitoring

    There have been no changes since our last update. We will provide our next updated by 9:00PM UTC or sooner as new information becomes available. We are currently aware of an ongoing incident impacting Atlassian Cloud services due to an outage with our public cloud provider, AWS. We understand the impact this issue is having on your operations and want to assure you that resolving this matter is our highest priority and we are closely monitoring the health of AWS services. While we do not have a definitive ETA at this time, we remain committed to full resolution and deeply appreciate your patience as we work through this situation.

13. 2025년 10월 20일 오후 9:26 UTCmonitoring

    Monitoring - We've started seeing continued product experience improvement. While we still have a backlog of event processing, we are seeing improvements in systems operational capabilities across all products. We estimate a significant improvement with the next few hours and will continue to monitor the health of AWS services and the effects on Atlassian customers. We appreciate your continued patience and remain committed to full resolution as we work through this situation. We will post our next update in two hours.

14. 2025년 10월 20일 오후 10:58 UTCmonitoring

    Our teams are continuing to monitor the recovery of systems across Atlassian products. This update is to inform that the Atlassian Support portal is fully operational at this time for customers that wish to contact support.

15. 2025년 10월 21일 오전 12:03 UTCmonitoring

    Our team is now seeing recovery across all impacted Atlassian products. We are continuing to monitor for individual products that may still be processing backlogged items now that services are restored. The Atlassian Support portal is currently still displaying a message directing customers to our temporary support channel. Please note that our support portal is currently fully functional for those attempting to raise requests. We are continuing to look into this alert to remove this message. We will provide further update on our recovery status in one hour.

16. 2025년 10월 21일 오전 1:07 UTCmonitoring

    We continue to see recovery progressing across all impacted products as backlogged items continue to be processed. The Atlassian Support portal is currently displaying a message directing customers to our temporary support channel. Please note that our support portal is currently fully functional for those attempting to raise requests. We are continuing to look into this alert to remove this message. We will provide further update on our recovery status in two hours.

17. 2025년 10월 21일 오전 3:24 UTCmonitoring

    The issue relating to the Atlassian Support portal displaying a message to customers to use our temporary support channel has now been resolved. The Atlassian Support portal is fully functional for any ongoing support issues. With regards to other Atlassian products, we continue to see recovery continuing across all impacted products and our teams are continuing to monitor as the recovery continues. We will provide further update on our recovery status within two hours.

18. 2025년 10월 21일 오전 5:24 UTCresolved

    Our team is now able to see full recovery across the vast majority of Atlassian products. We are aware of some ongoing issues with specific components such as migrations and JSM virtual service agents, and our team is continuing to investigate with urgency. We apologise for the inconvenience that this incident has caused and we will provide further information when the Post Incident Investigation has been completed.

19. 2025년 10월 24일 오후 5:54 UTCpostmortem

    ### Postmortem publish date: Nov 19th, 2025 ### Summary All dates and times below are in UTC unless stated otherwise. Customers utilizing Atlassian products experienced elevated error rates and degraded performance between Oct 20, 2025 06:48 and Oct 21, 2025 04:05. The service disruptions were triggered due to an [AWS DynamoDB outage](https://aws.amazon.com/message/101925/#:~:text=1%3A50%20PM.-,DynamoDB,-Between%2011%3A48) and further affected by subsequent failures in [AWS EC2](https://aws.amazon.com/message/101925/#:~:text=service%20disruption%20event.-,Amazon%20EC2,-Between%2011%3A48) and [AWS Network Load Balancer](https://aws.amazon.com/message/101925/#:~:text=service%20disruption%20event.-,Amazon%20EC2,-Between%2011%3A48) within the us-east-1 region. The incident started at Oct 20, 2025 06:48 and was detected within six minutes by our automated monitoring systems. Our teams worked to restore all core services by Oct 21, 2025 04:05. Final cleanup of backlogged processes and minor issues was completed on Oct 22, 2025. We recognize the critical role our products play in your daily operations, and we offer our sincere apologies for any impact this incident had on your teams. We are taking immediate steps to enhance the reliability and performance of our services, so that you continue to receive the standard of service you have come to trust. ### IMPACT Before examining product-level impacts, it's helpful to understand Atlassian's service topology and internal dependencies. Products such as Jira and Confluence are deployed across multiple AWS regions. The data for each tenant is stored and processed exclusively within its designated host region. This design is intentional and represents the desired operational state, as it limits the impact of any regional outage strictly to tenants in-region, in this case us-east-1. While in-scope application data is pinned to the region selected by the customer, there are times when systems need to call other internal services that may be based in a different region. If a problem occurs in the main region where these services operate, systems are designed to automatically fail over to a backup region, usually within three minutes. However, if unexpected issues arise during this failover, it can take longer to restore services. In rare cases, this could affect customers in more than one region. It’s important to note that all in-scope application data for supported products is pinned according to a customer’s chosen region. **Jira** Between Oct 20, 2025 06:48 and Oct 20, 2025 20:00, customers with tenants hosted in the us-east-1 region experienced increased error rates when accessing core entities such as Issues, Boards, and Backlogs. This disruption was caused by AWS's inability to allocate AWS EC2 instances and elevated errors in AWS Network Load Balancer \(NLB\). During this window, users may also have observed intermittent timeouts, slow page loads, and failures when performing operations like creating or updating issues, loading board views, and executing workflow transitions. Between Oct 20, 2025 08:36 and Oct 20, 2025 09:23, customers across all regions experienced elevated failure rates when attempting to load Jira pages. This disruption was caused by the regional frontend service entering an unhealthy state during this specific time interval. Normally, the frontend service connects to the primary AWS DynamoDB instance located in the us-east-1 to retrieve the most recent configuration data necessary for proper operation. Additionally, the service is designed with a fallback mechanism that references static configuration data in the event that the primary database becomes inaccessible. Unfortunately, a latent bug existed in the local fallback path. When the frontend service nodes restarted, they were unable to load critical operational configuration data from primary or fallback sources, leading to the observed failures experienced by customers. Between Oct 20, 2025 06:48 and Oct 21, 2025 06:30, customers experienced significant delays and missing Jira in-app notifications across all regions. The notification ingestion service, which is hosted exclusively in us-east-1, exhibited an increased failure rate when processing notification messages due to AWS EC2 and NLB issues. This issue resulted in notifications being delayed - and in some cases, not delivered at all - to users worldwide. **Jira Service Management \(JSM\)** JSM was impacted similarly to Jira above, with the same timeframes and for the same reasons. Between Oct 20, 2025 08:36 and Oct 20, 2025 09:23, customers across all regions experienced significantly elevated failure rates when attempting to load JSM pages. This affected all JSM experiences including the Help Centre, Portal, Queues, Work Items, Operations, and Alerts. **Confluence** Between Oct 20, 2025 06:48 and Oct 21, 2025 02:45, customers using Confluence in the us-east-1 region experienced elevated failure rates when performing common operations such as editing pages or adding comments. The primary cause of this service degradation was the system's inability to auto-scale due to AWS EC2 issues to manage peak traffic load effectively. Though the AWS outage ended at Oct 20, 21:09, a subset of customers continued to experience failures as some Confluence web server nodes across multiple clusters remained in an unhealthy state. This was ultimately mitigated by recycling the affected nodes. To protect our systems while AWS recovered, we made a deliberate decision to enable node termination protection. This action successfully preserved our server capacity but, as a trade-off, it extended the time required for a full recovery once AWS services were restored. **Automation** Between Oct 20, 2025 06:55 and Oct 20, 2025 23:59, automation customers whose rules are processed in us-east-1 experienced delays of up to 23 hours in rule execution. During this window, some events triggering rule executions were processed out of order because they arrived later during backlog processing. This caused potential inconsistencies in workflow executions, as rules were run in the order events were received, not when the action causing the event occurred. Additionally, some rule actions failed because they depend on first-party and third-party systems, which were also affected by the AWS outage. Customers can see most of these failures in their audit logs; however, a few updates were not logged due to the nature of the outage. By Oct 21, 2025 5:30, the backlog of rule runs in us-east-1 was cleared. Although most of these delayed rules were successfully handled, there were some additional replays of events to ensure completeness. Our investigation confirmed that a few events may never have triggered their associated rules due to the outage. Between Oct 20, 2025 06:55 and Oct 20, 2025 11:20, all non-us-east-1 regional automation services experienced delays of up to 4 hours in rule execution. This was caused by an upstream service that was unable to deliver events as expected. The delivery service encountered a failure due to a cross-region dependency call to a service hosted in the us-east-1 region. Because of this dependency issue, the delivery service was unable to successfully deliver events throughout this time frame, resulting in customer-defined rules not being executed in a timely manner. **Bitbucket and Pipelines** Between Oct 20, 2025 06:48 and Oct 20, 2025 09:33, Bitbucket experienced intermittent unavailability across core services. During this period, users faced increased error rates and latency when signing in, navigating repositories, and performing essential actions such as creating, updating, or approving pull requests. The primary cause was an AWS DynamoDB outage that impacted downstream services. Between Oct 20, 2025 06:48 and Oct 20, 2025 22:46, numerous Bitbucket Pipeline steps failed to start, stalled mid-execution, or experienced significant queueing delays. Impact varied, with partial recoveries followed by degradation as downstream components re-synchronized. The primary cause was an AWS DynamoDB outage, compounded by instability in AWS EC2 instance availability and AWS Network Load Balancers. Furthermore, Bitbucket Pipelines continued to experience a low but persistent rate of step timeouts and scheduling errors due to AWS bare-metal capacity shortages in select availability zones. Atlassian coordinated with AWS to provision additional bare-metal hosts and addressed a significant backlog of pending pods, successfully restoring services by 01:30 on Oct 21, 2025. **Trello** Between Oct 20, 2025 06:48 and Oct 20, 2025 15:25, users of Trello experienced widespread service degradation and intermittent failures due to upstream AWS issues affecting multiple components, including AWS DynamoDB and subsequent AWS EC2 capacity constraints. During this period, customers reported elevated error rates when loading boards, opening cards, adding comments or attachments. **Login** Between Oct 20, 2025 06:48 and Oct 20, 2025 09:30, a small subset of users experienced failures when attempting to initiate new login sessions using SAML tokens. This resulted in an inability for those users to access Atlassian products during that time period. However, users who already had valid active sessions were not affected by this issue and continued to have uninterrupted access. The issue impacted all regions globally because regional identity services relied on a write replica located in the us-east-1 region to synchronize profile data. When the primary region became unavailable, the failover to a secondary database in another region failed, which delayed recovery. This failover defect has since been addressed. **Statuspage** Between Oct 20, 2025 06:48 and Oct 20, 2025 09:30, Statuspage customers who were not already logged in to the management portal were unable to log in to create or update incident statuses. This impact was restricted only to users who were not already logged in at the time. The root cause was the same as described in the Login section above, and it was resolved by the same remediation steps. ### REMEDIAL ACTION PLAN & NEXT STEPS We have completed the following critical actions designed to help prevent cross-region impact from similar issues: * Resolved the code defect in the fallback option to ensure that Jira Frontend Services in other regions remain unaffected during a region-wide outage. * Fixed the issue that prevented timely failover of the identity service which impacted new login sessions. * Resolved the code defect so that delivery services in unaffected regions remain operational during region-wide outages. Additionally, we are prioritizing the following improvement actions: * Implement mitigation strategies to strengthen resilience against region-wide outages in the notification ingestion service. Although disruptions to our cloud services are sometimes unavoidable during outages of the underlying cloud provider, we continuously evaluate and improve test coverage to strengthen resilience of our cloud services against these issues. We recognize the critical importance of our products to your daily operations and overall productivity, and we extend our sincere apologies for any disruptions this incident may have caused your teams. If you were impacted and require additional details for internal post-incident reviews, please reach out to your Atlassian support representative with affected timeframes and tenant identifiers so we can correlate logs and provide guidance. Thanks, Atlassian Customer Support

1. 2025년 9월 30일 오후 2:07 UTCinvestigating

   We are observing delays in Opsgenie, JSM ops and Compass ops notification flows. No alert has been lost and our team is actively working on it to mitigate the delays. We'll keep you posted with further updates.

2. 2025년 9월 30일 오후 2:09 UTCidentified

   We continue to work on resolving the notification delays for Jira Service Management, Opsgenie, and Compass. We have identified the root cause and expect recovery shortly.

3. 2025년 9월 30일 오후 2:22 UTCmonitoring

   We have identified the root cause of the notification delay for Opsgenie, JSM Ops and Compass Ops and have mitigated the problem. We are now monitoring closely.

4. 2025년 9월 30일 오후 2:32 UTCresolved

   Between 13:10 UTC to 14:00 UTC, we experienced notification delay for Jira Service Management, Opsgenie, and Compass in only EU-region. The issue has been resolved and the service is operating normally.

1. 2025년 9월 10일 오후 3:23 UTCinvestigating

   We identified degraded performance in web experiences and Alert Rest API for some Jira Service Management and Opsgenie Cloud customers in the US Region. The team has taken actions to mitigate the issue and minimize the impact on search functionality.

2. 2025년 9월 10일 오후 3:26 UTCidentified

   We continue to work on resolving the incident for Jira Service Management and Opsgenie. We have identified the root cause and taken actions to mitigate the issue and minimize the impact on search functionality.

3. 2025년 9월 10일 오후 5:56 UTCidentified

   We are continuing to work on a fix for this issue.

4. 2025년 9월 10일 오후 8:10 UTCidentified

   Our team is still actively working to resolve the issue and making progress toward a resolution. Thank you for your patience.

5. 2025년 9월 11일 오전 12:11 UTCidentified

   Our team has been able to identify the root cause of these performance issues and have put a mitigation into place. We are now continuing to see recovery of Jira Service Management operations and Opsgenie services. Existing alerts are continuing to be processed at this time. We will provide further update as soon as possible.

6. 2025년 9월 11일 오전 1:10 UTCidentified

   We are continuing to see recovery of Jira Service Management operations and Opsgenie services. There are a large number of missed alerts to be processed due to the prior service issue and our team is actively investigating methods to help expedite this processing. Alerts should trigger in the meantime but the content of these alerts may not be visible at this time. Our teams are continuing to investigate with urgency. We will provide further updates within the hour.

7. 2025년 9월 11일 오전 2:10 UTCidentified

   While new alerts should now correctly notify users, the content of those alerts for impacted customers will remain empty until missed alerts are fully processed. Our team is continuing to investigate with urgency potential infrastructure options to expedite this process. We will provide further updates within the hour.

8. 2025년 9월 11일 오전 3:10 UTCidentified

   New alerts should still continue to notify users as they are generated, however the alert content will still be missing. Our teams are still actively engaging with infrastructure teams to try and expedite historical processing of messages to fully restore services. We will provide further updates within the hour.

9. 2025년 9월 11일 오전 3:49 UTCidentified

   We have received reports that some customers are experiencing further issues relating to loading pages within Opsgenie and Jira Service Management which may be related to the infrastructure efforts underway to restore services to these products. Please be aware we also have teams actively investigating these issues to ensure as fast a resolution as possible. We want to reiterate that your alerts are still sending notifications correctly, and there is no data loss occurring. However, the underlying issue is affecting the propagation of alert details to notifications, and to the Web and Mobile UI. Viewing schedules may also be impacted at this time while we continue to process the tasks required for recovery. We will provide further updates as soon as possible.

10. 2025년 9월 11일 오전 4:50 UTCidentified

    Our team is continuing to investigate with the highest level of urgency in order to restore Jira Service Management and Opsgenie services. At this time we are prioritising infrastructure to try and restore new and active alerts to be populated with their alert content as soon as possible, while continuing to deliver existing messages as they are ready. We will continue to provide updates as we progress, and will ensure we have an update posted within the hour.

11. 2025년 9월 11일 오전 5:51 UTCidentified

    Our infrastructure teams are working diligently to try and ensure all services are restored to Jira Service Management and Opsgenie as soon as possible. New alerts are continuing to notify users, however we are still prioritising infrastructure to restore the content inside these alerts for Web and Mobile UIs. We will continue providing updates when available and will ensure we have further update within the hour.

12. 2025년 9월 11일 오전 6:51 UTCidentified

    Our teams priority remains to restore full functionality to new alerts to users from Jira Service Management and Opsgenie. Escalation teams are fully engaged on this incident to try and ensure we can get to a resolution as soon as possible. At this time these alerts are still successfully notifying users, but without the intended content within the alerts. We will continue providing specific updates when available and will ensure we have further update within the hour.

13. 2025년 9월 11일 오전 7:50 UTCidentified

    All concerned teams are engaged and are working to restore full functionality to new alerts to users from Jira Service Management and Opsgenie. Escalation teams are also engaged on this incident to try and ensure we can get to a resolution as soon as possible. Currently, the alerts continue to successfully notify users, but without the intended content within the alerts. We will continue providing specific updates when available and will ensure we have further update within the hour.

14. 2025년 9월 11일 오전 8:53 UTCidentified

    Atlassian teams remain engaged with priority to restore full functionality to new alerts to users from Jira Service Management, Opsgenie and Compass. Escalation teams are also engaged on this incident to ensure we can get to a resolution as soon as possible. At this point, the alerts continue to successfully notify users, except that the responder name may be missing. We will continue providing specific updates when available and will ensure we have further update within the hour.

15. 2025년 9월 11일 오전 9:51 UTCidentified

    Atlassian teams continue to remain engaged with priority to restore full functionality to new alerts to users from Jira Service Management, Opsgenie and Compass. Escalation teams also continue to remain engaged on this incident to ensure our focus is not diluted from a resolution as soon as possible. Currently, the alerts continue to successfully notify users, except that the responder name may be missing. We will continue providing specific updates when available and will ensure we have further update within the hour.

16. 2025년 9월 11일 오전 10:59 UTCidentified

    Our teams have are identified some potential resolution steps for this issue and continue to test the same on priority to be able to restore full functionality to users from Jira Service Management, Opsgenie and Compass. Currently, the alerts continue to successfully notify users, except that the responder name may be missing. We will continue providing specific updates when available and will ensure we have further update within the hour.

17. 2025년 9월 11일 오전 11:53 UTCidentified

    Atlassian teams continue to test the identified potential resolution steps for this issue. Our goal is to restore full functionality to users from Jira Service Management, Opsgenie and Compass. Currently, the alerts continue to successfully notify users, except that the responder name may be missing. We will continue providing specific updates when available and will ensure we have further update within the hour.

18. 2025년 9월 11일 오후 1:02 UTCidentified

    We are progressing well with the testing of identified potential mitigation steps for this issue; and we are seeing positive early results. We will update with an ETA and further information in the next update. Alerts continue to successfully notify users, in some cases the responder name may be missing. Our goal continues to restore full functionality to users from Jira Service Management, Opsgenie and Compass. We will continue providing specific updates when available and will ensure we have further update within the hour.

19. 2025년 9월 11일 오후 2:00 UTCidentified

    We have progressed further in preparing our system with a probable fix. Initial testing has been successful. We are in the process of completing further testing and finalising the fix. We expect the fix to be completed in next few hours. Alerts continue to successfully notify users, in some cases the responder name may be missing. Our goal continues to restore full functionality to users from Jira Service Management, Opsgenie and Compass. We will continue providing specific updates when available and will ensure we have further update within the hour.

20. 2025년 9월 11일 오후 3:05 UTCmonitoring

    We have deployed the fix and all operations are back to normal. We will continue to monitor the operations. We will continue providing specific updates when available.

21. 2025년 9월 11일 오후 6:45 UTCresolved

    Between September 10, 2025, 3:19 PM UTC and September 11, 2025, 2:44 PM UTC, there was degraded performance in web experiences and the REST APIs for some Jira Service Management, Opsgenie, and Compass Cloud customers in the US region. We have deployed a fix to mitigate the issue and have verified that the services have recovered. The issue has been resolved and the service is operating normally.

22. 2025년 9월 24일 오전 6:24 UTCpostmortem

    ### **SUMMARY** Between September 10, 2025, at 13:56 UTC and September 11, 2025, at 14:40 UTC, Jira Service Management, Compass Cloud Operations, and Opsgenie customers in the U.S. region experienced degraded performance across web and mobile applications, as well as Alert REST APIs. Certain customers experienced difficulties with alert searches and page loading, particularly on alert-related pages. This incident was initiated by an EBS volume upgrade conducted within the cloud-based managed ElasticSearch clusters. Our automated monitoring systems identified the incident within minutes, and it was resolved after 24 hours and 43 minutes by manually implementing vertical and horizontal scaling measures. ### **IMPACT** Between September 10, 2025, 13:56 UTC and September 11, 2025, 14:40 UTC, some customers in the U.S. region experienced degraded performance in Jira Service Management, Opsgenie, and Compass Cloud. During this time, web and mobile applications, as well as the Alert REST APIs, were impacted. Some customers may have seen issues with alert searches and slow loading of alert pages. During the incident, 22% of customers in the US region experienced failures in web API functionality, 8.6% encountered failures with the REST Alert API, and 1.3% experienced delays in notification delivery. Importantly, there was no loss of data during this event. ### **ROOT CAUSE** The degradation was caused by an EBS volume upgrade in the cloud-based managed ElasticSearch clusters, which necessitated a Blue/Green deployment strategy. One of the ElasticSearch nodes approached its shard size threshold, prompting the upgrade and subsequent deployment. This deployment resulted in elevated latency, increased 4xx HTTP response codes, and timeouts affecting both search and indexing operations. Recovery time exceeded expectations due to the prolonged blue/green deployment. After completion, the ElasticSearch cluster remained unhealthy and did not return to its normal state. Additionally, the switchover to the backup region failed because it was configured similarly in size and setup to the primary cluster. ### **REMEDIAL ACTIONS PLAN & NEXT STEPS** We understand that outages can affect your productivity. We are prioritizing the following improvement actions to help prevent similar incidents in the future: * Increase ElasticSearch cluster capacity and optimize settings to handle peak search loads efficiently. * Strengthen our high availability \(HA\) and failover architecture to ensure rapid and reliable recovery in the event of primary region failures. * Optimize cluster upgrade and shard rebalancing strategies to prevent similar issues. We apologize to customers whose services were impacted during this incident. We are taking steps to improve the platform’s performance and availability.   Thanks, Atlassian Customer Support

1. 2025년 6월 19일 오후 3:21 UTCinvestigating

   We are investigating degraded performance issues in Jira Service Management, Opsgenie, and Compass Cloud customers. We will provide more details within the next hour.

2. 2025년 6월 19일 오후 3:42 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

3. 2025년 6월 19일 오후 4:37 UTCresolved

   This incident has been resolved.

1. 2025년 6월 17일 오후 12:37 UTCinvestigating

   We are investigating cases of degraded performance for some Jira Service Management and Opsgenie Cloud customers. We will provide more details within the next hour.

2. 2025년 6월 17일 오후 12:39 UTCmonitoring

   We have identified the root cause of the performance degraded service and have mitigated the problem. We are now monitoring closely.

3. 2025년 6월 17일 오후 12:47 UTCresolved

   This incident has been resolved, and Jira Service Management and Opsgenie are back in operational mode.

1. 2025년 5월 10일 오전 7:08 UTCidentified

   We identified degraded performance at alert search functionality for some Jira Service Management and Opsgenie Cloud customers due to the infrastructure issue from cloud provider. No impact has been observed at alert critical flows like notification. The team has taken actions to mitigate the issue and minimize the impact to search functionality

2. 2025년 5월 10일 오전 9:49 UTCmonitoring

   The problem is mitigated, and we are now monitoring closely.

3. 2025년 5월 10일 오전 10:51 UTCresolved

   All search functionality is operational without any latency. Thank you for your patience.

1. 2025년 4월 4일 오전 10:05 UTCinvestigating

   We are investigating cases of degraded performance for Alert Schedules experiencing timeouts and slowness for Opsgenie Cloud customers. Requests have been taking more than 30s and some have been timing out. We will provide more details within the next hour.

2. 2025년 4월 4일 오전 10:52 UTCresolved

   This incident has been resolved.

1. 2025년 3월 5일 오전 6:34 UTCinvestigating

   We are aware of an issue where some API calls configured to use api.opsgenie.com/eu are intermittently returning a 404 error with the message 'no Route matched with those values'. API calls to api.eu.opsgenie.com and api.opsgenie.com (without /eu) are not affected. Our team is investigating this issue.

2. 2025년 3월 5일 오전 7:24 UTCresolved

   Issues where some API calls configured to use api.opsgenie.com/eu were intermittently returning a 404 error with the message 'no Route matched with those values' should now be resolved. API calls to api.eu.opsgenie.com and api.opsgenie.com (without /eu) were not affected at this time.

1. 2025년 1월 14일 오후 3:48 UTCinvestigating

   Our engineering team is actively investigating this incident and working to bring the Opsgenie service back up as quickly as possible. Users affected by this incident may notice that Services functionality is slow or completely unavailable for the web page We will update this page as we have additional information.

2. 2025년 1월 14일 오후 3:55 UTCidentified

   Our team has identified the issue with the Services page in Opsgenie and is working to fix it.

3. 2025년 1월 14일 오후 4:47 UTCresolved

   Our engineers have been closely monitoring the platform and are declaring this incident resolved. Thank you for your patience.

1. 2024년 12월 16일 오후 5:18 UTCresolved

   Our team has identified the issue in Schedule API between 15:30 UTC and 17:00 UTC. We saw performance degradation and 5XX errors in response. Faulty deployment has been reverted quickly in the USA region and rapid recovery is seen. We are monitoring the system for a full recovery right now. The Schedule API is up and running again without any data loss.

1. 2024년 11월 6일 오후 7:47 UTCinvestigating

   We've noticed that Opsgenie Web UI is responding slowly or unavailable in US region. Our engineering team is actively investigating this incident and working to bring Opsgenie back up to speed as quickly as possible. We'll keep you posted with further updates on this page.

2. 2024년 11월 6일 오후 7:49 UTCmonitoring

   Our engineering team has implemented fixes. We will continue to monitor all systems. Thank you for your patience.

3. 2024년 11월 6일 오후 7:59 UTCresolved

   Our engineers have been closely monitoring the platform and are declaring this incident resolved. Thank you for your patience.

1. 2024년 9월 24일 오후 7:55 UTCinvestigating

   We've been notified that a number of OEC clients have been failing to create Jira tickets. We're currently investigating the issue.

2. 2024년 9월 24일 오후 7:56 UTCinvestigating

   We are continuing to investigate this issue.

3. 2024년 9월 24일 오후 8:22 UTCidentified

   We've identified a recent change that has broken some endpoint routing configurations and caused OEC endpoint requests to be directed to wrong service. We're reverting that change on production at the moment.

4. 2024년 9월 24일 오후 8:49 UTCmonitoring

   We reverted the faulty routing configuration change and started getting traffic on OEC endpoints

5. 2024년 9월 24일 오후 8:54 UTCresolved

   We've verified that OEC endpoints are back online.

1. 2024년 9월 12일 오전 2:08 UTCinvestigating

   Users attempting to sign up are encountering reCaptcha errors that are preventing a successful signup.

2. 2024년 9월 12일 오전 2:40 UTCmonitoring

   We have identified the root cause and the issue appears to be resolved.

3. 2024년 9월 12일 오전 3:28 UTCresolved

   This issue has been resolved.

1. 2024년 8월 28일 오후 3:01 UTCidentified

   Our team has identified the issue with Opsgenie Web Application / Edit Rotation feature in US, EU and Sydney regions and is working to fix it. Check back soon for another update! Our team is working hard to get the feature up and running again.

2. 2024년 8월 28일 오후 3:02 UTCidentified

   We are continuing to work on a fix for this issue.

3. 2024년 8월 28일 오후 3:17 UTCresolved

   Our engineers have been closely monitoring the platform and are declaring this incident resolved. Thank you for your patience.

1. 2024년 7월 3일 오후 8:51 UTCinvestigating

   We are investigating an issue with that is impacting Atlassian, Atlassian Partners, Atlassian Support, Confluence, Jira Work Management, Jira Service Management, Jira, Opsgenie, Atlassian Developer, Atlassian (deprecated), Trello, Atlassian Bitbucket, Guard, Jira Align, Jira Product Discovery, Atlas, Atlassian Analytics, and Rovo Cloud customers. We will provide more details within the next hour.

2. 2024년 7월 3일 오후 9:45 UTCmonitoring

   We have mitigated the problem and continue looking into the root cause. The outage was between 8:08pm 03/07 UTC - 08:31pm 03/07 UTC We are now monitoring closely.

3. 2024년 7월 4일 오전 12:36 UTCresolved

   Between 03-07-2024 20:08 UTC to 03-07-2024 20:31 UTC, we experienced downtime for Opsgenie. The issue has been resolved and the service is operating normally.

1. 2024년 6월 21일 오전 12:17 UTCinvestigating

   We are investigating an intermittent issue with accessing Atlassian Cloud services that is impacting some Atlassian Cloud customers. We will provide more details once we identify the root cause.

2. 2024년 6월 21일 오전 1:31 UTCmonitoring

   We have identified the root cause of the intermittent errors and have mitigated the problem. We are now monitoring closely.

3. 2024년 6월 21일 오전 2:47 UTCresolved

   Between 2024-06-20 22:04 UTC to 2024-06-20 22:28 UTC, we experienced intermittent issue for users to access the services for some Atlassian Cloud customers. The issue has been resolved and the service is operating normally.

1. 2024년 6월 3일 오후 11:11 UTCidentified

   We are investigating an issue with error responses for some Cloud customers across multiple products. We have identified the root cause and expect recovery shortly.

2. 2024년 6월 4일 오전 12:32 UTCresolved

   Between 22:18 UTC to 22:56 UTC, we experienced errors for multiple Cloud products. The issue has been resolved and the service is operating normally.

3. 2024년 6월 11일 오전 12:56 UTCpostmortem

   ### Summary On June 3rd, between 09:43pm and 10:58 pm UTC, Atlassian customers using multiple product\(s\) were unable to access their services. The event was triggered by a change to the infrastructure API Gateway, which is responsible for routing the traffic to the correct application backends. The incident was detected by the automated monitoring system within five minutes and mitigated by correcting a faulty release feature flag, which put Atlassian systems into a known good state. The first communications were published on the Statuspage at 11:11pm UTC. The total time to resolution was about 75 minutes. ### **IMPACT** The overall impact was between 09:43pm and 10:17pm UTC, with the system initially in a degraded state, followed by a total outage between 10:17pm and 10:58pm UTC. _The Incident caused service disruption to customers in all regions and affected the following products:_ * Jira Software * Jira Service Management * Jira Work Management * Jira Product Discovery * Jira Align * Confluence * Trello * Bitbucket * Opsgenie * Compass ### **ROOT CAUSE** A policy used in the infrastructure API gateway was being updated in production via a feature flag. The combination of an erroneous value entered in a feature flag, and a bug in the code resulted in the API Gateway not processing any traffic. This created a total outage, where all users started receiving 5XX errors for most Atlassian products. Once the problem was identified and the feature flag updated to the correct values, all services started seeing recovery immediately. ### **REMEDIAL ACTIONS PLAN & NEXT STEPS** We know that outages impact your productivity. While we have several testing and preventative processes in place, this specific issue wasn’t identified because the change did not go through our regular release process and instead was incorrectly applied through a feature flag. We are prioritizing the following improvement actions to avoid repeating this type of incident: * Prevent high-risk feature flags from being used in production * Improve the policy changes testing * Enforcing longer soak time for policy changes * Any feature flags should go through progressive rollouts to minimize broad impact * Review the infrastructure feature flags to ensure they all have appropriate defaults * Improve our processes and internal tooling to provide faster communications to our customers We apologize to customers whose services were affected by this incident and are taking immediate steps to address the above gaps. Thanks, Atlassian Customer Support