Opsgenie の障害履歴

1. 2026年9月14日 16:46 UTCinvestigating

   OpsgenieとJira Service Managementに影響するサービスの中断に関するレポートを積極的に調査しています。 更新情報はこちらよりお知らせ致します.

2. 2026年9月14日 17:16 UTCidentified

   私たちは、問題の可能性のある原因を特定し、当社のチームは緩和に従事しています。 今後も追加更新情報を公開してまいります.

3. 2026年9月14日 17:27 UTCmonitoring

   問題は解決され、サービスがすべての影響を受けた顧客のために正常に動作しています。 今後も、安定状態を把握し、より一層の安定性を確保してまいります.

4. 2026年9月14日 17:43 UTCresolved

   2026年9月14日、OpsgenieとJira Service Managementは、混乱を経験し、影響を受けたユーザーは、ユーザーインターフェイスのアラートを表示するための遅延アラート通知と不当性を見た。 問題は解決され、サービスはすべての影響を受けた顧客のために普通作動しています.

5. 2026年9月18日 17:02 UTCpostmortem

   #### 概要 2026年9月14日、15:30～17:10 UTC、Opsgenie、Jira Service Management、Compass を使用したアトラスジアの顧客は、Ops の機能を利用する際のアラート通知や断続的な失敗を受けている。 問題は、コアサービスで長期にわたる取引によってトリガーされ、これにより、プールの排気をスレッド化し、その後の要求が応答しなくなった。 2026年9月9日と2026年9月11日の間にリリースされた構成変更により、長期にわたる取引が行われました。 チームは、システムの変更の影響を積極的に監視し、9月14日、2026 UTCを早期に監視し、異常を観察しました。 しかし、2026年9月14日の米国労働時間におけるトラフィックの増加に伴い、予想外の長期取引が発生しました。 事故は、自動監視システムによって1分以内に検出され、17:10 UTCで9月14日(火)の変更を取り戻すと、フルサービスの安定化が発生しました。 ##IMPACT ## ##IMPACT ##IMPACT ##IMPACT ##IMPACT ##IMPACT ##IMPACT ##IMPACT ##IMPACT ##IMPACT ##IMPACT ##IMPACT ##IMPACT ##IMPACT ##IMPACT ##IMPACT ##IMPACT ##IMPACT ##IMPACT ##IMPACT ##IMPACT ##IMPACT ##IMPACT ##IMPACT ##IMPACT ##IMPACT ##IMPACT 米国地域でホストされている顧客向けに、Opsgenie、Jira Service Management、Compass を中心に、Opsgenie、Jira Service Management、および Compass の機能を主に影響しました。 インシデント中、アラート作成と通知配信のエンドツーエンドフローは、22分の平均遅延を経験しました。 アラートや通知ペイロードがインシデント中にドロップされなかったり、すべてのキュードイベントが正常に処理され、サービスとして回復されたりしました。 さらに、関連するOps APIエンドポイントとWeb UIは、経験豊富な高レイテンシと断続的なエラーをフローします。 破壊は15:30 UTCで始まり、17:10 UTC(合計時間:1時間40分)で解決しました。 エンジニアリングチームは、バックアップ災害管理パイプラインを経由してすぐに警告され、遅延なく緩和を開始しました。 しかしながら、内部の通知と調整に影響を受けた混乱は、外部のStatuspageの更新を出版する遅延が生じた、機能横断チーム(カスタマーサポートとインシデントコミュニケーションを含む)間の流れに影響しました。 ## ROOT CAUSE(ロートキューズ) 新しくリリースされたコンフィギュレーション変更がすべてのページ負荷で冗長なダウンストリームサービスコールを生成したときに、インシデントがトリガーされました。 2026年9月9日と2026年9月11日の間にリリースされたコンフィギュレーションの変更は、2026年9月14日(木)までの衝撃を積極的に監視しました。 しかし、アメリカ就業時の交通事故は予期しない問題を引き起こした。 ダウンストリームサービスは、速度制限の要求を開始しました, 十分なバックオフ保持ワーカースレッドなしで積極的な再試行戦略, プール排気と重要なトラフィックのタイムアウトにつながる. ##ReMEDIALアクションプランとNEXT STEP インシデントは、インターサービスレートの制限をトリガーする機能フラグを無効にすることで緩和されました。 アトラスシアン製品への信頼できるアクセスは、チームにとって不可欠です。 再発防止のため、次の取り組みを推進しています。 - **サービスへのサービス向上** - 内部サービス間のレート制限の設定を見直し、調整します。 - 長期取引を防止し、問題点に分離された影響を保つために再試行の戦略を最大限に活用して下さい。 - **建築分離の改善** - このコアサービスのアーキテクチャを再定義し、異なるトランザクション間でより多くの分離を持っているため、1つのトランザクションタイプで劣化が関連しない重要なフローに影響を与えません。 - **積極的な警告を改善** - エンドユーザーに影響を与える前に、スレッドプールの飽和とサービス間のレート制限の応答を上昇させるための早期警告を追加します。 - カスタマーサポートとインシデントコミュニケーションパスのバックアップアラートプロセスを実装 弊社では、この事件でサービスに影響を受けたお客様を誠意をもってお詫び申し上げます。当社は、プラットフォームのパフォーマンスと可用性を向上させるために、即時に手順を講じています。 お問い合わせ アトラスシアンカスタマーサポート

公式のインシデント更新を自動翻訳しています。

1. 2026年8月27日 10:26 UTCidentified

   2026年8月19日、Opsgenieスタンドアロンの顧客に対するMicrosoft Teams通知の配信に影響を与える問題を特定しました。 根本的な原因が特定され、当社のチームはこの問題を解決するために、パートナーと協力しています。 この問題は、Jira Service Management Operationの顧客には影響しません。 次回の更新は24時間体制で行います.

2. 2026年8月28日 10:38 UTCidentified

   当社のチームは、Microsoft Teamsの通知配信問題を解決するために引き続きパートナーと協力して、Opsgenieのスタンドアロンの顧客を通知します。 資料更新があれば、72時間以内に更新を行います.

3. 2026年8月31日 10:29 UTCidentified

   今後も、Opsgenieのスタンドアロンの顧客に対するMicrosoft Teamsの通知配信問題の解決に向けて、パートナーと協力して取り組んでまいります。 重要なアップデートがある場合は、次の更新を24時間早く行います.

4. 2026年9月2日 10:17 UTCidentified

   当社のチームは、Microsoft Teamsの通知配信問題を解決するために引き続きパートナーと協力して、Opsgenieのスタンドアロンの顧客を通知します。 重要なアップデートがある場合は、次の更新を24時間早く行います.

5. 2026年9月3日 6:06 UTCresolved

   8月19日、当社は、Opsgenieスタンドアローン顧客のMicrosoft Teams通知の配信に影響を与える問題を特定しました。 根本的な原因は、外部のパートナーによって引き起こされる運用上の誤りに起因し、サービスを復元するための是正措置を講じました。 それでも問題が発生した場合は、以下の手順に従って統合を復元する: https://support.atlassian.com/opsgenie/docs/restore-the-opsgenie-microsoft-teams-v2-integration/. さらなるサポートが必要な場合は、support.atlassian.com/contact からアトラスシアンサポートまでお問い合わせ下さい.

公式のインシデント更新を自動翻訳しています。

1. 2026年7月21日 9:48 UTCidentified

   通知プロバイダと連携して課題を解決します。 詳細は24時間以内に更新いたします.

2. 2026年7月22日 6:59 UTCidentified

   インドを拠点とする携帯電話番号を使用して、Jira Service Management および Opsgenie 顧客に対する小さなボイス通知が失敗する場合があります。 顧客が失敗した音声通話ログを見たり、予想されるメッセージングコールを受けられないことがあります。 第3回パーティ通知プロバイダは、音声通知配信の復元に積極的に取り組んでおり、問題が解決したらアップデートを提供します。 さらなるサポートや情報が必要な場合は、サポートチームにご連絡ください.

3. 2026年7月24日 13:07 UTCresolved

   当社の通知プロバイダは、一部のインドベースの携帯電話番号に音声通話障害を引き起こしている問題のルートを特定しました。 提供者がサービスを休止する上で動作する間、この不便は、あなたの忍耐を引き起こし、感謝することがあります。 音声通知に問題が発生した場合は、サポートにお問い合わせください.

公式のインシデント更新を自動翻訳しています。

1. 2025年10月20日 7:56 UTCinvestigating

   We have noticed that Atlassian Cloud services are impacted and our teams are actively investigating the same. We shall keep you informed of the progress every hour.

2. 2025年10月20日 8:24 UTCinvestigating

   Atlassian Cloud services are impacted and we are aware that our customers might not be able to create support tickets. Our teams are actively investigating the same. We shall keep you informed of the progress every hour.

3. 2025年10月20日 9:24 UTCinvestigating

   We are experiencing an outage due to some issue at the end of our public cloud provider. We are working closely with them to get this resolved or mitigated as quickly as possible. ETA of the same is not know at the moment. We shall continue to share updates every hour, if not sooner.

4. 2025年10月20日 9:58 UTCidentified

   We understand that our public cloud provider has identified the cause of the issue. We are starting to see some recovery and is working towards mitigation. We appreciate your patience. We shall continue to share updates every hour, if not sooner.

5. 2025年10月20日 11:07 UTCidentified

   We continue to work with our public cloud provider towards mitigating the issue at the earliest. We appreciate your patience. We shall continue to share updates every hour, if not sooner.

6. 2025年10月20日 12:06 UTCidentified

   Atlassian team is actively engaged and continues to work with our public cloud provider to mitigate this issue at the earliest. We are starting to see partial operations succeed. We appreciate your patience. We shall continue to share updates every hour, if not sooner.

7. 2025年10月20日 13:58 UTCidentified

   Our public cloud provider is working to mitigate this issue quickly. We are seeing some early positive indicators and are continuing to monitor. We appreciate your patience and will continue to provide updates every hour or sooner.

8. 2025年10月20日 14:02 UTCidentified

   We understand your pain and mitigating or fixing this issue is of utmost importance. Our public cloud provider is actively working to mitigate this issue on priority. We have been seeing partial operational success. We appreciate your patience and will continue to provide updates every hour or sooner.

9. 2025年10月20日 16:36 UTCidentified

   Update - Thank you for your continued patience. We understand the impact this issue is having on your operations and want to assure you that resolving this matter is our highest priority. Our public cloud provider is still actively working to mitigate this issue with urgency and while we do not have a definitive ETA at this time, we remain committed to full resolution and deeply appreciate your patience as we work through this situation. We will be providing hourly updates on this issue.

10. 2025年10月20日 17:27 UTCidentified

    Update - We understand the impact this issue is having on your operations and want to assure you that resolving this matter is our highest priority. Our public cloud provider is actively working to mitigate this issue with urgency. While we do not have a definitive ETA at this time, we remain committed to full resolution and deeply appreciate your patience as we work through this situation. We will continue to provide updates every hour or sooner as new information becomes available.

11. 2025年10月20日 17:43 UTCidentified

    We are currently aware of an ongoing incident impacting Atlassian Cloud services due to an outage with our public cloud provider, AWS. We understand the impact this issue is having on your operations and want to assure you that resolving this matter is our highest priority and we are closely monitoring the health of AWS services. While we do not have a definitive ETA at this time, we remain committed to full resolution and deeply appreciate your patience as we work through this situation. We will continue to provide updates every hour or sooner as new information becomes available.

12. 2025年10月20日 18:38 UTCmonitoring

    There have been no changes since our last update. We will provide our next updated by 9:00PM UTC or sooner as new information becomes available. We are currently aware of an ongoing incident impacting Atlassian Cloud services due to an outage with our public cloud provider, AWS. We understand the impact this issue is having on your operations and want to assure you that resolving this matter is our highest priority and we are closely monitoring the health of AWS services. While we do not have a definitive ETA at this time, we remain committed to full resolution and deeply appreciate your patience as we work through this situation.

13. 2025年10月20日 21:26 UTCmonitoring

    Monitoring - We've started seeing continued product experience improvement. While we still have a backlog of event processing, we are seeing improvements in systems operational capabilities across all products. We estimate a significant improvement with the next few hours and will continue to monitor the health of AWS services and the effects on Atlassian customers. We appreciate your continued patience and remain committed to full resolution as we work through this situation. We will post our next update in two hours.

14. 2025年10月20日 22:58 UTCmonitoring

    Our teams are continuing to monitor the recovery of systems across Atlassian products. This update is to inform that the Atlassian Support portal is fully operational at this time for customers that wish to contact support.

15. 2025年10月21日 0:03 UTCmonitoring

    Our team is now seeing recovery across all impacted Atlassian products. We are continuing to monitor for individual products that may still be processing backlogged items now that services are restored. The Atlassian Support portal is currently still displaying a message directing customers to our temporary support channel. Please note that our support portal is currently fully functional for those attempting to raise requests. We are continuing to look into this alert to remove this message. We will provide further update on our recovery status in one hour.

16. 2025年10月21日 1:07 UTCmonitoring

    We continue to see recovery progressing across all impacted products as backlogged items continue to be processed. The Atlassian Support portal is currently displaying a message directing customers to our temporary support channel. Please note that our support portal is currently fully functional for those attempting to raise requests. We are continuing to look into this alert to remove this message. We will provide further update on our recovery status in two hours.

17. 2025年10月21日 3:24 UTCmonitoring

    The issue relating to the Atlassian Support portal displaying a message to customers to use our temporary support channel has now been resolved. The Atlassian Support portal is fully functional for any ongoing support issues. With regards to other Atlassian products, we continue to see recovery continuing across all impacted products and our teams are continuing to monitor as the recovery continues. We will provide further update on our recovery status within two hours.

18. 2025年10月21日 5:24 UTCresolved

    Our team is now able to see full recovery across the vast majority of Atlassian products. We are aware of some ongoing issues with specific components such as migrations and JSM virtual service agents, and our team is continuing to investigate with urgency. We apologise for the inconvenience that this incident has caused and we will provide further information when the Post Incident Investigation has been completed.

19. 2025年10月24日 17:54 UTCpostmortem

    ### Postmortem publish date: Nov 19th, 2025 ### Summary All dates and times below are in UTC unless stated otherwise. Customers utilizing Atlassian products experienced elevated error rates and degraded performance between Oct 20, 2025 06:48 and Oct 21, 2025 04:05. The service disruptions were triggered due to an [AWS DynamoDB outage](https://aws.amazon.com/message/101925/#:~:text=1%3A50%20PM.-,DynamoDB,-Between%2011%3A48) and further affected by subsequent failures in [AWS EC2](https://aws.amazon.com/message/101925/#:~:text=service%20disruption%20event.-,Amazon%20EC2,-Between%2011%3A48) and [AWS Network Load Balancer](https://aws.amazon.com/message/101925/#:~:text=service%20disruption%20event.-,Amazon%20EC2,-Between%2011%3A48) within the us-east-1 region. The incident started at Oct 20, 2025 06:48 and was detected within six minutes by our automated monitoring systems. Our teams worked to restore all core services by Oct 21, 2025 04:05. Final cleanup of backlogged processes and minor issues was completed on Oct 22, 2025. We recognize the critical role our products play in your daily operations, and we offer our sincere apologies for any impact this incident had on your teams. We are taking immediate steps to enhance the reliability and performance of our services, so that you continue to receive the standard of service you have come to trust. ### IMPACT Before examining product-level impacts, it's helpful to understand Atlassian's service topology and internal dependencies. Products such as Jira and Confluence are deployed across multiple AWS regions. The data for each tenant is stored and processed exclusively within its designated host region. This design is intentional and represents the desired operational state, as it limits the impact of any regional outage strictly to tenants in-region, in this case us-east-1. While in-scope application data is pinned to the region selected by the customer, there are times when systems need to call other internal services that may be based in a different region. If a problem occurs in the main region where these services operate, systems are designed to automatically fail over to a backup region, usually within three minutes. However, if unexpected issues arise during this failover, it can take longer to restore services. In rare cases, this could affect customers in more than one region. It’s important to note that all in-scope application data for supported products is pinned according to a customer’s chosen region. **Jira** Between Oct 20, 2025 06:48 and Oct 20, 2025 20:00, customers with tenants hosted in the us-east-1 region experienced increased error rates when accessing core entities such as Issues, Boards, and Backlogs. This disruption was caused by AWS's inability to allocate AWS EC2 instances and elevated errors in AWS Network Load Balancer \(NLB\). During this window, users may also have observed intermittent timeouts, slow page loads, and failures when performing operations like creating or updating issues, loading board views, and executing workflow transitions. Between Oct 20, 2025 08:36 and Oct 20, 2025 09:23, customers across all regions experienced elevated failure rates when attempting to load Jira pages. This disruption was caused by the regional frontend service entering an unhealthy state during this specific time interval. Normally, the frontend service connects to the primary AWS DynamoDB instance located in the us-east-1 to retrieve the most recent configuration data necessary for proper operation. Additionally, the service is designed with a fallback mechanism that references static configuration data in the event that the primary database becomes inaccessible. Unfortunately, a latent bug existed in the local fallback path. When the frontend service nodes restarted, they were unable to load critical operational configuration data from primary or fallback sources, leading to the observed failures experienced by customers. Between Oct 20, 2025 06:48 and Oct 21, 2025 06:30, customers experienced significant delays and missing Jira in-app notifications across all regions. The notification ingestion service, which is hosted exclusively in us-east-1, exhibited an increased failure rate when processing notification messages due to AWS EC2 and NLB issues. This issue resulted in notifications being delayed - and in some cases, not delivered at all - to users worldwide. **Jira Service Management \(JSM\)** JSM was impacted similarly to Jira above, with the same timeframes and for the same reasons. Between Oct 20, 2025 08:36 and Oct 20, 2025 09:23, customers across all regions experienced significantly elevated failure rates when attempting to load JSM pages. This affected all JSM experiences including the Help Centre, Portal, Queues, Work Items, Operations, and Alerts. **Confluence** Between Oct 20, 2025 06:48 and Oct 21, 2025 02:45, customers using Confluence in the us-east-1 region experienced elevated failure rates when performing common operations such as editing pages or adding comments. The primary cause of this service degradation was the system's inability to auto-scale due to AWS EC2 issues to manage peak traffic load effectively. Though the AWS outage ended at Oct 20, 21:09, a subset of customers continued to experience failures as some Confluence web server nodes across multiple clusters remained in an unhealthy state. This was ultimately mitigated by recycling the affected nodes. To protect our systems while AWS recovered, we made a deliberate decision to enable node termination protection. This action successfully preserved our server capacity but, as a trade-off, it extended the time required for a full recovery once AWS services were restored. **Automation** Between Oct 20, 2025 06:55 and Oct 20, 2025 23:59, automation customers whose rules are processed in us-east-1 experienced delays of up to 23 hours in rule execution. During this window, some events triggering rule executions were processed out of order because they arrived later during backlog processing. This caused potential inconsistencies in workflow executions, as rules were run in the order events were received, not when the action causing the event occurred. Additionally, some rule actions failed because they depend on first-party and third-party systems, which were also affected by the AWS outage. Customers can see most of these failures in their audit logs; however, a few updates were not logged due to the nature of the outage. By Oct 21, 2025 5:30, the backlog of rule runs in us-east-1 was cleared. Although most of these delayed rules were successfully handled, there were some additional replays of events to ensure completeness. Our investigation confirmed that a few events may never have triggered their associated rules due to the outage. Between Oct 20, 2025 06:55 and Oct 20, 2025 11:20, all non-us-east-1 regional automation services experienced delays of up to 4 hours in rule execution. This was caused by an upstream service that was unable to deliver events as expected. The delivery service encountered a failure due to a cross-region dependency call to a service hosted in the us-east-1 region. Because of this dependency issue, the delivery service was unable to successfully deliver events throughout this time frame, resulting in customer-defined rules not being executed in a timely manner. **Bitbucket and Pipelines** Between Oct 20, 2025 06:48 and Oct 20, 2025 09:33, Bitbucket experienced intermittent unavailability across core services. During this period, users faced increased error rates and latency when signing in, navigating repositories, and performing essential actions such as creating, updating, or approving pull requests. The primary cause was an AWS DynamoDB outage that impacted downstream services. Between Oct 20, 2025 06:48 and Oct 20, 2025 22:46, numerous Bitbucket Pipeline steps failed to start, stalled mid-execution, or experienced significant queueing delays. Impact varied, with partial recoveries followed by degradation as downstream components re-synchronized. The primary cause was an AWS DynamoDB outage, compounded by instability in AWS EC2 instance availability and AWS Network Load Balancers. Furthermore, Bitbucket Pipelines continued to experience a low but persistent rate of step timeouts and scheduling errors due to AWS bare-metal capacity shortages in select availability zones. Atlassian coordinated with AWS to provision additional bare-metal hosts and addressed a significant backlog of pending pods, successfully restoring services by 01:30 on Oct 21, 2025. **Trello** Between Oct 20, 2025 06:48 and Oct 20, 2025 15:25, users of Trello experienced widespread service degradation and intermittent failures due to upstream AWS issues affecting multiple components, including AWS DynamoDB and subsequent AWS EC2 capacity constraints. During this period, customers reported elevated error rates when loading boards, opening cards, adding comments or attachments. **Login** Between Oct 20, 2025 06:48 and Oct 20, 2025 09:30, a small subset of users experienced failures when attempting to initiate new login sessions using SAML tokens. This resulted in an inability for those users to access Atlassian products during that time period. However, users who already had valid active sessions were not affected by this issue and continued to have uninterrupted access. The issue impacted all regions globally because regional identity services relied on a write replica located in the us-east-1 region to synchronize profile data. When the primary region became unavailable, the failover to a secondary database in another region failed, which delayed recovery. This failover defect has since been addressed. **Statuspage** Between Oct 20, 2025 06:48 and Oct 20, 2025 09:30, Statuspage customers who were not already logged in to the management portal were unable to log in to create or update incident statuses. This impact was restricted only to users who were not already logged in at the time. The root cause was the same as described in the Login section above, and it was resolved by the same remediation steps. ### REMEDIAL ACTION PLAN & NEXT STEPS We have completed the following critical actions designed to help prevent cross-region impact from similar issues: * Resolved the code defect in the fallback option to ensure that Jira Frontend Services in other regions remain unaffected during a region-wide outage. * Fixed the issue that prevented timely failover of the identity service which impacted new login sessions. * Resolved the code defect so that delivery services in unaffected regions remain operational during region-wide outages. Additionally, we are prioritizing the following improvement actions: * Implement mitigation strategies to strengthen resilience against region-wide outages in the notification ingestion service. Although disruptions to our cloud services are sometimes unavoidable during outages of the underlying cloud provider, we continuously evaluate and improve test coverage to strengthen resilience of our cloud services against these issues. We recognize the critical importance of our products to your daily operations and overall productivity, and we extend our sincere apologies for any disruptions this incident may have caused your teams. If you were impacted and require additional details for internal post-incident reviews, please reach out to your Atlassian support representative with affected timeframes and tenant identifiers so we can correlate logs and provide guidance. Thanks, Atlassian Customer Support

1. 2025年9月30日 14:07 UTCinvestigating

   We are observing delays in Opsgenie, JSM ops and Compass ops notification flows. No alert has been lost and our team is actively working on it to mitigate the delays. We'll keep you posted with further updates.

2. 2025年9月30日 14:09 UTCidentified

   We continue to work on resolving the notification delays for Jira Service Management, Opsgenie, and Compass. We have identified the root cause and expect recovery shortly.

3. 2025年9月30日 14:22 UTCmonitoring

   We have identified the root cause of the notification delay for Opsgenie, JSM Ops and Compass Ops and have mitigated the problem. We are now monitoring closely.

4. 2025年9月30日 14:32 UTCresolved

   Between 13:10 UTC to 14:00 UTC, we experienced notification delay for Jira Service Management, Opsgenie, and Compass in only EU-region. The issue has been resolved and the service is operating normally.

1. 2025年9月10日 15:23 UTCinvestigating

   We identified degraded performance in web experiences and Alert Rest API for some Jira Service Management and Opsgenie Cloud customers in the US Region. The team has taken actions to mitigate the issue and minimize the impact on search functionality.

2. 2025年9月10日 15:26 UTCidentified

   We continue to work on resolving the incident for Jira Service Management and Opsgenie. We have identified the root cause and taken actions to mitigate the issue and minimize the impact on search functionality.

3. 2025年9月10日 17:56 UTCidentified

   We are continuing to work on a fix for this issue.

4. 2025年9月10日 20:10 UTCidentified

   Our team is still actively working to resolve the issue and making progress toward a resolution. Thank you for your patience.

5. 2025年9月11日 0:11 UTCidentified

   Our team has been able to identify the root cause of these performance issues and have put a mitigation into place. We are now continuing to see recovery of Jira Service Management operations and Opsgenie services. Existing alerts are continuing to be processed at this time. We will provide further update as soon as possible.

6. 2025年9月11日 1:10 UTCidentified

   We are continuing to see recovery of Jira Service Management operations and Opsgenie services. There are a large number of missed alerts to be processed due to the prior service issue and our team is actively investigating methods to help expedite this processing. Alerts should trigger in the meantime but the content of these alerts may not be visible at this time. Our teams are continuing to investigate with urgency. We will provide further updates within the hour.

7. 2025年9月11日 2:10 UTCidentified

   While new alerts should now correctly notify users, the content of those alerts for impacted customers will remain empty until missed alerts are fully processed. Our team is continuing to investigate with urgency potential infrastructure options to expedite this process. We will provide further updates within the hour.

8. 2025年9月11日 3:10 UTCidentified

   New alerts should still continue to notify users as they are generated, however the alert content will still be missing. Our teams are still actively engaging with infrastructure teams to try and expedite historical processing of messages to fully restore services. We will provide further updates within the hour.

9. 2025年9月11日 3:49 UTCidentified

   We have received reports that some customers are experiencing further issues relating to loading pages within Opsgenie and Jira Service Management which may be related to the infrastructure efforts underway to restore services to these products. Please be aware we also have teams actively investigating these issues to ensure as fast a resolution as possible. We want to reiterate that your alerts are still sending notifications correctly, and there is no data loss occurring. However, the underlying issue is affecting the propagation of alert details to notifications, and to the Web and Mobile UI. Viewing schedules may also be impacted at this time while we continue to process the tasks required for recovery. We will provide further updates as soon as possible.

10. 2025年9月11日 4:50 UTCidentified

    Our team is continuing to investigate with the highest level of urgency in order to restore Jira Service Management and Opsgenie services. At this time we are prioritising infrastructure to try and restore new and active alerts to be populated with their alert content as soon as possible, while continuing to deliver existing messages as they are ready. We will continue to provide updates as we progress, and will ensure we have an update posted within the hour.

11. 2025年9月11日 5:51 UTCidentified

    Our infrastructure teams are working diligently to try and ensure all services are restored to Jira Service Management and Opsgenie as soon as possible. New alerts are continuing to notify users, however we are still prioritising infrastructure to restore the content inside these alerts for Web and Mobile UIs. We will continue providing updates when available and will ensure we have further update within the hour.

12. 2025年9月11日 6:51 UTCidentified

    Our teams priority remains to restore full functionality to new alerts to users from Jira Service Management and Opsgenie. Escalation teams are fully engaged on this incident to try and ensure we can get to a resolution as soon as possible. At this time these alerts are still successfully notifying users, but without the intended content within the alerts. We will continue providing specific updates when available and will ensure we have further update within the hour.

13. 2025年9月11日 7:50 UTCidentified

    All concerned teams are engaged and are working to restore full functionality to new alerts to users from Jira Service Management and Opsgenie. Escalation teams are also engaged on this incident to try and ensure we can get to a resolution as soon as possible. Currently, the alerts continue to successfully notify users, but without the intended content within the alerts. We will continue providing specific updates when available and will ensure we have further update within the hour.

14. 2025年9月11日 8:53 UTCidentified

    Atlassian teams remain engaged with priority to restore full functionality to new alerts to users from Jira Service Management, Opsgenie and Compass. Escalation teams are also engaged on this incident to ensure we can get to a resolution as soon as possible. At this point, the alerts continue to successfully notify users, except that the responder name may be missing. We will continue providing specific updates when available and will ensure we have further update within the hour.

15. 2025年9月11日 9:51 UTCidentified

    Atlassian teams continue to remain engaged with priority to restore full functionality to new alerts to users from Jira Service Management, Opsgenie and Compass. Escalation teams also continue to remain engaged on this incident to ensure our focus is not diluted from a resolution as soon as possible. Currently, the alerts continue to successfully notify users, except that the responder name may be missing. We will continue providing specific updates when available and will ensure we have further update within the hour.

16. 2025年9月11日 10:59 UTCidentified

    Our teams have are identified some potential resolution steps for this issue and continue to test the same on priority to be able to restore full functionality to users from Jira Service Management, Opsgenie and Compass. Currently, the alerts continue to successfully notify users, except that the responder name may be missing. We will continue providing specific updates when available and will ensure we have further update within the hour.

17. 2025年9月11日 11:53 UTCidentified

    Atlassian teams continue to test the identified potential resolution steps for this issue. Our goal is to restore full functionality to users from Jira Service Management, Opsgenie and Compass. Currently, the alerts continue to successfully notify users, except that the responder name may be missing. We will continue providing specific updates when available and will ensure we have further update within the hour.

18. 2025年9月11日 13:02 UTCidentified

    We are progressing well with the testing of identified potential mitigation steps for this issue; and we are seeing positive early results. We will update with an ETA and further information in the next update. Alerts continue to successfully notify users, in some cases the responder name may be missing. Our goal continues to restore full functionality to users from Jira Service Management, Opsgenie and Compass. We will continue providing specific updates when available and will ensure we have further update within the hour.

19. 2025年9月11日 14:00 UTCidentified

    We have progressed further in preparing our system with a probable fix. Initial testing has been successful. We are in the process of completing further testing and finalising the fix. We expect the fix to be completed in next few hours. Alerts continue to successfully notify users, in some cases the responder name may be missing. Our goal continues to restore full functionality to users from Jira Service Management, Opsgenie and Compass. We will continue providing specific updates when available and will ensure we have further update within the hour.

20. 2025年9月11日 15:05 UTCmonitoring

    We have deployed the fix and all operations are back to normal. We will continue to monitor the operations. We will continue providing specific updates when available.

21. 2025年9月11日 18:45 UTCresolved

    Between September 10, 2025, 3:19 PM UTC and September 11, 2025, 2:44 PM UTC, there was degraded performance in web experiences and the REST APIs for some Jira Service Management, Opsgenie, and Compass Cloud customers in the US region. We have deployed a fix to mitigate the issue and have verified that the services have recovered. The issue has been resolved and the service is operating normally.

22. 2025年9月24日 6:24 UTCpostmortem

    ### **SUMMARY** Between September 10, 2025, at 13:56 UTC and September 11, 2025, at 14:40 UTC, Jira Service Management, Compass Cloud Operations, and Opsgenie customers in the U.S. region experienced degraded performance across web and mobile applications, as well as Alert REST APIs. Certain customers experienced difficulties with alert searches and page loading, particularly on alert-related pages. This incident was initiated by an EBS volume upgrade conducted within the cloud-based managed ElasticSearch clusters. Our automated monitoring systems identified the incident within minutes, and it was resolved after 24 hours and 43 minutes by manually implementing vertical and horizontal scaling measures. ### **IMPACT** Between September 10, 2025, 13:56 UTC and September 11, 2025, 14:40 UTC, some customers in the U.S. region experienced degraded performance in Jira Service Management, Opsgenie, and Compass Cloud. During this time, web and mobile applications, as well as the Alert REST APIs, were impacted. Some customers may have seen issues with alert searches and slow loading of alert pages. During the incident, 22% of customers in the US region experienced failures in web API functionality, 8.6% encountered failures with the REST Alert API, and 1.3% experienced delays in notification delivery. Importantly, there was no loss of data during this event. ### **ROOT CAUSE** The degradation was caused by an EBS volume upgrade in the cloud-based managed ElasticSearch clusters, which necessitated a Blue/Green deployment strategy. One of the ElasticSearch nodes approached its shard size threshold, prompting the upgrade and subsequent deployment. This deployment resulted in elevated latency, increased 4xx HTTP response codes, and timeouts affecting both search and indexing operations. Recovery time exceeded expectations due to the prolonged blue/green deployment. After completion, the ElasticSearch cluster remained unhealthy and did not return to its normal state. Additionally, the switchover to the backup region failed because it was configured similarly in size and setup to the primary cluster. ### **REMEDIAL ACTIONS PLAN & NEXT STEPS** We understand that outages can affect your productivity. We are prioritizing the following improvement actions to help prevent similar incidents in the future: * Increase ElasticSearch cluster capacity and optimize settings to handle peak search loads efficiently. * Strengthen our high availability \(HA\) and failover architecture to ensure rapid and reliable recovery in the event of primary region failures. * Optimize cluster upgrade and shard rebalancing strategies to prevent similar issues. We apologize to customers whose services were impacted during this incident. We are taking steps to improve the platform’s performance and availability.   Thanks, Atlassian Customer Support

1. 2025年6月19日 15:21 UTCinvestigating

   We are investigating degraded performance issues in Jira Service Management, Opsgenie, and Compass Cloud customers. We will provide more details within the next hour.

2. 2025年6月19日 15:42 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

3. 2025年6月19日 16:37 UTCresolved

   This incident has been resolved.

1. 2025年6月17日 12:37 UTCinvestigating

   We are investigating cases of degraded performance for some Jira Service Management and Opsgenie Cloud customers. We will provide more details within the next hour.

2. 2025年6月17日 12:39 UTCmonitoring

   We have identified the root cause of the performance degraded service and have mitigated the problem. We are now monitoring closely.

3. 2025年6月17日 12:47 UTCresolved

   This incident has been resolved, and Jira Service Management and Opsgenie are back in operational mode.

1. 2025年5月10日 7:08 UTCidentified

   We identified degraded performance at alert search functionality for some Jira Service Management and Opsgenie Cloud customers due to the infrastructure issue from cloud provider. No impact has been observed at alert critical flows like notification. The team has taken actions to mitigate the issue and minimize the impact to search functionality

2. 2025年5月10日 9:49 UTCmonitoring

   The problem is mitigated, and we are now monitoring closely.

3. 2025年5月10日 10:51 UTCresolved

   All search functionality is operational without any latency. Thank you for your patience.

1. 2025年4月4日 10:05 UTCinvestigating

   We are investigating cases of degraded performance for Alert Schedules experiencing timeouts and slowness for Opsgenie Cloud customers. Requests have been taking more than 30s and some have been timing out. We will provide more details within the next hour.

2. 2025年4月4日 10:52 UTCresolved

   This incident has been resolved.

1. 2025年3月5日 6:34 UTCinvestigating

   We are aware of an issue where some API calls configured to use api.opsgenie.com/eu are intermittently returning a 404 error with the message 'no Route matched with those values'. API calls to api.eu.opsgenie.com and api.opsgenie.com (without /eu) are not affected. Our team is investigating this issue.

2. 2025年3月5日 7:24 UTCresolved

   Issues where some API calls configured to use api.opsgenie.com/eu were intermittently returning a 404 error with the message 'no Route matched with those values' should now be resolved. API calls to api.eu.opsgenie.com and api.opsgenie.com (without /eu) were not affected at this time.

1. 2025年1月14日 15:48 UTCinvestigating

   Our engineering team is actively investigating this incident and working to bring the Opsgenie service back up as quickly as possible. Users affected by this incident may notice that Services functionality is slow or completely unavailable for the web page We will update this page as we have additional information.

2. 2025年1月14日 15:55 UTCidentified

   Our team has identified the issue with the Services page in Opsgenie and is working to fix it.

3. 2025年1月14日 16:47 UTCresolved

   Our engineers have been closely monitoring the platform and are declaring this incident resolved. Thank you for your patience.

1. 2024年12月16日 17:18 UTCresolved

   Our team has identified the issue in Schedule API between 15:30 UTC and 17:00 UTC. We saw performance degradation and 5XX errors in response. Faulty deployment has been reverted quickly in the USA region and rapid recovery is seen. We are monitoring the system for a full recovery right now. The Schedule API is up and running again without any data loss.

1. 2024年11月6日 19:47 UTCinvestigating

   We've noticed that Opsgenie Web UI is responding slowly or unavailable in US region. Our engineering team is actively investigating this incident and working to bring Opsgenie back up to speed as quickly as possible. We'll keep you posted with further updates on this page.

2. 2024年11月6日 19:49 UTCmonitoring

   Our engineering team has implemented fixes. We will continue to monitor all systems. Thank you for your patience.

3. 2024年11月6日 19:59 UTCresolved

   Our engineers have been closely monitoring the platform and are declaring this incident resolved. Thank you for your patience.

1. 2024年9月24日 19:55 UTCinvestigating

   We've been notified that a number of OEC clients have been failing to create Jira tickets. We're currently investigating the issue.

2. 2024年9月24日 19:56 UTCinvestigating

   We are continuing to investigate this issue.

3. 2024年9月24日 20:22 UTCidentified

   We've identified a recent change that has broken some endpoint routing configurations and caused OEC endpoint requests to be directed to wrong service. We're reverting that change on production at the moment.

4. 2024年9月24日 20:49 UTCmonitoring

   We reverted the faulty routing configuration change and started getting traffic on OEC endpoints

5. 2024年9月24日 20:54 UTCresolved

   We've verified that OEC endpoints are back online.

1. 2024年9月12日 2:08 UTCinvestigating

   Users attempting to sign up are encountering reCaptcha errors that are preventing a successful signup.

2. 2024年9月12日 2:40 UTCmonitoring

   We have identified the root cause and the issue appears to be resolved.

3. 2024年9月12日 3:28 UTCresolved

   This issue has been resolved.

1. 2024年8月28日 15:01 UTCidentified

   Our team has identified the issue with Opsgenie Web Application / Edit Rotation feature in US, EU and Sydney regions and is working to fix it. Check back soon for another update! Our team is working hard to get the feature up and running again.

2. 2024年8月28日 15:02 UTCidentified

   We are continuing to work on a fix for this issue.

3. 2024年8月28日 15:17 UTCresolved

   Our engineers have been closely monitoring the platform and are declaring this incident resolved. Thank you for your patience.

1. 2024年7月3日 20:51 UTCinvestigating

   We are investigating an issue with that is impacting Atlassian, Atlassian Partners, Atlassian Support, Confluence, Jira Work Management, Jira Service Management, Jira, Opsgenie, Atlassian Developer, Atlassian (deprecated), Trello, Atlassian Bitbucket, Guard, Jira Align, Jira Product Discovery, Atlas, Atlassian Analytics, and Rovo Cloud customers. We will provide more details within the next hour.

2. 2024年7月3日 21:45 UTCmonitoring

   We have mitigated the problem and continue looking into the root cause. The outage was between 8:08pm 03/07 UTC - 08:31pm 03/07 UTC We are now monitoring closely.

3. 2024年7月4日 0:36 UTCresolved

   Between 03-07-2024 20:08 UTC to 03-07-2024 20:31 UTC, we experienced downtime for Opsgenie. The issue has been resolved and the service is operating normally.

1. 2024年6月21日 0:17 UTCinvestigating

   We are investigating an intermittent issue with accessing Atlassian Cloud services that is impacting some Atlassian Cloud customers. We will provide more details once we identify the root cause.

2. 2024年6月21日 1:31 UTCmonitoring

   We have identified the root cause of the intermittent errors and have mitigated the problem. We are now monitoring closely.

3. 2024年6月21日 2:47 UTCresolved

   Between 2024-06-20 22:04 UTC to 2024-06-20 22:28 UTC, we experienced intermittent issue for users to access the services for some Atlassian Cloud customers. The issue has been resolved and the service is operating normally.

1. 2024年6月3日 23:11 UTCidentified

   We are investigating an issue with error responses for some Cloud customers across multiple products. We have identified the root cause and expect recovery shortly.

2. 2024年6月4日 0:32 UTCresolved

   Between 22:18 UTC to 22:56 UTC, we experienced errors for multiple Cloud products. The issue has been resolved and the service is operating normally.

3. 2024年6月11日 0:56 UTCpostmortem

   ### Summary On June 3rd, between 09:43pm and 10:58 pm UTC, Atlassian customers using multiple product\(s\) were unable to access their services. The event was triggered by a change to the infrastructure API Gateway, which is responsible for routing the traffic to the correct application backends. The incident was detected by the automated monitoring system within five minutes and mitigated by correcting a faulty release feature flag, which put Atlassian systems into a known good state. The first communications were published on the Statuspage at 11:11pm UTC. The total time to resolution was about 75 minutes. ### **IMPACT** The overall impact was between 09:43pm and 10:17pm UTC, with the system initially in a degraded state, followed by a total outage between 10:17pm and 10:58pm UTC. _The Incident caused service disruption to customers in all regions and affected the following products:_ * Jira Software * Jira Service Management * Jira Work Management * Jira Product Discovery * Jira Align * Confluence * Trello * Bitbucket * Opsgenie * Compass ### **ROOT CAUSE** A policy used in the infrastructure API gateway was being updated in production via a feature flag. The combination of an erroneous value entered in a feature flag, and a bug in the code resulted in the API Gateway not processing any traffic. This created a total outage, where all users started receiving 5XX errors for most Atlassian products. Once the problem was identified and the feature flag updated to the correct values, all services started seeing recovery immediately. ### **REMEDIAL ACTIONS PLAN & NEXT STEPS** We know that outages impact your productivity. While we have several testing and preventative processes in place, this specific issue wasn’t identified because the change did not go through our regular release process and instead was incorrectly applied through a feature flag. We are prioritizing the following improvement actions to avoid repeating this type of incident: * Prevent high-risk feature flags from being used in production * Improve the policy changes testing * Enforcing longer soak time for policy changes * Any feature flags should go through progressive rollouts to minimize broad impact * Review the infrastructure feature flags to ensure they all have appropriate defaults * Improve our processes and internal tooling to provide faster communications to our customers We apologize to customers whose services were affected by this incident and are taking immediate steps to address the above gaps. Thanks, Atlassian Customer Support