Loom 中断历史

1. 2026年5月8日 UTC 03:00investigating

   我们正面临多种Atlas产品的问题。 我们的小组正在进一步调查,并将在一小时内分享更多的最新情况.

2. 2026年5月8日 UTC 04:10identified

   我们已确认,问题的根源与我国公共云层提供者的基础设施中断有关。 我们正在同它们密切合作来缓解这一问题。 我们将在获得这些资料后提供进一步的最新资料.

3. 2026年5月8日 UTC 04:10identified

   我们的小组继续努力减轻我们公共云层提供者的基础设施停用。 如有进一步的最新资料,我们将予以提供.

4. 2026年5月8日 UTC 04:10identified

   我们正在继续与我们的公共云提供方合作来缓解这一问题。 我们开始看到美国东部以外地区的复苏,然而,全球用户可能仍然面临具有某些产品特征的问题。 这些都列在每个产品页的底部.

5. 2026年5月8日 UTC 05:53monitoring

   影响同步事件处理的公共基础设施的根本问题已经得到缓解,所有受影响的服务正在恢复。 我们现在正在努力清理排队事件的积压,这意味着一些行动(如通知、自动化触发和数据同步)可能处于退化状态。 我们将继续在清理积压后进行监测并提供最新情况.

6. 2026年5月8日 UTC 08:03monitoring

   随着服务的恢复,我们继续监测局势。 我们目前正在清理排队活动的积压。 我们将在大约一个小时后提供进一步的最新情况.

7. 2026年5月8日 UTC 11:12monitoring

   我们的服务现已全面运作。 我们继续回顾事件中错过的任何事件,并正在取得良好进展。 重播完成后,我们将进一步更新。 如果你遇到问题 请联系我们的支援小组 我们为干扰道歉,并感谢您的耐心.

8. 2026年5月8日 UTC 19:45resolved

   2026年5月8日,一些使用Atlassian产品的客户出现了较高的出错率和退化的性能. 这个问题现已解决,这项服务对所有受影响的客户正常运行.

9. 2026年5月20日 UTC 06:57postmortem

   以下所有日期和时间均在协调世界时,除非另有说明. 页:1 2026年5月8日 00:22到06:08,我们一个托管商在正东方的特定可用区遭遇了一起重大事件,导致Atlassian客户的性能下降,背景操作和自动化执行出现延误. 事件发生于2026年5月8日00:22分,由自动监测系统在4分钟内检测出. 我们的团队努力在06:08恢复核心接入. 最后清理积压的流程和小问题的工作在19:15之前连续完成。 缩略语** 受此事件影响的主要基础设施是正东方地区的事件处理管道,该管道在Atlassian服务之间分配事件,并支撑诸如自动化执行、搜索索引、通知、许可同步等背景业务。 * 00:22至06:08之间,我们托管商的一次基础设施事故引发了我们事件处理管道的摄入故障. * 02:50,事件摄入失败后进入未受影响的可用区,逐渐恢复了活动流量。 * 06:08,新摄入正东方的可靠性恢复到100%。 剩下的工作是清除跨区域积压的信息,这些信息于17:00完成。 * 至18:48,自动化处理其积压的处理过程中生成的事件 ** 提名** 00:22至02:50之间,由于来自亲东方地区的事件而引发的具有自动化规则的客户,规则被处决的情况显著减少. 在这个窗口中,事件触发自动化规则没有被发射,因为触发它们的事件没有被交付. 规则的编写、保存和规则通过时间表或网络呼号手动触发并不受影响。 02:50,事件处理基础设施在未受影响可用区失败,将现场事件交付自动化,并允许新的事件触发规则开始正常执行. 然而,在处理延迟的自动化之前,撞击窗口内产生的事件仍需重新播放. 从08:28开始,上游服务以相协调的顺序重放其排队事件,所有重放事件均由18:48处理. 在回放窗口中,客户可能经历过比预期执行晚的自动化规则,少数规则由于压缩回放而达到日常处理限制,以及时间敏感规则如果超过内部出放阈值,则没有如预期的那样完成. ** 吉拉和吉拉服务管理** 00:22至02:50之间,有租户在普罗德-东区的客户遭遇了对Jira和Jira Service Management(英语:Jira Service Management)事件驱动功能的干扰,以及基础设施故障期间的短暂高发出错. 在整个事件中,仍然有包括问题观察、板子和项目导航在内的吉拉核心经验。 Jira事件交付受到主要影响,使下游服务无法接收发布生命周期事件。 这影响了由Jira事件所引发的自动化规则,由Jira的AI代理管弦乐,关于发行更新和过渡的通知,针对新创建或修改的问题的搜索索引,以及由事件驱动的Jira与其他Atlassian产品的集成. 02:50,事件处理基础设施在未受影响的可用区失败,恢复了新事件的交付. 撞击窗口期间产生的所有事件都保留在恢复队列中并需要重放. 开始于08:28,12:00完成. 在回放窗口期间,客户可能经历了比预期执行晚的自动化规则,触发行动后数小时到达的延迟通知,对撞击窗口期间创建或修改的内容搜索结果的临时漏洞,以及AI代理工作流程在内部超时阈值时没有如预期完成. ** 影响** 00:22-02:50之间,有租户在普罗德-东区托管的顾客在Confact被中断了事件驱动的服务. 这导致搜索索引、通知、自动化规则执行和许可同步方面的延误。 潜在的事件处理基础设施未能进入一个未受影响的可用区,随后恢复了现场聚合作业。 然而,撞击窗口所生成的事件被排出排队进行回放,一些背景服务仍然被推迟,直到回放和相关验证工作完成. 10:14至17:00之间,完成了所有排队租户重放任务的批量回放,以恢复数据一致性. 在回放窗口期间和紧接后,客户可能经历了无法反映在出道期间创建或修改内容的搜索结果,页面和评论活动的通知被延迟或缺失,自动化规则比预期的晚发,以及依赖递增身份同步的租户在许可同步方面的短暂延迟. ** Bitbucket和管道** 00:22至06:08之间,使用Bitbucket和管道的客户在由事件驱动的工作流程中经历了故障和功能退化. 包括推、拉和克隆在内的核心基特行动没有受到影响,在整个事件中继续正常运作。 在撞击窗口期间,无法使用由推力或拉力请求事件启动的管道自动触发器。 合并队列,自定义合并检查,基于Forge的触发,工作空间权限更改,一些工作空间提供流量也受到影响. 使用合并队列的客户无法合并拉动请求,一些管道步骤失败了,因为排队的工作导致币值限制提高. 大约03:57,管道被重新配置,通过另一条路径消耗事件,恢复了自动管道触发. 随着基本事件处理基础设施的恢复,合并队列,自定义合并检查,Forge触发等受影响的工作流程被逐步恢复. 所有Bitbucket和管道服务在06:08前被确认全面运营. 在恢复后,对排队事件进行了审查并重新播放,以便安全地恢复用于计费、审计记录和其他背景过程的数据一致性。 ** 身份服务** 00:22至02:50之间,在东部地区有租户的客户在传播身份和改变下游Atlassian产品群体成员方面出现延误。 核心身份行动,包括认证、登录和直接团体管理行动,没有受到影响,在整个事件中继续正常运作。 其影响仅限于以事件处理管道为基础的同步、事件驱动的操作。 这包括在提供团体成员资格方面出现延误,以及对Jira和Concult等产品的用户配置变化,这影响了下游许可同步和人群同步流量。 少数基于SCIM的身份同步和站点提供工作流程也经历了暂时的延迟. 事件处理基础设施恢复后,根据需要重新播放备份身份和群体目录,恢复受影响产品的下游一致性。 没有丢失身份数据。 团体成员变更,用户简介更新,以及撞击窗口期间发生的提供相关事件,在恢复后被保留并处理. ** 紧急行动计划和下一步** 我们知道停电会影响你的生产力 虽然我们的监测和恢复进程有助于我们迅速作出反应,但这一事件凸显出进一步加强事件驱动服务复原力的机会。 我们正在优先作出改进,以便: * ** 扩大故障覆盖** 这样,在基础设施中断期间,关键事件处理能够更顺利地恢复。 * ** 加强回收处理** 所以重放的事件可以更快地处理。 我们向服务在这一事件中受到影响的客户道歉;我们正在立即采取步骤改善平台的性能和可用性。 谢谢 Atlassian客户支助.

自动翻译自官方事件更新。

1. 2026年4月29日 UTC 07:00investigating

   We are investigating degraded performance issues affecting Loom's core functionalities - authentication, recording, viewing and editing looms. We will provide more details within the next hour.

2. 2026年4月29日 UTC 07:50resolved

   The Team has identified the root cause and the issue has now been resolved, and the impacted services are operating normally. Team will continue to monitor the impacted services.

1. 2026年2月3日 UTC 10:24investigating

   Customers utilizing Confluence and Jira Mobile may experience disruptions in OAuth authentication flows. Forge installations and invocations might also be disrupted. Our team is actively investigating the same and we shall keep you informed of the progress within next 60 mins or sooner.

2. 2026年2月3日 UTC 10:50resolved

   Team has identified the root cause and the issue has now been resolved, and the impacted services are operating normally. Team will continue to monitor the impacted services.

1. 2025年11月19日 UTC 22:32identified

   We are aware of some customers that are experiencing no audio when viewing their Loom recordings. Our team has identified this issue and is working urgently to restore the audio for these recordings. Please rest assured that the audio has been recorded correctly, and we expect this to be fully available when this incident is resolved. New recordings are no longer impacted by this issue. We will provide an update within two hours with our progress as the team continues to investigate.

2. 2025年11月20日 UTC 00:33identified

   Our team believes that we have identified the cause of the issue impacting audio during playback of videos for some customers. As mentioned in our previous updates, new recordings will be working as expected, however videos recorded between 19 Nov 16:51 UTC and 19 Nov 21:30 UTC are expected to be impacted by this issue. A fix for this issue to restore audio to these recordings is now being developed and tested, and we expect to have further update within 2 hours on progress for this resolution.

3. 2025年11月20日 UTC 02:30identified

   The fix to resolve issues on audio recordings impacted by this incident is now being prepared for deployment. Currently the expected duration of the rollout is 3 hours, at which point we expect the audio from the impacted recordings to be fully restored. We will provide further update at that time with the progress of the restoration and provide any additional information that may be required.

4. 2025年11月20日 UTC 05:39resolved

   The rollout of the fix for impacted videos without audio has now completed. Customers should now be able to review existing videos without audio and see that it has been properly restored. If you are continuing to see any issues with playback of your videos, please reach out to Loom support for assistance. We apologize for any inconvenience caused by this issue.

1. 2025年10月27日 UTC 15:35investigating

   我们正在调查一个问题 记录,Loom AI, 认证,和账单 正在影响一些Loom云客户。 我们将在下一个小时内提供更多的细节.

2. 2025年10月27日 UTC 16:33investigating

   我们继续通过记录、Loom AI、认证和账单来调查这个问题, 我们正在调查,并将在协调世界时17:35或更早时发布最新消息.

3. 2025年10月27日 UTC 17:34investigating

   我们继续通过记录、Loom AI、认证和账单来调查这个问题, 我们会在协调世界时18点35分 或更早地发布最新消息 随着事态的发展.

4. 2025年10月27日 UTC 18:32investigating

   我们继续用记录、Loom AI、认证和帐单来积极调查这个问题, 我们感谢你的耐心。 我们将在协调世界时19点35分或更早地发布另一份最新消息.

5. 2025年10月27日 UTC 19:06monitoring

   我们通过记录、Loom AI、认证以及影响一些Loom Cloud客户的账单,确定了问题的根源。 我们已实施缓解措施来解决这一问题,目前正在密切监测。 我们将在协调世界时20:00或更早的时间内,随着事态的发展,再次提供最新消息.

6. 2025年10月27日 UTC 19:59monitoring

   我们正积极实施进一步缓解措施,以解决影响记录、Loom AI、认证和为一些Loom Cloud客户开账单的问题。 我们将继续密切监视,并在协调世界时21:00或更早的时间内,随着事态的发展,再次提供最新情况.

7. 2025年10月27日 UTC 21:06monitoring

   我们实施进一步缓解措施,以解决影响记录、Loom AI、认证和一些Loom Cloud客户的账单的问题。 我们正在看到恢复的良好迹象,所有服务都已投入运作。 随着事态的发展,我们将继续密切监测,并在协调世界时22:00或更早时再提供最新情况.

8. 2025年10月27日 UTC 22:00resolved

   2025年10月27日 UTC中午12点半至22点15分,我们经历了退化的性能,影响了记录,Loom AI,认证,以及一些Loom客户的Billing服务. 我们已部署一个解决办法来缓解这一问题并核实服务已经恢复。 造成问题的条件已经解决,我们正积极研究永久解决. 问题已经解决,服务正常运行. 一旦我们完成了内部事件审查程序,我们将公布一个更详尽的后验尸结果,说明出了什么错误,同时我们正采取步骤来防止将来再次发生这种情况. 说明: 受此事件影响,退化性能时期所记录的一些Looms可能一直处于"上载"状态. 这些录像无法取回,已从用户的图书馆归档.

9. 2025年11月21日 UTC 19:49postmortem

   页:1 在协调世界时2025年10月27日12:30至20:55,Loom为所有顾客大幅退化. 2025年10月28日20:26至20:33世界协调世界时和2025年10月30日19:22至19:27世界协调世界时,用户也经历了短暂的退化. 事件期间,用户遭遇了间歇回放或故障回放,录音,笔录生成,发帖登录的问题,以及整个服务的一般错误或缓慢. 我们的监测显示,20%至80%的用户互动在这些期间都失败了,事故后半部分的失败在增加。 我们知道停电会影响你的生产力 我们向在这起事件中受到影响的客户道歉 我们正在根据我们对这一事件的教训,立即采取步骤提高Loom的可靠性。 分析 从2025年10月25日开始,Loom的主要数据库集群显示了一个问题的早期指标。 资源使用量增长缓慢,复制档没有跟上。 工程人员于2025年10月26日被警示并开始调查. 然而,我们确实发现了一个问题,一个重新尝试的背景工作没有成功。 我们为这个问题运送了一个临时的固定设备,并计划第二天进一步调查。 2025年10月27日,星期一,随着使用量的增加,这个问题大幅再现,我们开始观察客户的影响. 工程师们正在密切监测这个问题,并立即介入。 显然,我们的数据库没有按照预期运行,在接下来的几个小时里,我们多次尝试减缓。 值得注意的是,我们扩大了数据库撰写器和两个阅读器,以减轻资源压力。 这没有帮助,用户的影响保持不变。 在接下来的几个小时里,我们多次尝试恢复数据库的正常性能. 我们回滚了最近的变化,优化了个人查询的性能,增加了新的索引,更新了数据库配置,并真空了大型系统和应用表。 这些变化提供了一些短期缓解,但业绩一再恶化。 放弃复制槽确实解决了资源压力,但没有改善总体查询性能。 我们接着指出,同样的查询在编剧身上的策划和执行速度远快于我们的读者实例,因此我们决定将所有查询流量都转移到单一的编剧实例. 这一变化立即减轻了客户的影响。 尽管这一行动解决了眼前的问题,但对我们的长期服务来说,它不是一个稳定的状态。 在接下来的几天里,我们多次尝试恢复我们典型的单写,双读的架构. 每次,我们都看到同样的问题再次出现,导致短期退化。 在每次尝试之间,我们进一步优化了我们的数据库使用,但我们仍然没有看到我们在增加读者实例时所期望的表现。 这尤其具有挑战性,因为我们的服务如预期的那样与单一读者合作,我们只是在为分组增加第二名读者时才注意到这种退化。 2025年10月31日,我们尝试了用更小的实例大小来完成故障,类似于垂直缩放尝试之前的我们最初的架构. 这成功使Loom建筑恢复正常,并完全缓解了事件. 根原因 : 广义上,这一事件可以分为两部分. 首先,最初的退化是由事发前几个月的多起Loom变化所引起. 第二,由于我们努力减轻个案规模的扩大,延长了缓解期。 这起事件的第一部分是由Loom服务的多起旧有和最近变化的复杂互动引起的: * 多年来,Loom一直有一个背景工作,使用嵌入式数据库交易,这是我们代码库中罕见的模式. 为整个工作开启父母交易,每批工作都有子女交易。 这份工作一直很可靠,但一旦失败,将多次重试. 这项工作数量相对较少,而且通常速度快,但在罕见的情况下可能需要大量的数据库工作. * 在过去几个月里,为了支持即将出现的特征和迁移、基线负荷和Loom数据库的活动大幅增加。 * 2025年10月14日,也为了支持即将到来的功能,对Loom app处理数据库交易的方式做了广泛的修改. 这一变化只是为了在我们试验和中转环境中生效,但在与巢交易有关的特殊情况中,它也适用于生产。 * 2025年10月25日,这些合并变化引发了我们数据库的缓慢和日益退化。 一些大背景任务合并运行并失败,导致重试将永久开放的交易保留在数据库上. 这种退化的复制槽和自动蒸发,导致整个数据库的表血。 2025年10月27日,这种退化开始影响我们的用户. * 在移除了最初的触发器后,数据库继续退化,需要大量人工干预来稳定。 恢复很复杂,因为试图缓解\(垂直实例缩放\)引发了一个新的问题。 事故的后半部分 是因为我们早先的缓解努力 扩大我们的例子规模 来处理增加的负载 从第一部分: * 事发初期,由于数据库资源使用量增加,我们试图通过纵向扩大规模来减轻损失。 我们一般认为,这是安全和往往有效的缓解措施。 * 这一变化出人意料地使数据库退化更严重。 我们从较大的事例中观察到的退化情况与我们在事件的第一部分中已经观察到的退化情况类似。 事件发生后,我们从云层提供者那里直接得知,较大的实例规模使用一个单独的内存架构,对我们的工作量表现得更糟糕。 * 在开通新连接时,实例类型之间的这种性能差异特别明显。 作为调查的一部分,我们发现一个问题 与我们的连接池 导致过度连接churn 当指向 DNS 负载平衡的阅读器端点。 这解释了为什么我们只在集群中增加第二读取器时才观察到问题,而不是使用单写器的单读取器配置. ** 补救和改进** 作为我们在事件期间调查和减轻影响努力的一部分,我们已经完成了几项解决类似问题的关键行动,包括: * 优化申请内的交易处理。 * 提高高数量和业绩不佳的查询的性能。 * 改进对真空状态和表血的监测和可观察性。 * 内部工具改进以方便调试 * 产品中标,在事件期间提醒用户。 此外,我们正在优先采取下列改进行动: * 进一步改善交易处理和监测。 * 优化连接集成配置。 * 优化数据库索引使用。 * 数据库配置改进。 * 改进自体真空参数。 * 改进我们衡量和监测客户影响的方法。 谢谢 阿特拉斯语

自动翻译自官方事件更新。

1. 2025年10月20日 UTC 07:56investigating

   We have noticed that Atlassian Cloud services are impacted and our teams are actively investigating the same. We shall keep you informed of the progress every hour.

2. 2025年10月20日 UTC 08:24investigating

   Atlassian Cloud services are impacted and we are aware that our customers might not be able to create support tickets. Our teams are actively investigating the same. We shall keep you informed of the progress every hour.

3. 2025年10月20日 UTC 09:24investigating

   We are experiencing an outage due to some issue at the end of our public cloud provider. We are working closely with them to get this resolved or mitigated as quickly as possible. ETA of the same is not know at the moment. We shall continue to share updates every hour, if not sooner.

4. 2025年10月20日 UTC 09:58identified

   We understand that our public cloud provider has identified the cause of the issue. We are starting to see some recovery and is working towards mitigation. We appreciate your patience. We shall continue to share updates every hour, if not sooner.

5. 2025年10月20日 UTC 11:07identified

   We continue to work with our public cloud provider towards mitigating the issue at the earliest. We appreciate your patience. We shall continue to share updates every hour, if not sooner.

6. 2025年10月20日 UTC 12:06identified

   Atlassian team is actively engaged and continues to work with our public cloud provider to mitigate this issue at the earliest. We are starting to see partial operations succeed. We appreciate your patience. We shall continue to share updates every hour, if not sooner.

7. 2025年10月20日 UTC 13:58identified

   Our public cloud provider is working to mitigate this issue quickly. We are seeing some early positive indicators and are continuing to monitor. We appreciate your patience and will continue to provide updates every hour or sooner.

8. 2025年10月20日 UTC 14:03identified

   We understand your pain and mitigating or fixing this issue is of utmost importance. Our public cloud provider is actively working to mitigate this issue on priority. We have been seeing partial operational success. We appreciate your patience and will continue to provide updates every hour or sooner.

9. 2025年10月20日 UTC 16:36identified

   Update - Thank you for your continued patience. We understand the impact this issue is having on your operations and want to assure you that resolving this matter is our highest priority. Our public cloud provider is still actively working to mitigate this issue with urgency and while we do not have a definitive ETA at this time, we remain committed to full resolution and deeply appreciate your patience as we work through this situation. We will be providing hourly updates on this issue.

10. 2025年10月20日 UTC 17:27identified

    Update - We understand the impact this issue is having on your operations and want to assure you that resolving this matter is our highest priority. Our public cloud provider is actively working to mitigate this issue with urgency. While we do not have a definitive ETA at this time, we remain committed to full resolution and deeply appreciate your patience as we work through this situation. We will continue to provide updates every hour or sooner as new information becomes available.

11. 2025年10月20日 UTC 17:43identified

    We are currently aware of an ongoing incident impacting Atlassian Cloud services due to an outage with our public cloud provider, AWS. We understand the impact this issue is having on your operations and want to assure you that resolving this matter is our highest priority and we are closely monitoring the health of AWS services. While we do not have a definitive ETA at this time, we remain committed to full resolution and deeply appreciate your patience as we work through this situation. We will continue to provide updates every hour or sooner as new information becomes available.

12. 2025年10月20日 UTC 18:38monitoring

    There have been no changes since our last update. We will provide our next updated by 9:00PM UTC or sooner as new information becomes available. We are currently aware of an ongoing incident impacting Atlassian Cloud services due to an outage with our public cloud provider, AWS. We understand the impact this issue is having on your operations and want to assure you that resolving this matter is our highest priority and we are closely monitoring the health of AWS services. While we do not have a definitive ETA at this time, we remain committed to full resolution and deeply appreciate your patience as we work through this situation.

13. 2025年10月20日 UTC 21:26monitoring

    Monitoring - We've started seeing continued product experience improvement. While we still have a backlog of event processing, we are seeing improvements in systems operational capabilities across all products. We estimate a significant improvement with the next few hours and will continue to monitor the health of AWS services and the effects on Atlassian customers. We appreciate your continued patience and remain committed to full resolution as we work through this situation. We will post our next update in two hours.

14. 2025年10月20日 UTC 22:58monitoring

    Our teams are continuing to monitor the recovery of systems across Atlassian products. This update is to inform that the Atlassian Support portal is fully operational at this time for customers that wish to contact support.

15. 2025年10月21日 UTC 00:03monitoring

    Our team is now seeing recovery across all impacted Atlassian products. We are continuing to monitor for individual products that may still be processing backlogged items now that services are restored. The Atlassian Support portal is currently still displaying a message directing customers to our temporary support channel. Please note that our support portal is currently fully functional for those attempting to raise requests. We are continuing to look into this alert to remove this message. We will provide further update on our recovery status in one hour.

16. 2025年10月21日 UTC 01:07monitoring

    We continue to see recovery progressing across all impacted products as backlogged items continue to be processed. The Atlassian Support portal is currently displaying a message directing customers to our temporary support channel. Please note that our support portal is currently fully functional for those attempting to raise requests. We are continuing to look into this alert to remove this message. We will provide further update on our recovery status in two hours.

17. 2025年10月21日 UTC 03:24monitoring

    The issue relating to the Atlassian Support portal displaying a message to customers to use our temporary support channel has now been resolved. The Atlassian Support portal is fully functional for any ongoing support issues. With regards to other Atlassian products, we continue to see recovery continuing across all impacted products and our teams are continuing to monitor as the recovery continues. We will provide further update on our recovery status within two hours.

18. 2025年10月21日 UTC 05:24resolved

    Our team is now able to see full recovery across the vast majority of Atlassian products. We are aware of some ongoing issues with specific components such as migrations and JSM virtual service agents, and our team is continuing to investigate with urgency. We apologise for the inconvenience that this incident has caused and we will provide further information when the Post Incident Investigation has been completed.

19. 2025年10月24日 UTC 17:53postmortem

    ### Postmortem publish date: Nov 19th, 2025 ### Summary All dates and times below are in UTC unless stated otherwise. Customers utilizing Atlassian products experienced elevated error rates and degraded performance between Oct 20, 2025 06:48 and Oct 21, 2025 04:05. The service disruptions were triggered due to an [AWS DynamoDB outage](https://aws.amazon.com/message/101925/#:~:text=1%3A50%20PM.-,DynamoDB,-Between%2011%3A48) and further affected by subsequent failures in [AWS EC2](https://aws.amazon.com/message/101925/#:~:text=service%20disruption%20event.-,Amazon%20EC2,-Between%2011%3A48) and [AWS Network Load Balancer](https://aws.amazon.com/message/101925/#:~:text=service%20disruption%20event.-,Amazon%20EC2,-Between%2011%3A48) within the us-east-1 region. The incident started at Oct 20, 2025 06:48 and was detected within six minutes by our automated monitoring systems. Our teams worked to restore all core services by Oct 21, 2025 04:05. Final cleanup of backlogged processes and minor issues was completed on Oct 22, 2025. We recognize the critical role our products play in your daily operations, and we offer our sincere apologies for any impact this incident had on your teams. We are taking immediate steps to enhance the reliability and performance of our services, so that you continue to receive the standard of service you have come to trust. ### IMPACT Before examining product-level impacts, it's helpful to understand Atlassian's service topology and internal dependencies. Products such as Jira and Confluence are deployed across multiple AWS regions. The data for each tenant is stored and processed exclusively within its designated host region. This design is intentional and represents the desired operational state, as it limits the impact of any regional outage strictly to tenants in-region, in this case us-east-1. While in-scope application data is pinned to the region selected by the customer, there are times when systems need to call other internal services that may be based in a different region. If a problem occurs in the main region where these services operate, systems are designed to automatically fail over to a backup region, usually within three minutes. However, if unexpected issues arise during this failover, it can take longer to restore services. In rare cases, this could affect customers in more than one region. It’s important to note that all in-scope application data for supported products is pinned according to a customer’s chosen region. **Jira** Between Oct 20, 2025 06:48 and Oct 20, 2025 20:00, customers with tenants hosted in the us-east-1 region experienced increased error rates when accessing core entities such as Issues, Boards, and Backlogs. This disruption was caused by AWS's inability to allocate AWS EC2 instances and elevated errors in AWS Network Load Balancer \(NLB\). During this window, users may also have observed intermittent timeouts, slow page loads, and failures when performing operations like creating or updating issues, loading board views, and executing workflow transitions. Between Oct 20, 2025 08:36 and Oct 20, 2025 09:23, customers across all regions experienced elevated failure rates when attempting to load Jira pages. This disruption was caused by the regional frontend service entering an unhealthy state during this specific time interval. Normally, the frontend service connects to the primary AWS DynamoDB instance located in the us-east-1 to retrieve the most recent configuration data necessary for proper operation. Additionally, the service is designed with a fallback mechanism that references static configuration data in the event that the primary database becomes inaccessible. Unfortunately, a latent bug existed in the local fallback path. When the frontend service nodes restarted, they were unable to load critical operational configuration data from primary or fallback sources, leading to the observed failures experienced by customers. Between Oct 20, 2025 06:48 and Oct 21, 2025 06:30, customers experienced significant delays and missing Jira in-app notifications across all regions. The notification ingestion service, which is hosted exclusively in us-east-1, exhibited an increased failure rate when processing notification messages due to AWS EC2 and NLB issues. This issue resulted in notifications being delayed - and in some cases, not delivered at all - to users worldwide. **Jira Service Management \(JSM\)** JSM was impacted similarly to Jira above, with the same timeframes and for the same reasons. Between Oct 20, 2025 08:36 and Oct 20, 2025 09:23, customers across all regions experienced significantly elevated failure rates when attempting to load JSM pages. This affected all JSM experiences including the Help Centre, Portal, Queues, Work Items, Operations, and Alerts. **Confluence** Between Oct 20, 2025 06:48 and Oct 21, 2025 02:45, customers using Confluence in the us-east-1 region experienced elevated failure rates when performing common operations such as editing pages or adding comments. The primary cause of this service degradation was the system's inability to auto-scale due to AWS EC2 issues to manage peak traffic load effectively. Though the AWS outage ended at Oct 20, 21:09, a subset of customers continued to experience failures as some Confluence web server nodes across multiple clusters remained in an unhealthy state. This was ultimately mitigated by recycling the affected nodes. To protect our systems while AWS recovered, we made a deliberate decision to enable node termination protection. This action successfully preserved our server capacity but, as a trade-off, it extended the time required for a full recovery once AWS services were restored. **Automation** Between Oct 20, 2025 06:55 and Oct 20, 2025 23:59, automation customers whose rules are processed in us-east-1 experienced delays of up to 23 hours in rule execution. During this window, some events triggering rule executions were processed out of order because they arrived later during backlog processing. This caused potential inconsistencies in workflow executions, as rules were run in the order events were received, not when the action causing the event occurred. Additionally, some rule actions failed because they depend on first-party and third-party systems, which were also affected by the AWS outage. Customers can see most of these failures in their audit logs; however, a few updates were not logged due to the nature of the outage. By Oct 21, 2025 5:30, the backlog of rule runs in us-east-1 was cleared. Although most of these delayed rules were successfully handled, there were some additional replays of events to ensure completeness. Our investigation confirmed that a few events may never have triggered their associated rules due to the outage. Between Oct 20, 2025 06:55 and Oct 20, 2025 11:20, all non-us-east-1 regional automation services experienced delays of up to 4 hours in rule execution. This was caused by an upstream service that was unable to deliver events as expected. The delivery service encountered a failure due to a cross-region dependency call to a service hosted in the us-east-1 region. Because of this dependency issue, the delivery service was unable to successfully deliver events throughout this time frame, resulting in customer-defined rules not being executed in a timely manner. **Bitbucket and Pipelines** Between Oct 20, 2025 06:48 and Oct 20, 2025 09:33, Bitbucket experienced intermittent unavailability across core services. During this period, users faced increased error rates and latency when signing in, navigating repositories, and performing essential actions such as creating, updating, or approving pull requests. The primary cause was an AWS DynamoDB outage that impacted downstream services. Between Oct 20, 2025 06:48 and Oct 20, 2025 22:46, numerous Bitbucket Pipeline steps failed to start, stalled mid-execution, or experienced significant queueing delays. Impact varied, with partial recoveries followed by degradation as downstream components re-synchronized. The primary cause was an AWS DynamoDB outage, compounded by instability in AWS EC2 instance availability and AWS Network Load Balancers. Furthermore, Bitbucket Pipelines continued to experience a low but persistent rate of step timeouts and scheduling errors due to AWS bare-metal capacity shortages in select availability zones. Atlassian coordinated with AWS to provision additional bare-metal hosts and addressed a significant backlog of pending pods, successfully restoring services by 01:30 on Oct 21, 2025. **Trello** Between Oct 20, 2025 06:48 and Oct 20, 2025 15:25, users of Trello experienced widespread service degradation and intermittent failures due to upstream AWS issues affecting multiple components, including AWS DynamoDB and subsequent AWS EC2 capacity constraints. During this period, customers reported elevated error rates when loading boards, opening cards, adding comments or attachments. **Login** Between Oct 20, 2025 06:48 and Oct 20, 2025 09:30, a small subset of users experienced failures when attempting to initiate new login sessions using SAML tokens. This resulted in an inability for those users to access Atlassian products during that time period. However, users who already had valid active sessions were not affected by this issue and continued to have uninterrupted access. The issue impacted all regions globally because regional identity services relied on a write replica located in the us-east-1 region to synchronize profile data. When the primary region became unavailable, the failover to a secondary database in another region failed, which delayed recovery. This failover defect has since been addressed. **Statuspage** Between Oct 20, 2025 06:48 and Oct 20, 2025 09:30, Statuspage customers who were not already logged in to the management portal were unable to log in to create or update incident statuses. This impact was restricted only to users who were not already logged in at the time. The root cause was the same as described in the Login section above, and it was resolved by the same remediation steps. ### REMEDIAL ACTION PLAN & NEXT STEPS We have completed the following critical actions designed to help prevent cross-region impact from similar issues: * Resolved the code defect in the fallback option to ensure that Jira Frontend Services in other regions remain unaffected during a region-wide outage. * Fixed the issue that prevented timely failover of the identity service which impacted new login sessions. * Resolved the code defect so that delivery services in unaffected regions remain operational during region-wide outages. Additionally, we are prioritizing the following improvement actions: * Implement mitigation strategies to strengthen resilience against region-wide outages in the notification ingestion service. Although disruptions to our cloud services are sometimes unavoidable during outages of the underlying cloud provider, we continuously evaluate and improve test coverage to strengthen resilience of our cloud services against these issues. We recognize the critical importance of our products to your daily operations and overall productivity, and we extend our sincere apologies for any disruptions this incident may have caused your teams. If you were impacted and require additional details for internal post-incident reviews, please reach out to your Atlassian support representative with affected timeframes and tenant identifiers so we can correlate logs and provide guidance. Thanks, Atlassian Customer Support

1. 2025年9月12日 UTC 20:43monitoring

   From 5:03 AM ET to 4:15 PM ET, some customers were unable to record Zoom meetings using Loom Meeting Recordings. Attempts made during this time may appear in your library but will not be playable, as no recording was captured. Unfortunately, these recordings cannot be recovered. We have released a fix and we are monitoring recovery.

2. 2025年9月12日 UTC 22:14resolved

   The issue has been resolved and the service is operating normally.

1. 2025年9月8日 UTC 21:46resolved

   From 5:20AM to 3:45PM UTC today some customers using the Loom Chrome Extension experienced issues with recording. Looms recorded during this time may show up in the library but will not be playable. Unfortunately we are unable to recover these videos. If you are still experiencing this issue please make sure you are recording with extension version 5.5.121.

1. 2025年8月15日 UTC 19:35identified

   We are investigating reports of issues for Loom customers. Including names showing as IDs. We will provide more details once we identify the cause.

2. 2025年8月15日 UTC 19:50monitoring

   We have identified the cause of the issues and we are monitoring recovery.

3. 2025年8月15日 UTC 19:58resolved

   Between 18:45 and 19:35 UTC we experienced degraded functionality for Loom. The issue has been resolved and the service is operating normally.

1. 2025年4月14日 UTC 19:07identified

   Customers are unable to choose a monthly plan when upgrading or purchasing from loom.com. We are investigating this issue.

2. 2025年4月14日 UTC 19:30monitoring

   A fix has been implemented and we are monitoring the results

3. 2025年4月14日 UTC 19:34resolved

   This issue is now resolved and customers can checkout as expected.