AWS outage history

We are investigating elevated packet loss in the EU-SOUTH-2 Region.

We are experiencing elevated packet loss with a single Availability Zone (eus2-az1) in the EU-SOUTH-2 Region. This issue is also resulting in elevated API latencies and error rates for other workflows and AWS Services. Engineers were automatically engaged and are working to identify the root cause and to mitigate the issue. We will provide you with another update by 10:15 AM PDT or sooner as new information becomes available.

Our efforts to mitigate the elevated packet loss with a single Availability Zone (eus2-az1) in the EU-SOUTH-2 Region is still underway. We are seeing some signs of improvement as a result of multiple parallel efforts. We continue to work to confirm root cause and fully mitigate the issue. While we currently do not have an estimate for how long our actions will take, we will keep you informed of our progress and provide you with another update by 11:00 AM PDT or sooner as new information becomes available.

We have identified the root cause and are currently focusing our mitigation efforts to drive full recovery. While we don't have an exact estimate for how long our current efforts will take, we do expect to see incremental improvements as our efforts proceed. We will provide you with another update by 11:45 AM PDT, or sooner if new information becomes available.

We are seeing early signs of recovery. We will provide you with another update by 12:00 PM PDT.

Between 8:48 AM and 11:28 AM PDT, we experienced elevated packet loss within a single Availability Zone (eus2-az1) in the EU-SOUTH-2 Region. This issue resulted in elevated API latencies and error rates for multiple workflows and AWS Services. We were automatically alerted to the issue at 8:54 AM, and immediately began work to identify the root cause while taking multiple parallel paths to mitigate the issue. By 10:00 AM, our mitigation efforts had led to some improvements. By 10:38 AM, we identified the root cause as related to a networking configuration change that we were able to revert the change which led to full mitigation by 11:28 AM. The issue has been resolved and the service is operating normally.

We are investigating increased API error rates in the US-EAST-1 Region.

We can confirm increased error rates affecting new instance launches, including elevated API error rates for RunInstances, CreateFleet, and related EC2 launch workflows in the US-EAST-1 Region. Customers attempting to launch new EC2 instances may experience launch failures or elevated API response latency. Our engineers are actively engaged and working towards recovering the issue. We are seeing initial signs of recovery. Existing instances remain unaffected. We recommend retrying failed RunInstances and CreateFleet API calls. We will provide an update by 5:30 PM PDT or sooner.

Between 3:49 PM and 5:06 PM PDT, we experienced increased error rates affecting new instance launches in the US-EAST-1 Region. During this time, customers may have experienced elevated error rates when attempting to launch new EC2 instances through RunInstances, CreateFleet, and related EC2 launch workflows. AWS services that launch EC2 instances on behalf of customers, including Amazon ECS, AWS Fargate, Amazon EMR, amongst others were also impacted during this period. We identified the root cause as a recent deployment to a subsystem within the EC2 service responsible for processing instance launch requests. We initiated a rollback of the problematic deployment and began observing initial signs of recovery at 4:44 PM. Existing instances remain unaffected. The issue has been resolved and the service is operating normally.

Starting at 2:12 PM PDT, we began experiencing increased API error rates for STS and Sign In when using SAML in the US-WEST-2 Region. Our engineering team was automatically engaged at 2:19 PM to begin investigating the root cause. There is no work around available at this time. We will provide another update by 3:30 PM PDT.

We are seeing early signs of recovery and continue to monitor for full recovery. We will provide another update at 4:15 PM, or sooner if we have additional information to share.

We continue to see recovery holding steady for the STS AssumeRoleWithSAML and AssumeRoleWithWebIdentity APIs in the US-WEST-2 Region. Error rates are now back to pre-event levels, and we continue actively monitoring to confirm full recovery. We will provide another update by 5:15 PM or earlier.

Between 2:12 PM and 3:18 PM PDT, we experienced increased API error rates affecting the STS AssumeRoleWithSAML and AssumeRoleWithWebIdentity APIs in the US-WEST-2 Region. The root cause was determined to be due to an issue with an STS subsystem responsible for communicating with external identity providers. Other AWS Services that rely on these identity federation protocols were also affected. At 3:18 PM, we observed signs of recovery and continued to monitor to ensure stability and full recovery. The issue is resolved and the service is operating normally at this time.

AP-NORTHEAST-1リージョンにおいて、エラー率が上昇しており、リアルタイムメトリクスに影響が出ています。お客様はリアルタイムメトリクスデータの欠落または遅延を経験する可能性があります。| We are experiencing increased error rates affecting real-time metrics in the AP-NORTHEAST-1 Region. Customers may experience missing or delayed real-time metric data.

日本時間 8:58 AM から 10:34 AM の間、AP-NORTHEAST-1 リージョンにおける Amazon Connect のリアルタイムメトリクスに影響する遅延が増加し、データが欠落する、またはデータが最新の状態に更新されない事象が発生しました。この間、お客様は分析レポート内で一部のデータが反映されない状況を経験した可能性があり、また、コンタクトフロー内でリアルタイムメトリクス (例: 人員の確認ブロック) にアクセスする際に問題が発生した可能性があります。根本原因は、メトリクスイベント配信を担うサブシステムの問題であると特定しました。日本時間 10:13 AM に緩和策の適用を開始し、 10:34 AM までに影響を解消しました。本事象は解決済みであり、サービスは正常に稼働しています。| Between 4:58 PM and 6:34 PM PDT, we experienced increased delays affecting real-time metrics for Amazon Connect in the AP-NORTHEAST-1 Region, resulting in missing or stale data. During this time, customers may have experienced missing data within analytics reports, and may have observed issues if accessing realtime metrics within contact Flows, such as checking agent staffing. We identified the root cause to be an issue with the subsystem responsible for metric event delivery. We started applying the mitigations at 6:13 PM and mitigated the issue by 6:34 PM. The issue has been resolved and the service is operating normally.

We are investigating an issue that is impacting launching new EC2 instances and resources in a newly launched Availability Zone (euw2-az4) in the EU-WEST-2 Region. During this time, affected customers may experience issues when creating or modifying resources in the Region. Other AWS services may also be impacted. For immediate recovery, we recommend that customers use alternative Availability Zones (euw2-az1, euw2-az2, and euw2-az3) where applicable. Existing running instances and resources are not affected. We will provide another update by 10:00 AM PDT, or sooner if we have additional information to share.

On August 18 we launched a new Availability Zone (euw2-az4) in the EU-WEST-2 Region. After the launch, we began experiencing errors launching EC2 instances in the new Availability Zone when a default subnet is not present. We can confirm that existing running instances and resources are not affected. Workflows that automatically get a list of Availability Zones in the Region via the DescribeAvailabilityZones API and then attempt to launch new instances or create resources in the new Availability Zone may encounter errors. For EC2 instance launch failures, we are taking mitigating steps to automatically create default subnets, where one is not already present, when an EC2 instance launch is targeting the new Availability Zone. For customers and workflows that require immediate remediation <a href="https://docs.aws.amazon.com/vpc/latest/userguide/work-with-default-vpc.html#create-default-subnet">you may create a default subnet</a> in the new Availability Zone. This will enable EC2 instance launches to successfully complete. For other resources, such as Lambda functions, where the new Availability Zone is currently not supported, we recommend customers update their workflows to exclude the newly launched Availability Zone and continue resource creation using the other Availability Zones in the Region. While we don't have an exact estimate for how long our mitigation efforts will take, we will keep you up to date on our progress and provide you with another update by 1:00 PM PDT or sooner as new information becomes available.

Between August 18 5:00 PM and August 19 11:00 AM PDT, we experienced elevated errors launching EC2 instances in a newly launched Availability Zone (euw2-az4) in the EU-WEST-2 Region. After the new Availability Zone launch, we began experiencing errors when using a default VPC. We discovered the root cause of the issue on August 19 at 9:00 AM and began deploying a change to resolve the issue at 9:30 AM. While the change was underway, we began to see incremental improvements in new instance launches, with full recovery at 11:00 AM. Existing running instances and resources were not affected. Some regional services, such as Lambda functions or Aurora databases, were not available at the launch of the new Availability Zone and service availability will be added over time. Customers attempting to create resources before the services become available will see a message reporting that it is not supported in the Availability Zone. The issue has been resolved and the service is operating normally.

We are investigating increased packet loss, impacting AWS Direct Connect connectivity for some customers in the EU-CENTRAL-1 Region.

We can confirm packet loss impacting Direct Connect connections in the EU-CENTRAL-1 Region. Engineers were automatically engaged and immediately began working to both identify the root cause, and identify multiple parallel paths to mitigate the issue. At this time, we are seeing early signs of recovery. We will provide another update in 60 minutes, or sooner if we have additional information to share.

Starting at 7:33 PM PDT, we began experiencing increased packet loss impacting AWS Direct Connect connectivity for some customers in the EU-CENTRAL-1 Region. While we have made progress, connections to the following Direct Connect location are still impaired: Equinix FR5, Frankfurt, DEU. Customers who have multi-site redundancy configured with their Direct Connect paths should not be observing impact at this time. Customers whom only have connections at the Equinix FR5, Frankfurt, DEU location will continue to experience connectivity issues. We are actively working to mitigate the impact and work toward full recovery, but expect full recovery is multiple hours away. We will provide an update in 90 minutes, or sooner if we have additional information to share.

We are actively working to restore connectivity through the Direct Connect location: Equinix FR5, Frankfurt, DEU. Customers whom only have connections at the Equinix FR5, Frankfurt, DEU location will continue to experience connectivity issues. For a workaround impacted customers who have the option available to failover to VPN are recommended to do so to achieve recovery. For customers using Direct Connect gateway and Transit Gateway, we recommend creating a AWS Site-to-Site VPN and attach it to your Transit Gateway, refer steps <a href="https://aws.amazon.com/premiumsupport/knowledge-center/dx-configure-dx-and-vpn-failover-tgw/">here</a>. For other customers we recommend establishing a AWS Site-to-Site VPN as a temporary backup path, refer steps <a href="https://docs.aws.amazon.com/vpn/latest/s2svpn/SetUpVPNConnections.html">here</a>. As of this time, we expect recovery is multiple hours away. We will provide another update in 90 minutes, or sooner if we have additional information to share.

We continue to work towards recovery of connectivity for AWS Direct Connect connections at Equinix FR5, Frankfurt, DEU. The root cause is related to a facility infrastructure issue at the location that is impacting network infrastructure. Customers with connections solely at this location will continue to experience packet loss or connectivity degradation. Customers with multi-site or redundant configurations across other locations are not impacted. For a workaround, impacted customers who have the option available to failover to VPN are recommended to do so. For customers using Direct Connect gateway and Transit Gateway, we recommend creating a AWS Site-to-Site VPN and attaching it to your Transit Gateway, refer to steps <a href="https://aws.amazon.com/premiumsupport/knowledge-center/dx-configure-dx-and-vpn-failover-tgw/">here</a>. For other customers we recommend establishing an AWS Site-to-Site VPN as a temporary backup path, refer to steps <a href="https://docs.aws.amazon.com/vpn/latest/s2svpn/SetUpVPNConnections.html">here</a>. As of this time, we expect recovery is multiple hours away. We will provide another update within 2 hours or as soon as we have more information to share.

AWS Direct Connect connectivity remains impaired for customers with connections at the Equinix FR5 location in Frankfurt, DEU. Customers with multi-site or redundant configurations across other locations continue to be unaffected. Engineers are actively working to restore connectivity, with efforts ongoing across multiple workstreams to resolve the underlying facility issue and bring the impacted network equipment back into service. We continue to expect recovery is multiple hours away. For a workaround, impacted customers who have the option to failover to VPN are recommended to do so. For customers using Direct Connect gateway and Transit Gateway, we recommend creating a AWS Site-to-Site VPN and attaching it to your Transit Gateway, refer to steps <a href="https://aws.amazon.com/premiumsupport/knowledge-center/dx-configure-dx-and-vpn-failover-tgw/">here</a>. For other customers we recommend establishing an AWS Site-to-Site VPN as a temporary backup path, refer to steps <a href="https://docs.aws.amazon.com/vpn/latest/s2svpn/SetUpVPNConnections.html">here</a>. We will provide another update within 2 hours or as soon as we have more information to share.

Engineers continue work to restore connectivity at Equinix FR5 location in Frankfurt, DEU. Our co-location partner is working to resolve the underlying facility infrastructure issue and while the improvements are not yet customer visible, we are making positive progress towards resolution. For customers who require immediate recovery, we recommend failing over to VPN, as outlined in our previous updates. We will provide another update by 9:30 AM PDT, or sooner if we have additional information to share.

Our co-location partner continues to work to resolve the underlying facility infrastructure issue at the Equinix FR5 location in Frankfurt, DEU. Access to the affected area is currently restricted due to safety concerns, which is impacting our ability to assess the physical condition of the network equipment and provide a more accurate recovery timeline. Based on current information, full recovery is not expected in the near term and may extend beyond today. AWS Direct Connect connections at this location remain impaired. Customers with redundant connections through other locations remain unaffected. For customers using Direct Connect gateway and Transit Gateway, we recommend creating a AWS Site-to-Site VPN and attaching it to your Transit Gateway, refer to steps <a href="https://aws.amazon.com/premiumsupport/knowledge-center/dx-configure-dx-and-vpn-failover-tgw/">here</a>. For other customers we recommend establishing an AWS Site-to-Site VPN as a temporary backup path, refer to steps <a href="https://docs.aws.amazon.com/vpn/latest/s2svpn/SetUpVPNConnections.html">here</a>. We will provide another update by 3:30 PM PDT, or sooner if we have additional information to share.

Our co-location partner is continuing work to restore safe access to the affected area at the Equinix FR5 location in Frankfurt, DEU. Once safe access has been secured, our engineers will be able to assess the affected network devices. We continue to closely track progress and will share an update by 9:30 PM PDT, or sooner as new information becomes available.

We are actively engaged with our co-location partner to restore connectivity at Equinix FR5 location in Frankfurt, DEU. Since our last update, we have made incremental progress to restore safe access to the affected area at the Equinix FR5 location in Frankfurt, DEU. In parallel, we have prioritized the order in which critical and high-priority racks will be restored, as part of the mitigation efforts. Based on our current assessment, full recovery is not expected in the near term and may extend beyond today. AWS Direct Connect connections at this location remain impaired. Customers with connections exclusively at this location will continue to experience packet loss. Customers with multi-site or redundant configurations across other Direct Connect locations remain unaffected. For a workaround, impacted customers who have the option available to failover to VPN are recommended to do so. For customers using Direct Connect gateway and Transit Gateway, we recommend creating a AWS Site-to-Site VPN and attaching it to your Transit Gateway, refer to steps <a href="https://aws.amazon.com/premiumsupport/knowledge-center/dx-configure-dx-and-vpn-failover-tgw/">here</a>. For other customers we recommend establishing an AWS Site-to-Site VPN as a temporary backup path, refer to steps <a href="https://docs.aws.amazon.com/vpn/latest/s2svpn/SetUpVPNConnections.html">here</a>. We continue to closely track progress and will share an update by August 16 3:30 AM PDT, or sooner as new information becomes available.

We continue to work with our co-location partner to restore connectivity at the Equinix FR5 location in Frankfurt, DEU. Since our last update, we have made significant progress toward restoring safe access to the affected area. The electrical isolation procedure is now underway, with our teams on site in the electrical room executing the de-energization of the affected infrastructure. Once isolation is verified and confirmed safe, engineers will begin a physical inspection of the impacted network equipment to determine the scope of replacement required. Based on our current assessment, full recovery is not expected in the near term due to the scope of potential impacted to equipment. AWS Direct Connect connections at this location remain impaired. Customers with connections exclusively at this location will continue to experience packet loss. Customers with multi-site or redundant configurations across other Direct Connect locations remain unaffected. For a workaround, impacted customers who have the option available to failover to VPN are recommended to do so. For customers using Direct Connect gateway and Transit Gateway, we recommend creating a AWS Site-to-Site VPN and attaching it to your Transit Gateway, refer to steps <a href="https://aws.amazon.com/premiumsupport/knowledge-center/dx-configure-dx-and-vpn-failover-tgw/">here</a>. For other customers we recommend establishing an AWS Site-to-Site VPN as a temporary backup path, refer to steps <a href="https://docs.aws.amazon.com/vpn/latest/s2svpn/SetUpVPNConnections.html">here</a>. We continue to closely track progress and will share an update by August 16 9:30 AM PDT, or sooner as new information becomes available.

The electrical isolation at Equinix FR5 in Frankfurt, DEU is now complete and our engineers have begun physically inspecting the impacted network equipment. We do not yet have a timeline for full resolution while we continue to assess the extent of impact to equipment. Direct Connect connections at this location remain impaired. Customers with connections exclusively at this location will continue to experience packet loss. Customers with multi-site or redundant configurations across other Direct Connect locations are not affected. We recommend that impacted customers failover to VPN until we have more clarity on next steps and a recovery timeline. For customers using Direct Connect gateway and Transit Gateway, you can create an AWS Site-to-Site VPN and attach it to your Transit Gateway, refer to steps <a href="https://aws.amazon.com/premiumsupport/knowledge-center/dx-configure-dx-and-vpn-failover-tgw/">here</a>. For other customers, we recommend establishing an AWS Site-to-Site VPN as a temporary backup path, refer to steps <a href="https://docs.aws.amazon.com/vpn/latest/s2svpn/SetUpVPNConnections.html">here</a>. We will provide another update by August 16 5:30 PM PDT, or sooner as new information becomes available.

We have completed our assessment of the impacted network equipment at the Equinix FR5 location in Frankfurt, DEU and now have a clear understanding of the scope of impact. We are making progress toward restoring connectivity and will be taking a phased approach to remediation. Customers with connections exclusively at this location will continue to experience packet loss until remediation is complete. Customers with multi-site or redundant configurations across other Direct Connect locations are not affected. We will provide another update by August 16 10:30 PM PDT, or sooner as new information becomes available.

We continue to make progress in our phased remediation at the Equinix FR5 location in Frankfurt, DEU. Since our last update, some dependent network infrastructure has been restored. Remediation of remaining infrastructure is ongoing, with a portion of recovery dependent on the delivery of replacement hardware. Cooling has been fully restored, with environmental conditions stable within normal operating thresholds. Customers with connections exclusively at this location will continue to experience packet loss as remediation progresses. Customers with multi-site or redundant configurations across other Direct Connect locations are not affected. Previously communicated mitigation guidance and recommendations remain unchanged at this time. We will provide another update by August 17 4:30 AM PDT, or sooner as remediation progresses.

We continue to make progress in our phased remediation at the Equinix FR5 location in Frankfurt, DEU. Network infrastructure and dependent systems continue to improve as we bring affected hardware back online. Some replacement hardware has been delivered and installation is proceeding as components arrive on-site. In parallel, we are shifting network traffic to allow restored devices to begin serving customers as they come online. As we progress through recovery, customers will observe restoration occurring in two stages. In the first stage, BGP sessions will re-establish but IP prefixes will not yet be advertised, this indicates that recovery is still in progress and the underlying infrastructure is not yet ready to carry traffic. In the second stage, IP prefix advertisement will resume, at which point the infrastructure is fully remediated and connectivity is restored. While we do not currently have an ETA for full recovery, we continue to work as quickly and safely as possible to mitigate the impact for customers. We will provide another update by August 17 10:30 AM PDT, or sooner as remediation progresses.

We continue to work on phased remediation incrementally at the Equinix FR5 location in Frankfurt, DEU. We are seeing early signs of recovery while we continue to fully remediate the issue. We are actively working to bring the remaining affected hardware back online and we will provide another update by 12:30 PM PDT, or sooner as remediation progresses.

We are seeing broad signs of recovery at the Equinix FR5 location in Frankfurt, DEU. We have restored connectivity for the majority of affected hardware and most of the connections are fully recovered and stable. There are a small number of customers that will remain affected until the remaining devices are fully restored. We will provide another update by 2:00 PM PDT, or sooner as remediation progresses.

We continue to work on bringing affected hardware back online. Since our last update we have made progress that will not be visible to customers, but is required for recovery. We are working in parallel to bring all devices online as safely as possible. This work is expected to take several hours to complete and validate. For customers that require workarounds, we recommend that you consider failing over to VPN. For customers using Direct Connect gateway and Transit Gateway, you can create an AWS Site-to-Site VPN and attach it to your Transit Gateway, refer to steps <a href="https://aws.amazon.com/premiumsupport/knowledge-center/dx-configure-dx-and-vpn-failover-tgw/">here</a>. For other customers, we recommend establishing an AWS Site-to-Site VPN as a temporary backup path, refer to steps <a href="https://docs.aws.amazon.com/vpn/latest/s2svpn/SetUpVPNConnections.html">here</a>. We will provide another update by 7:00 PM PDT or sooner as new information becomes available.

We are seeing significant recovery for most of the customer connections at this stage. While we are not yet fully recovered, restoration efforts are progressing as expected at the Equinix FR5 location in Frankfurt, DEU. Remediation of remaining infrastructure involves completion of hardware replacements and traffic validation, both of which are actively underway. We anticipate further customer-visible recovery as the remaining infrastructure is brought back into service. Customers with connections exclusively at this location will continue to experience packet loss until remediation is complete. Previously communicated mitigation guidance and recommendations remain unchanged at this time. We will provide another update by August 17 11:00 PM PDT or sooner.

Starting August 14 7:33 PM PDT, we experienced increased packet loss impacting AWS Direct Connect connectivity for customers with connections at the Equinix FR5 location in Frankfurt, DEU. Engineers were automatically engaged at 7:45 PM on August 14 and immediately began investigating mitigations. By 8:30 PM, we identified that network equipment at the FR5 location was impaired due to water ingress into the co-location facility. As a result, cooling system was impaired which resulted in devices overheating and shutting down. Water also affected power distribution systems which disabled the power for the network devices. Initial recovery efforts were delayed as environmental conditions within the facility required stabilization before engineers could safely access the affected area. Throughout August 15 and 16, our engineers worked in coordination with the facility operator to restore impaired network devices while the underlying infrastructure issue was addressed. By 7:26 PM on August 17, all impaired network equipment was successfully restored and connectivity to the location was verified as fully operational with sustained recovery. We do not expect this issue to recur. Customers with redundant connections across other Direct Connect locations maintained connectivity through their alternate paths throughout this event and require no further action. Customers who implemented VPN failover as a workaround may now safely revert to their primary Direct Connect paths. Connectivity has been verified as stable and fully operational. Customers requiring further assistance may contact AWS Support through the AWS Management Console or the <a href="https://console.aws.amazon.com/support">AWS Support Center</a>.

We can confirm elevated network packet loss, impacting AWS Direct Connect connectivity in AP-SOUTH-1 Region. Our engineering team was automatically engaged at 9:54 AM to begin investigating the issue. There are no workarounds available at this time. We will provide another update by 11:30 AM PDT.

We have identified the root cause to be related to a change made to a configuration system responsible for assigning routes to the devices. We have begun the work to reduce the packet loss that is impacting AWS Direct Connect in the AP-SOUTH-1 Region, and we expect recovery to take place gradually over the next 30 minutes. As we gain confidence in these efforts, we will seek to parallelize our efforts to speed recovery. We will provide another update by 12:00 PM PDT.

Between 9:42 AM and 11:44 AM PDT, we experienced elevated network packet loss impacting AWS Direct Connect connectivity in the AP-SOUTH-1 Region. Our engineering team was automatically engaged at 9:46 AM to begin investigating. By 10:51 AM, we understood the root cause to be a configuration change made to a system responsible for assigning routes to devices. As we gained confidence in our mitigation steps, we parallelized our efforts to further reduce the packet loss. The issue is resolved and the service is operating normally.

We are investigating connecivity issues impacting multiple AWS services in the US-WEST-2 Region.

We are seeing initial signs of recovery and continue to work toward full recovery.

We continue to see significant signs of recovery as a result of our mitigation efforts for the connectivity issues impacting multiple AWS services in the US-WEST-2 Region. We have identified the root cause as an issue with a networking device responsible for network routing from the Region to the Seattle Metro. Engineers have finished all mitigation work. As routes continue to be restored, customers should see a continued reduction in error rates and timeouts when connecting to affected services. We are monitoring recovery progress closely and will continue working until all routes have been fully restored and service metrics return to pre-event levels. We will provide another update in the next 30-45 minutes.

Between 3:55 AM and 4:15 AM PDT, we experienced connectivity issues that impacted connectivity to the US-WEST-2 Region. This impacted multiple AWS services in the Region. Some customers may have also experienced issues accessing the AWS Management Console, with connection timeouts and unresponsive pages. Connectivity within the Region was not affected. Our engineers were automatically engaged at 4:01 AM PDT, and immediately began investigating this issue. We identified the root cause as an issue with networking devices responsible for network routing from the Region to the Seattle Metro, and began working in parallel on multiple paths to mitigate the impact. We took mitigation measures that led to initial recovery at 4:15 AM PDT. As the network continued to stabilize following our mitigation actions, a brief reconvergence event occurred between 4:47 AM and 4:59 AM PDT. During this reconvergence period, some customers may have experienced intermittent connectivity issues to the Region as network routes were re-established. By 4:59 AM PDT, all routes had been fully restored and service metrics returned to pre-event levels. Customers using AWS Direct Connect through EqSe2, Westin Building Exchange, Seattle experienced an extended impact window from 3:55 AM to 5:12 AM PDT. These customers would have experienced connectivity issues until network routes for this specific path were fully restored at 5:12 AM PDT. Customers connected redundantly through other AWS Direct Connect locations were not impacted by this event. The issue has been resolved and all AWS services are operating normally.

We are investigating issues with Cost Explorer reflecting inaccurate estimated billing data.

Beginning on July 16 7:38 PM PDT, we began displaying incorrect estimated billing data in the Billing and Cost Management Console. Our engineering teams are engaged and investigating root cause. We will provide another update by 3:00 AM PDT or sooner if more information becomes available.

We continue to work to resolve the issue affecting estimated cost and usage data displayed in the Billing and Cost Management Console. We have identified the root cause as an issue with unit pricing within the estimated billing computation subsystem and we are working on a mitigation. The displayed billing estimates do not reflect actual usage and charges. There are no customer actions required at this time. Once the issue has been mitigated, we expect full resolution to take multiple hours as we work through recomputing the estimated billing data. We will provide another update by 4:00 AM PDT or sooner if more information becomes available.

We continue to work to resolve the issue affecting estimated cost and usage data displayed in the Billing and Cost Management Console. As previously shared, we have identified the root cause as an issue with unit pricing within the estimated billing computation subsystem. To prevent further inaccurate billing estimates from being displayed, we have paused estimated billing computations. Customers who are currently seeing normal bill estimates will continue to see those estimates, and customers who are seeing inflated estimates will not see them increase further while we work toward resolution. The displayed billing estimates do not reflect actual usage and charges. We continue to work on fully mitigating the issue. Once the issue has been mitigated, we expect full resolution to take multiple hours as we work through recomputing the estimated billing data. There are no customer actions required at this time. We will provide another update by 5:00 AM PDT or sooner if more information becomes available.

We continue to work to resolve the issue affecting estimated cost and usage data displayed in the Billing and Cost Management Console. We are actively working on multiple mitigation paths in parallel. The first path involves reverting to the last known good estimated bill computation. With this approach, customers will only see cost and usage data through July 15, however the inflated cost data will be removed. The second path involves rolling back a recent change to the billing computation subsystem. The displayed billing estimates do not reflect actual usage and charges. There are no customer actions required at this time. Once the issue has been mitigated, we expect full resolution to take multiple hours as we work through recomputing the estimated billing data. We will provide another update by 6:00 AM PDT or sooner if more information becomes available.

We continue to work on multiple mitigation paths in parallel to resolve the issue affecting estimated cost and usage data displayed in the Billing and Cost Management Console, including the Cost and Usage Report. We are evaluating resuming estimated billing computations, as our internal monitoring indicates the billing computation subsystem is now producing accurate estimates. We are conducting additional validation before proceeding with this path. The displayed billing estimates do not reflect actual usage and charges. There are no customer actions required at this time. We will provide another update by 8:00 AM PDT or sooner if more information becomes available.

We continue to work to resolve the issue affecting estimated cost and usage data displayed in the Billing and Cost Management Console, including the Cost and Usage Report. The rollback of a recent change did not resolve the issue and we are continuing to investigate multiple mitigation paths. Estimated bill updates remain paused. We are in the process of reverting to the last accurate estimated billing data. The displayed billing estimates do not reflect actual usage and charges. There are no customer actions required at this time. We expect this mitigation to take several hours to complete as we work through recomputing the estimated billing data. We will provide another update by 10:00 AM PDT or sooner if more information becomes available.

We have identified the root cause and mitigated the underlying issue causing incorrect estimated cost and usage data to be displayed in the Billing and Cost Management Console, and Cost and Usage Reports. We have begun backfilling data to correct cost data for all customers. We expect some customers to begin seeing recovery within the next three hours, and full recovery for all customers by July 18 12:00 PM PDT. Until the backfill is complete, some customers may still see incorrect cost and usage data. The displayed billing estimates do not reflect actual usage and charges. There are no customer actions required at this time. We will provide another update by 1:00 PM, or sooner if information becomes available.

Our efforts to backfill corrected estimated cost and usage data are still underway. We are progressing slower than anticipated. While we are seeing some accounts recover with correct cost and usage data, we expect all affected accounts to be recovered by July 19 12:00 AM PDT. Until the backfill is complete, some customers may still see incorrect cost and usage data. The displayed billing estimates do not reflect actual usage and charges. There are no customer actions required at this time. We will provide another update by 7:00 PM, or sooner if information becomes available.

We continue to make steady progress toward resolving the issue affecting estimated cost and usage data displayed in the Billing and Cost Management Console. Our efforts to backfill corrected data remain underway, and we expect all affected accounts to be fully recovered by July 19, 12:00 AM PDT. Until the backfill is complete, some customers may still observe incorrect cost and usage data in the Billing and Cost Management Console and Cost and Usage Reports. These estimates do not reflect actual usage or charges. Customers who configured their Cost and Usage Report with the "Overwrite" option require no action — their report will be automatically updated with corrected data once the backfill completes. Customers who configured their Cost and Usage Report with the "Create new report versions" option retain all previous report deliveries in their S3 bucket. The report version delivered during the impacted window may contain inaccurate data. Once the data backfill is complete, a corrected report version will be delivered under a new assemblyId. Customers using this configuration should update any downstream processes (Athena tables, Redshift pipelines, Amazon QuickSight, or custom ETL) to reference the latest assemblyId for the affected billing period, and may delete or archive the impacted report version to prevent processing stale data. To identify the latest report, customers can follow the steps in our <a href="https://docs.aws.amazon.com/cur/latest/userguide/view-latest-cur.html">documentation</a>. We will provide another update by July 18, 1:00 AM PDT, or sooner if additional information becomes available.

We continue to make substantial progress toward resolving the issue affecting estimated cost and usage data displayed in the Billing and Cost Management Console. Our mitigation efforts are working as expected and we are seeing an increasing number of accounts reflecting correct cost and usage data. We expect all affected accounts to be fully recovered by July 19, 12:00 AM PDT. Until the backfill is complete, some customers may still observe incorrect cost and usage data in the Billing and Cost Management Console and Cost and Usage Reports. These estimates do not reflect actual usage or charges. We will provide another update by July 18, 7:00 AM PDT, or sooner if additional information becomes available.

Between July 16 at 7:38 PM PDT and July 18 at 6:00 AM PDT, we began displaying incorrect estimated billing data in the Billing and Cost Management Console, including the Cost and Usage Report. Customers may have received erroneous budget and cost anomaly detection alerts, and observed inflated estimated cost and usage data. On July 16 at 7:46 PM PDT, our alarms detected cost anomalies but failed to halt the estimated bill generation process or alert our engineering teams. We were alerted to this issue on July 17 at 12:19 AM PDT by customer escalations, and immediately began to investigate. We first informed customers via AWS Health on July 17 at 1:33 AM. At 8:24 AM PDT we paused further updates to estimated billing data and turned off budget and cost anomaly alerts as a precautionary measure. We identified the root cause on July 17 at 12:00 PM PDT as a configuration change in our bill computation system. This system relies on unit conversion data to calculate line item charges. The configuration change caused updates to the unit conversion data to fail, resulting in inflated line item costs, which propagated to the Billing and Cost Management console and triggered budget and cost anomaly alerts. We mitigated the issue on July 17 at 12:30 PM PDT which corrected the unit conversion configuration, and began reprocessing cost and usage data for all customer accounts. We started observing recovery at 4:19 PM PDT, and the majority of accounts were fully recovered by July 18 at 6:00 AM PDT. There are a small number of accounts still processing and we will post updates for these accounts on the Personal Health Dashboard. We have corrected our alarms to immediately halt processing and notify our engineering teams when anomalies occur. We apologize for the alarm this incident caused our customers and are conducting a thorough retrospective to prevent events like this from reoccurring, as well as improve our response when billing incidents occur. The issue has been resolved and all AWS services are now operating normally.

We are investigating increased 5xx errors for Cloudfront customers utilizing VPC Origins connectivity.

Starting at 12:45 AM PDT, we are experiencing increased 5xx errors for CloudFront customers utilizing VPC Origins connectivity. We have confirmed that customers utilizing other origin types are not impacted by this issue. Our engineers are engaged and are actively working to mitigate impact. As a workaround, customers who do not require VPC Origins can change their origin type to resolve the errors. We will provide another update by 3:15 AM PDT, or sooner if more information becomes available.

We continue working to resolve the increased 5xx errors for CloudFront customers utilizing VPC Origins connectivity. Customers utilizing other origin types remain unaffected by this issue. Based on our investigation, we believe the root cause is related to a packet processing subsystem responsible for routing requests from CloudFront's edge locations to resources within customer VPCs. We continue to recommend that customers who are able to do so temporarily change their origin type to resolve the errors. We will provide another update by 4:15 AM PDT, or sooner if additional information becomes available.

We continue working to resolve the increased 5xx errors for CloudFront customers utilizing VPC Origins connectivity. Customers utilizing other origin types remain unaffected by this issue. We have further scoped the issue down to routing table capacity within the packet processing subsystem responsible for routing requests from CloudFront's edge locations to resources within customer VPCs. We have identified and are currently testing a mitigation strategy to resolve the issue. Once testing is complete, we will deploy the mitigation in a phased approach. Based on the results from these tests, we will provide a clearer estimated time for resolution in our next update. We continue to recommend that customers who are able to do so temporarily change their origin type to resolve the errors. We will provide another update by 5:15 AM PDT, or sooner if additional information becomes available.

We are seeing initial signs of recovery and continue to work toward full recovery.

We continue to see significant signs of recovery as a result of our mitigation efforts, with full recovery expected within the next 45 minutes.

Between 12:45 AM and 4:18 AM PDT, we experienced increased 5xx errors for CloudFront customers utilizing VPC Origins connectivity. Our engineers were automatically engaged and immediately began investigating the root cause. By 2:57 AM PDT, we identified the root cause of the issue as an internal constraint on the fleet that manages connections to private VPC origins. When this constraint was reached, the system responsible for distributing routing configuration to our network processors failed to load the updated configuration data correctly, affecting routing of VPC Origin connections. At 3:52 AM PDT, we took multiple mitigation actions that led to to full recovery at 4:18 AM PDT. Now that the issue has been mitigated, customers who temporarily changed their origin type can safely revert these changes. Customers utilizing other origin types were not affected by this issue. The issue has been resolved and the service is operating normally.

We are investigating elevated connectivity issues with a single Avalability zone (euc1-az2) in the EU-CENTRAL-1 Region.

We are seeing early signs of recovery and continue to work toward full resolution. We will continue to provide updates.

Between 2:56 PM and 6:07 PM PDT, we experienced connectivity issues to a subset of EC2 instances in a single Availability Zone (euc1-az2) in the EU-CENTRAL-1 Region. During this time, customers may also have experienced increased error rates and latencies for new instance launches in the affected zone, along with some AWS APIs that use the affected EC2 instances. Some AWS Services also experienced connectivity issues and increased error rates within the affected zone. Engineers were automatically engaged and immediately began investigating. As part of our recovery effort, we shifted traffic away from the impacted Availability Zone for affected services at 3:04 PM. At 3:05 PM we identified the root cause to be a recent networking change causing the impact. Engineers immediately began reverting this change which completed at 4:28 PM. This resulted in restoration of network connectivity to the affected zone at 4:30 PM. We continued to work until we fully recovered the impacts at 6:07 PM. We do not expect this issue to reoccur. The issue has been resolved and the service is operating normally.

We are investigating increased error rates when calling EC2 Launch Template APIs in US-EAST-1 Region. During this time, affected customers may experience errors when creating, modifying, or referencing launch templates. Other AWS services that rely on launch templates may also be impacted. We will provide another update by 6:30 AM PDT or sooner, if we have additional information to share.

Starting at 2:56 AM PDT, we began experiencing increased error rates when calling EC2 Launch Template APIs in the US-EAST-1 Region. Our engineers have been engaged and are actively working to mitigate the impact. Additionally, Amazon Elastic Kubernetes (EKS) customers may experience errors when creating or updating clusters, or when launching and scaling nodes via Managed Node Groups, EKS Auto Mode, or Karpenter; this issue does not impact existing clusters and nodes. We have identified the root cause to be a congestion issue within an EC2 internal subsystem responsible for processing EC2 launch template workflows. We are pursuing multiple mitigation paths. We recommend that customers retry any failed requests during the impact window. While we do not currently have an ETA for full recovery, we are prioritizing this issue and will provide another update by 7:15 AM PDT or sooner if we have additional information to share.

We are seeing initial signs of recovery and continue to work toward full recovery.

Between 2:56 AM and 6:54 AM PDT, we experienced increased error rates when calling EC2 Launch Template APIs in the US-EAST-1 Region. During this time, affected customers may have experienced errors when creating, modifying, or describing Launch Templates. Other AWS services that rely on Launch Templates were also impacted. Amazon EC2 instances and Amazon EKS workloads already running on provisioned nodes continued to operate normally. Cluster modification operations, and Managed Node Group creation were also impacted. For EKS Auto Mode, impact was limited to operations requiring new capacity or changes, including node provisioning and pod scheduling. Our engineers were automatically engaged and immediately began investigating the root cause. We identified the root cause as a congestion issue within an EC2 internal subsystem responsible for processing EC2 launch template workflows. At 3:26 AM PDT, we took mitigation actions by introducing throttling for the affected APIs and we saw some recovery which was communicated directly with a subset of customers via the 'Your Account view' of the AWS Health Dashboard. We took multiple additional mitigation paths, incrementally lifting these throttle limits, and by 6:54 AM PDT, the issue was fully mitigated. We recommend that customers retry any failed requests. The issue has been resolved and all AWS services are now operating normally.

We are investigating increased launch errors and API errors in the EU-NORTH-1 Region. Existing instances are not affected by this issue.

We can confirm increased error rates for the EC2 APIs, as well as errors launching new EC2 instances in the EU-NORTH-1 Region. Other AWS Services that launch new instances or call the EC2 APIs as part of their workflows may also be affected by this issue. During this time, customers may receive an Internal Server Error in the Management Console and APIs. Engineers were automatically engaged and began investigating the issue. We are actively working on identifying the root cause. Existing instances are unaffected by this issue. We will provide an update by 3:15 PM, or sooner if we have additional information to share.

We are seeing early signs of recovery and continue to work toward full recovery.

Between 1:42 PM and 2:25 PM PDT we experienced increased error rates and latencies for EC2 APIs in the EU-NORTH-1 Region. This issue also affected new instance launches. Other AWS Services that launch new instances or call EC2 APIs as part of their workflows were also affected by this issue. Existing EC2 instances were unaffected by this issue. During this time, customers would have received an Internal Server Error in the Management Console and APIs. Engineers were automatically engaged and began investigating the root cause. We identified the root cause as a planned configuration change. This change was reverted and we began observing recovery at 2:19 PM. By 2:25 PM, the issue was fully mitigated. We do not expect this issue to reoccur. Since the issue was mitigated at 2:25 PM, we have been processing a backlog for ELB workflows and expect this backlog to complete within the next 30 minutes. We recommend customers retry requests that failed during this time. The issue has been resolved and all services are operating normally.

To support compliance with the US Government export control directive, Anthropic has asked us to revoke access to Claude Fable 5 and Claude Mythos 5 for all users in all regions. All other models, including Opus 4.8, are not affected and you can continue using them in full confidence. Please view the <a href="https://www.anthropic.com/news/fable-mythos-access">Anthropic statement</a> for further details.

Claude Fable 5 and Claude Mythos 5 models remain unavailable for all users in all regions. We are resolving this Health event. For further details please view the <a href="https://www.anthropic.com/news/fable-mythos-access">Anthropic statement</a>.

Between 5:50 PM and 7:15 PM PDT, we experienced connectivity issues that may have impacted Internet performance for some customers in the SA-EAST-1 Region. During this time, connectivity to instances and services within the Region was not affected. Our engineering team was automatically engaged at 5:51 PM PDT and immediately began investigating the issue. We identified the root cause and implemented a fix, which mitigated the issue at 7:15 PM PDT. The issue has been resolved and the service is operating normally.

We are investigating increased error rates for Route53 API calls.

Between 4:00 PM and 4:46 PM, we experienced increased error rates for the Route53 APIs. This issue did not impact resolution of existing DNS records. Engineers were automatically engaged and immediately began investigating the issue. During this time, customers may have received 500s for Route53 APIs and the Route53 Management Console. We have identified the root cause and have mitigated this issue. Other AWS Services that call the Route53 APIs in their workflows may also have been impacted during this time. We recommend retrying any failed operations or stuck workflows. We do not expect this issue to reoccur. The issue has been resolved and the service is operating normally.

We are investigating instance impairments in a single Availability Zone (use1-az4) in the US-EAST-1 Region. Other Availability Zones are not affected by the event and we are working to resolve the issue.

We continue to investigate instance impairments to a single Availability Zone (use1-az4) in the US-EAST-1 Region. We have experienced an increase in temperatures within a single data center, which in some cases has caused impairments for instances in the Availability Zone. EC2 instances and EBS volumes hosted on impacted hardware are affected by the loss of power during the thermal event. Other AWS services that depend on the affected EC2 instances and EBS volumes in this Availability Zone, may also experience impairments. We will continue to provide updates as recovery continues.

We continue to work towards mitigating the increased temperatures to its normal levels in the affected Availability Zone (use1-az4) in the US-EAST-1 Region. Other AWS services that depend on the affected EC2 instances and EBS volumes in this Availability Zone, may also experience impairments. We have weighed away traffic for most services at this time. We recommend customers utilize one of the other Availability Zones in the US-EAST-1 Region at this time, as existing instances in other AZ's remain unaffected by this issue. Customers may experience longer than usual provisioning times. We will provide an update by 7:45 PM PDT, or sooner if we have additional information to share.

We are actively working to restore temperatures to normal levels in the affected Availability Zone (use1-az4) in the US-EAST-1 Region, though progress is slower than originally anticipated. Since our last update we have made incremental progress to restore cooling systems within the affected AZ, which will not be visible to external customers but are required for the restoration of affected services. In the impacted Availability Zone, EC2 Instances, EBS Volumes, and other AWS Services are also experiencing elevated error rates and latencies for some workflows. As part of our recovery effort, we have shifted traffic away from the impacted Availability Zone for most services. We recommend customers utilize one of the other Availability Zones in the US-EAST-1 Region, as existing instances in other AZs remain unaffected by this issue. If immediate recovery is required, we recommend customers restore from EBS Snapshots and/or replace affected resources by launching new replacement resources in one of the unaffected zones. We will provide an update by 10:00 PM PDT, or sooner if we have additional information to share.

We are observing early signs of recovery. We continue to work towards restoring temperatures to normal levels and bring impacted racks back online in the affected Availability Zone (use1-az4) in the US-EAST-1 Region. We have been able to get additional cooling system capacity online, which has allowed us to recover some affected racks and are actively working to recover additional racks in a controlled and safe manner. In the impacted Availability Zone, EC2 Instances, EBS Volumes, and other AWS Services may continue to experience elevated error rates and latencies for some workflows until full recovery is achieved. We will provide an update by 11:30 PM PDT, or sooner if we have additional information to share.

We continue to make progress in resolving the impaired EC2 instances in the affected Availability Zone (use1-az4) in the US-EAST-1 Region, and are working towards full recovery. We are actively working to bring additional cooling system capacity online, which will enable us to recover the remaining affected racks in a controlled and safe manner. In the impacted Availability Zone, EC2 Instances, EBS Volumes, and other AWS Services may continue to experience elevated error rates and latencies for some workflows. Customers will continue to see some of their affected EC2 instances and EBS volumes as impaired until we achieve full recovery. We will provide an update by May 8, 1:30 AM PDT, or sooner if we have additional information to share.

Mitigation efforts remain underway to resolve the impaired EC2 instances and degraded EBS volumes in a single Availability Zone (use1-az4) in the US-EAST-1 Region. These EC2 instances and EBS volumes were impacted due to a loss of power during the thermal event. The work to bring additional cooling system capacity online, which will enable us to recover the remaining affected infrastructure in a controlled and safe manner, is taking longer than we had initially anticipated. Some services, such as IoT Core, ELB, NAT Gateway, and Redshift, have seen significant improvements in the recovery of their workflows. However, some customers will continue to see their affected EC2 instances and EBS volumes as impaired until we achieve full recovery. While we do not currently have an ETA for full recovery, we are prioritizing this issue and will provide another update by 3:30 AM PDT or sooner if additional information becomes available.

We continue to make progress towards resolving the impaired EC2 instances and degraded EBS volumes in a single Availability Zone (use1-az4) in the US-EAST-1 Region. At this time, we wanted to provide some more details on the issue. Beginning on May 7 at 4:20 PM PDT, we began experiencing an increase in instance impairments within the affected zone due to the loss of power during a thermal event. Engineers were automatically engaged within minutes and immediately began investigating multiple mitigations. By 9:12 PM PDT, we restored power to a subset of the affected infrastructure and observed some signs of recovery, which have remained stable. We continue working to bring additional cooling system capacity online, which will enable us to recover the remaining affected hardware in the impacted zone in a controlled and safe manner. Some AWS services, such as IoT Core, ELB, NAT Gateway, and Redshift, continue to see significant improvements in the recovery of their workflows. However, some customers will continue to see their affected EC2 instances and EBS volumes as impaired until we achieve full recovery. If immediate recovery is required, we recommend customers restore from EBS snapshots and/or replace affected resources by launching new replacement resources in one of the unaffected zones. Based on our current mitigation efforts, we expect full recovery to take several hours. We are prioritizing this issue and will provide another update by 6:30 AM PDT or sooner if additional information becomes available.

We continue working to resolve the impaired EC2 instances and degraded EBS volumes in a single Availability Zone (use1-az4) in the US-EAST-1 Region caused by a thermal event. During such an event, servers automatically shut down when the temperatures exceeded the operating thresholds in order to protect the hardware. We are actively working to bring additional cooling system capacity online, which will enable us to recover the remaining affected hardware in the impacted zone. Some customers will continue to see their affected EC2 instances and EBS volumes as impaired until we achieve full recovery. If immediate recovery is required, we recommend customers restore from EBS snapshots and/or replace affected resources by launching new replacement resources in one of the unaffected zones. In parallel, we are investigating increased error rates and query failures for Redshift clusters in the US-EAST-1 Region. During this time, affected customers may see errors for resume and restart workflows, as well as failover operations and availability issues. Our engineers are actively working to resolve this issue. Full recovery is still expected to take several hours. We are prioritizing this issue and will provide another update by 9:00 AM PDT or sooner if additional information becomes available.

We continue our efforts to work towards the recovery of the impaired EC2 instances and degraded EBS volumes in a single Availability Zone (use1-az4) in the US-EAST-1 Region. We are making progress towards the restoration of the cooling system capacity that is required to recover the affected hardware in the impacted zone. Some customers will continue to see their affected EC2 instances and EBS volumes as impaired until the affected racks are recovered. We continue to recommend that customers who require immediate recovery restore from EBS snapshots and/or replace affected resources by launching new replacement resources in one of the unaffected zones. As part of our parallel investigation, we have identified the root cause of the increased error rates and query failures for Redshift clusters in the US-EAST-1 Region. This has been confirmed to be related to impact from an upstream dependency. Affected customers may continue to see errors for resume and restart workflows, failover operations, and impact to general availability. We are actively working to resolve the issue. Our timeline for full recovery is still expected to take several hours and will be incremental as we bring racks online in phases. We will provide an additional update by 12:30 PM or sooner if we have new information to provide.

We have observed complete recovery of increased error rates and query failures for Redshift clusters in the US-EAST-1 Region. We were able to resolve the impact independently of the ongoing efforts to recover the affected hardware in the use1-az4 Availability Zone. The issue affecting Redshift has been resolved and the service is operating normally. We will provide an additional update regarding the efforts towards hardware restoration by 12:30 PM or sooner.

We are experiencing an increase in timeouts to Amazon Managed Streaming for Apache Kafka partitions on a subset of clusters as a result of the ongoing issue in a single Availability Zone (use1-az4) in the US-EAST-1 Region. We are working in parallel to determine a path towards mitigation for affected clusters. We will provide an additional update by 12:30 PM or sooner.

We continue to work towards the recovery of the impaired EC2 instances and degraded EBS volumes in a single Availability Zone (use1-az4) in the US-EAST-1 Region though efforts are slower than we had previously anticipated. We are taking measured steps to ensure that cooling capacity is brought online in a safe and controlled manner. As a result, EBS Volumes and EC2 instances affected by the issue will continue to experience impairments. We continue to recommend that customers who require immediate recovery restore from EBS snapshots and/or replace affected resourced by launching new replacement resources. Full recovery is still expected to take several hours. We will provide an additional update by 4:00 PM or sooner if we have new information to provide.

We have begun to see improvements in the overall number of affected EC2 instances and degraded EBS volumes in a single Availability Zone (use1-az4) in the US-EAST-1 Region. The steps taken to supply additional cooling capacity have been showing steady signs of progress. Some EBS Volumes and EC2 instances affected by the issue will continue to experience impairments while we continue to drive these efforts. We continue to recommend that customers who require immediate recovery restore from EBS snapshots and/or replace affected resources by launching new replacement resources. In parallel, we have seen some improvements in Amazon Managed Streaming for Apache Kafka as a result of the parallel mitigation efforts being performed. We are still experiencing timeouts to partitions but are seeing continued progress. We do anticipate that recovery will still take several hours. We will provide an additional update by 7:30 PM or sooner if we have new information to provide.

Starting May 7 4:20 PM PDT, we experienced increased impaired EC2 instances and degraded EBS volumes in a single facility (data center) within a single Availability Zone (use1-az4) in the US-EAST-1 Region. The issue was caused by a thermal event resulting in a loss of power. As part of our recovery effort, we shifted traffic away from the impacted Availability Zone for most services at May 7 5:06 PM. AWS services, like Elastic Load Balancing, Elastic Kubernetes Service, ElastiCache, Redshift, OpenSearch, Managed Streaming for Apache Kafka among others, that depend on the affected EC2 instances and EBS volumes in this Availability Zone, also experienced elevated error rates and latencies for some workflows and/or configurations. Our main effort during the event mitigation strategy was to bring back our cooling systems capacity. By May 8 1:50 PM, we were able to stabilize cooling system capacity to pre-event levels, which helped us to restore the majority of the impaired EC2 instances and EBS volumes. A small number of instances and EBS volumes remain impaired and we continue to work to recover all affected remaining resources. We will communicate with customers who are still impacted via the Your Account view of the AWS Health Dashboard. Customers that require further assistance with this event may contact AWS Support through the AWS Management Console or the AWS Support Center.

We are investigating instance connectivity issues in a single Availability Zone (euw3-az2) in the EU-WEST-3 Region.

Between 3:58 AM and 4:40 AM PDT, we experienced increased error rates and increased launch failures for EC2 instances in a single Availability Zone (euw3-az2) in the EU-WEST-3 Region. During this time, customers attempting to launch new EC2 instances in the affected Availability Zone would have experienced launch failures. Additionally, a subset of existing EC2 instances and EBS volumes in this Availability Zone were impacted and became unreachable. We have identified the root cause to be a loss of power to infrastructure within the affected Availability Zone. Engineers were engaged at 4:02 AM and immediately began working to restore power and assess the scope of impact. By 4:20 AM, power was successfully restored to the affected infrastructure. We then focused our efforts on recovering impacted EC2 instances and EBS volumes. By 4:40 AM, all impacted EC2 instances and EBS volumes had been fully recovered and were operating normally. No additional action is required for EC2 instances and EBS volumes that were impacted during the power loss event, as these have been fully recovered. While EC2 and EBS have recovered, some AWS services may take additional time to fully recover as they process backlogs and complete their own recovery procedures. The issue has been resolved and the service is operating normally.

We are investigating increased error rates in the EU-CENTRAL-2 Region.

We can confirm substantial error rates for PUT and GET requests to Amazon S3 in the EU-CENTRAL-2 Region. Engineers engaged immediately based on automated alarming. We have triangulated the issue to a subsystem responsible for assembling objects from bytes in storage. We have begun implementing mitigations, and are observing some improvement in error rates. We continue to work to identify the root cause, and are working on multiple parallel paths to fully mitigate the issue. Other AWS Services (such as EC2 launches) that rely on S3 are also affected by this issue. Existing EC2 instances are unaffected by this issue. We will provide another update by 12:45 PM PST, or sooner if we have additional information to share.

We are seeing early signs of recovery and continue to monitor and work toward full recovery.

Between 11:27 AM and 12:20 PM PST we experienced substantial error rates for S3 PUT/GET requests in EU-CENTRAL-2 Region. Engineers were engaged immediately based on automated alarming. We identified the root cause as an issue with a subsystem responsible for assembling objects bytes in storage. At 12:04 PM PST, we implemented mitigations and began observing early signs of recovery for S3. Error rates continued to improve, and other AWS Services continued to recover until 12:50 PM PST when we observed full recovery. We continue to work toward backfilling Cloudwatch logs, and expect that to continue over the next couple hours. We recommend customers retry any failed requests. The issue has been resolved and all services are operating normally.

We are investigating increased API error rates in a single Availability Zone (mes1-az2) in the ME-SOUTH-1 Region.

We are investigating connectivity and power issues affecting APIs and instances in a single Availability Zone (mes1-az2) in the ME-SOUTH-1 Region due to a localized power issue. Existing instances in this zone will also be affected. Other AWS Services may also be experiencing increased errors and latencies for their workflows, and we are working to route requests away from this affected Availability Zone. We recommend customers make use of other Availability Zones at this time. During this time, we are also experiencing delays in propagating DNS changes for Route53 to pops (Points of Presence) in ME-SOUTH-1.  Targeting new launches using RunInstances in the remaining AZs should succeed. Existing instances in the other AZs are not affected.

We continue to work on a localized power issue affecting a single Availability Zone (mes1-az2) in the ME-SOUTH-1 Region. In the impacted Availability Zone, EC2 Instances, DB Instances, EBS Volumes, and other AWS Services are also experiencing elevated error rates and latencies for some workflows. As part of our recovery effort, we have shifted traffic away from the impacted Availability Zone for most services. We recommend customers utilize one of the other Availability Zones in the ME-SOUTH-1 Region, as existing instances in other AZs remain unaffected by this issue. We are actively working to restore power and connectivity, at which time we will begin recovering affected resources. Currently, we expect recovery to take many hours. We will provide an update by 2:30 AM PST, or sooner if we have additional information to share.

We continue to work toward restoring power in the affected Availability Zone (mes1-az2) in the ME-SOUTH-1 Region. At this time, some AWS services have shifted traffic away from the affected Availability Zone and are seeing recovery for their affected operations and workflows. EC2 Instances, EBS Volumes, and other resources impacted in the affected Availability Zone will require a longer recovery timeline. Power has not yet been restored to the affected Availability Zone. If immediate recovery is required, we recommend customers restore from EBS Snapshots and/or launch replacement resources in one of the unaffected Availability Zones or an alternate Region. In parallel, we are actively working on reducing the error rates and latencies that some customers are experiencing with EC2 APIs. For now, we recommend continuing to retry any failed API requests. We will provide an update by 6:00 AM PST on March 2, or sooner if we have additional information to share.

We continue to work toward restoring power in the impacted Availability Zone (mes1-az2) in the ME-SOUTH-1 Region. Meanwhile, EC2 instance and networking APIs have been restored for the other Availability Zones. Additionally, we have made improvements to the availability of RDS multi-AZ databases while operating with the impaired Availability Zone. These improvements will help customers create database exports to preserve data, and we recommend customers with databases in the affected Availability Zone consider creating exports as a precautionary measure. EC2 Instances, EBS Volumes, and other resources impacted in the affected Availability Zone will require a longer recovery timeline, as power has not yet been restored. We are expecting recovery to take at least a day, as it requires repair of facilities, cooling and power systems, coordination with local authorities, and careful assessment to ensure the safety of our operators. If immediate recovery is required, we recommend customers restore from EBS Snapshots and/or launch replacement resources in one of the unaffected Availability Zones or an alternate AWS Region. We will provide an update by 11:00 AM PST on March 2, or sooner if we have additional information to share.

We continue to work towards restoring power in the affected Availability Zone (mes1-az2) in the ME-SOUTH-1 Region. We currently expect our recovery efforts to take at least a day. Our current guidance regarding immediate recovery remains unchanged from our previous update. Customers are able to disassociate Elastic IP addresses from resources in the affected Availability Zone and associate those with resources in the unaffected Availability Zones. This can be done by specifying --allow-reassociation when attempting to associate the Elastic IP to the new resource. We will provide you with further updates by 2:00 PM PST or sooner if new information becomes available.

We continue to work towards restoring power in the affected Availability Zone (mes1-az2) in the ME-SOUTH-1 Region. We have no updated guidance on expected recovery times, and still expect this to take at least a day to fully restore power and connectivity. We continue to advise customers to launch replacement resources in one of the unaffected Availability Zones or an alternate AWS Region. At this time we recommend that customers that are capable of backing up data outside of the region consider doing so. You can view the current status of affected AWS services below. We will provide you with another update by 7:00 PM PST, or sooner if we have additional information to share.

We are providing an update on the ongoing service disruptions affecting the AWS Middle East (UAE) Region (ME-CENTRAL-1) and the AWS Middle East (Bahrain) Region (ME-SOUTH-1). Due to the ongoing conflict in the Middle East, both affected regions have experienced physical impacts to infrastructure as a result of drone strikes. In the UAE, two of our facilities were directly struck, while in Bahrain, a drone strike in close proximity to one of our facilities caused physical impacts to our infrastructure. These strikes have caused structural damage, disrupted power delivery to our infrastructure, and in some cases required fire suppression activities that resulted in additional water damage. We are working closely with local authorities and prioritizing the safety of our personnel throughout our recovery efforts. In the ME-CENTRAL-1 (UAE) Region, two of our three Availability Zones (mec1-az2 and mec1-az3) remain significantly impaired. The third Availability Zone (mec1-az1) continues to operate normally, though some services have experienced indirect impact due to dependencies on the affected zones. In the ME-SOUTH-1 (Bahrain) Region, one facility has been impacted. Across both regions, customers are experiencing elevated error rates and degraded availability for services including Amazon EC2, Amazon S3, Amazon DynamoDB, AWS Lambda, Amazon Kinesis, Amazon CloudWatch, Amazon RDS, and the AWS Management Console and CLI. We are working to restore full service availability as quickly as possible, though we expect recovery to be prolonged given the nature of the physical damage involved. In parallel with efforts to restore the physical infrastructure at the affected sites, we are pursuing multiple software-based recovery paths that do not depend on the underlying facilities being fully brought back online. For Amazon S3 and Amazon DynamoDB, we are actively working to restore data access and service availability through software mitigations, including deploying updates to enable S3 to operate within the current infrastructure constraints and remediating impaired DynamoDB tables to restore read and write availability for dependent services. Our focus on restoring these foundational services is deliberate, as recovery of Amazon S3 and Amazon DynamoDB will in turn enable a broad range of dependent AWS services to recover. For other affected service APIs, we are deploying targeted software updates to reduce error rates and restore functionality where possible, independent of the physical recovery timeline. We are also working to restore access to the AWS Management Console and CLI through network-level changes that route traffic away from the affected infrastructure. While these software-based mitigations can address many of the service-level impacts, some recovery actions are constrained by the physical state of the affected facilities — meaning that full restoration of certain services will require the underlying infrastructure to be repaired and brought back online. Across all services, our teams are working in parallel on both the physical restoration of the affected facilities and these software-based mitigations, with the goal of restoring as much customer access as possible as quickly as possible, even ahead of full infrastructure recovery. In addition, we are prioritizing the restoration of services and tools that enable customers to back up and migrate their data and applications out of the affected regions. Finally, even as we work to restore these facilities, the ongoing conflict in the region means that the broader operating environment in the Middle East remains unpredictable. We recommend that customers with workloads running in the Middle East consider taking action now to backup data and potentially migrate your workloads to alternate AWS Regions. We recommend customers exercise their disaster recovery plans, recover from remote backups stored in other regions, and update their applications to direct traffic away from the affected regions. For customers requiring guidance on alternate regions, we recommend considering AWS Regions in the United States, Europe, or Asia Pacific, as appropriate for your latency and data residency requirements. We will continue to provide updates as recovery progresses and as the situation evolves. Our next update will be provided by 9:00 PM PST on March 2, 2026, or sooner if new information becomes available.

We continue to work towards restoring power in the affected Availability Zone (mes1-az2) in the ME-SOUTH-1 Region. We have no updated guidance on expected recovery times, and still expect this to take at least a day to fully restore power and connectivity. AWS infrastructure is designed to be highly resilient, but given the uncertainty of the current situation, we encourage our customers to replicate Amazon S3 and critical data from the ME-SOUTH-1 Region to another AWS Region. For customers requiring guidance on alternate regions, we recommend considering AWS Regions in the United States, Europe, or Asia Pacific, as appropriate for your latency and data residency requirements. We will provide another update by March 3 at 3:00 AM PST, or sooner if new information becomes available. For more information on Cross-Region Replication, refer [1]. For more information on S3 Batch Replication, see [2]. For a simple script to quickly set up and start S3 Replication, see [3]. If you have questions or concerns, please contact AWS Support [4]. [1] <a href="https://docs.aws.amazon.com/AmazonS3/latest/userguide/replication.html">https://docs.aws.amazon.com/AmazonS3/latest/userguide/replication.html</a> [2] <a href="https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-batch-replication-batch.html">https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-batch-replication-batch.html</a> [3] <a href="https://github.com/awslabs/aws-support-tools/blob/master/S3/Setup_Replication/setup_replication.py">https://github.com/awslabs/aws-support-tools/blob/master/S3/Setup_Replication/setup_replication.py</a> [4] <a href="https://aws.amazon.com/support">https://aws.amazon.com/support</a>

We continue to work toward restoring power in the affected Availability Zone (mes1-az2) in the ME-SOUTH-1 Region. The overall state of the region remains largely unchanged from our previous update. At this time, we have no updated guidance on expected timelines for fully restoring power and connectivity. We are taking all necessary steps to support the recovery process. While progress is being made, significant work remains before full restoration is complete. Given the ongoing uncertainty, we encourage customers to replicate their Amazon S3 data and other critical data from the ME-SOUTH-1 Region to another AWS Region, using the guidance provided in our previous update. We will continue to provide updates as recovery progresses and as the situation evolves. Our next update will be provided by 6:00 AM PST on March 3, or sooner if new information becomes available.

Recovery efforts in the affected Availability Zone (mes1-az2) in the ME-SOUTH-1 Region are ongoing, with the situation remaining consistent with our last update. We have no change to expected timelines for fully restoring power and connectivity. While progress is being made, significant work remains before full restoration is complete. We continue to recommend customers launch replacement resources in one of the unaffected Availability Zones or an alternate AWS Region. Given the extended nature of this event, we continue to encourage customers to replicate Amazon S3 data and other critical workloads from ME-SOUTH-1 to another AWS Region using the guidance shared previously. We will provide our next update by 12:00 PM PST on March 3, or sooner if conditions change.

We are providing an update on the ongoing service disruptions affecting the AWS Middle East (Bahrain) Region (ME-SOUTH-1). We continue to make progress on recovery efforts across multiple workstreams. With the immediate phase of this event now better understood, we are moving to a more targeted communication model. Going forward, updates will be delivered directly to affected customers through the AWS Personal Health Dashboard. Customers who require assistance with this event are encouraged to contact AWS Support through the AWS Management Console or the AWS Support Center. We continue to strongly recommend that customers with workloads running in the Middle East take action now to migrate those workloads to alternate AWS Regions. Customers should enact their disaster recovery plans, recover from remote backups stored in other Regions, and update their applications to direct traffic away from the affected Regions. For customers requiring guidance on alternate regions, we recommend considering AWS Regions in the United States, Europe, or Asia Pacific, as appropriate for your latency and data residency requirements.

We are providing an update on the ongoing service disruption. The Middle East (Bahrain) Region (ME-SOUTH-1) has suffered damage due to the conflict in the Middle East and is currently unavailable. Customers should recover their resources in other Regions from remote backups. Relevant billing operations are currently suspended while we restore normal operations in this AWS Region. This process is expected to take several months.

We are providing an update on the disruption affecting the Middle East (Bahrain) (me-south-1) Region. The damage to our infrastructure spanned multiple Availability Zones and exceeded what our regional and multi-AZ services are designed to withstand. After a thorough assessment, we have determined that we are unable to restore access to the resources and data hosted exclusively in this Region. When the first Availability Zone was damaged in March, we began recommending that customers migrate their workloads to other Regions, and most did so before the Region became unavailable following the disruption of a second Availability Zone in April. Since then, we have supported the remaining customers in re-establishing their operations in alternate Regions, using backups where available or implementing alternative solutions to mitigate the impact of inaccessible data. In parallel, we assessed all affected infrastructure and exhausted every option for restoring data and resources that had not been migrated before the Region became unavailable. We remain committed to supporting our customers in Bahrain in the long term and will share a further update in early 2027. We have notified the relevant authorities and continue to work with them toward that goal.