55 Cockroach Labs incidents · 2023年6月 — official updates, affected components, duration and resolution details.
全球气候方案欧洲西部-2-b和供应区目前正面临问题。 一旦掌握了更多信息,我们将继续监测这一局势并更新现状.
全球气候方案已缓解了造成欧洲-西2 区域这种中断的电力事件,并正在努力减轻受影响的服务。 一旦确认缓解措施,我们将予以更新.
GCP缓解了造成欧洲-西2地区这种干扰的动力事件,而大多数GCP服务恢复了. 我们将继续跟踪并确保受影响的CRL服务在解决之前恢复正常运作.
GCP缓解了造成欧洲-西2区这种干扰的动力事件. 所有GCP和CRL服务都恢复正常运行. 无需采取进一步行动.
自动翻译自官方事件更新。
目前,我们正在监测全球合作进程我们-中央1区域的情况(缺乏能力)。 如果可能,建议客户在创建新集群时使用不同的区域. 预计,随着能力的具备,集群创建和编辑将最终取得成功。 全球协商进程没有为何时能解决这些能力问题提供时间表,但我们将继续在了解更多情况时监测和更新这一页.
我们正在继续监测这些库存情况,并将在新信息出现时公布最新情况.
我们正在继续监测这一问题;然而,Google无法提供一份关于何时解决的ETA。 我们将继续同他们合作,以获得必要的能力。 仍建议客户在旋转新的集群时选择另一个区域来规避我们中央1组织观察到的能力问题.
自动翻译自官方事件更新。
设在南美洲全球气候方案(圣保罗/南美洲-东1)的CockroachDB Cloud(基本)集群目前正面临网络连接问题。 我们正在调查,不久将提供最新情况
这一事件已经得到解决。 GCP南美洲(圣保罗/南美洲-东1)的CockroachDB Cloud(基本)集群经历了网络连接问题,现已完全恢复。 连接已经恢复正常,所有受影响的集群正在按预期运行。 我们为这可能造成的破坏道歉。 如果您继续遇到任何问题,请联系支援.
自动翻译自官方事件更新。
Google Cloud正在经历网络中断, 影响着Cockroach Cloud的客户。 Cockroach Cloud 客户可能会遇到更长时间、延误或连接问题。 我们的小组了解并充分参与了这一事件。 已知当前影响 : 东一号
我们仍然在监测局势,在过去15分钟中看到一些改善,但是没有完全解决。 已知当前影响 : europe-west-4-b 欧统局
Google现在报告说,我们中央一级和东部一级都解决了,但欧洲西部4-b仍然受到积极的影响。 在过去15分钟里,我们的内部小组正在确认我们中部和东部地区没有进一步的错误。 已知当前影响 : 欧洲西部4-b
我们得知Google正在报告我们-中央1-b也受到了影响并正在经历退化。 在网络一级缓解努力之后,基本的网络基础设施已完全恢复。 整个Cloud的Google产品工程团队目前正在验证单个服务的恢复状态.
谷歌的影响现已报告为已完全解决。 内部小组已经确认了这份报告。 所有服务恢复。 需要帮助时,请联系支援部门办理罚单.
自动翻译自官方事件更新。
We are investigating an issue affecting a subset of Free Tier clusters in eu-west-1, where some clusters may be unable to connect or execute queries. Teams have been notified and are actively working on mitigation.
We are investigating an issue affecting a subset of Basic clusters in eu-west1, where some clusters may be unable to connect or execute queries. Teams have been notified and are actively working on mitigation.
We are investigating an issue affecting a subset of Basic and Standard clusters in eu-west1, where some clusters may be unable to connect or execute queries. Teams have been notified and are actively working on mitigation.
Teams have mitigated the immediate active impact, but work continues to ensure the problem does not recur and to fully understand the root cause. For now we will continue to monitor the situation. If you continue to experience impacts, please reach out to Support.
Following an extended period of monitoring, the team has confirmed that the cluster remains healthy and stable.
We are currently aware of and investigating a cluster creation failure. Teams are engaged and actively working to mitigate as quickly as possible.
The cluster creation issue has been restored. Issue is resolved.
We have identified a minor issue affecting Cloud customers who are unable to accept invites, preventing them from inviting new users to their organizations. Internal teams are aware and a fix is in flight. ETA: 30min - 1hour. We will provide the next update at the top of the hour or prior if the fix is deployed before that time.
Fix deployment is complete, tested and verified.
AWS is experiencing elevated error rates and degraded performance in a single availability zone in the us-east-1 region. CockroachDB clusters in us-east-1 remain available, and we are monitoring the situation and taking corrective action as appropriate to ensure SQL availability of CockroachDB clusters in this region.
AWS's latest status update reports that they are continuing to work on the situation, and progress is slow. AWS has advised a next update time at 05:00 UTC. We continue to monitor the situation and take appropriate action to ensure uptime and availability for Cockroach DB clusters. We'll next update by about 05:00 UTC when we learn more from AWS.
AWS reports continued progress towards resolving the issue in the US-EAST-1 Region and is working towards full recovery. Affected services may continue to experience elevated error rates and latencies until recovery is complete. We continue to monitor the situation and take appropriate action to ensure uptime and availability for CockroachDB clusters. We'll provide another update as we learn more from AWS.
All CockroachDB clusters in the affected AWS availability zone have been restored to a fully operational state. We will continue to monitor the upstream incident from AWS, but do not expect any impact to CockroachDB clusters.
Social logins for customers using personal/non-SSO Microsoft accounts is currently unavailable. Our teams are aware and currently investigating.
Social login for Microsoft accounts is now available. Please reach out to support if you continue to experience issues.
A power outage at an AWS facility in the me-central-1 (Mid-East) region may cause failures or slower than usual progress on cluster operations. AWS is aware of the issue and is working to restore power. We are monitoring the situation, and will provide updates as we receive them from AWS.
Current conflict in the region has led to objects impacting the AWS data center creating a fire earlier today. Power was shut down to prevent further damage. AWS is working on multiple mitigation paths to restore services as quickly as possible. We will provide the next update by 4PM PST or sooner if we have more to share.
AWS is reporting positive signs of recovery at this time, but are still working on multiple mitigation paths to fully restore services. We continue to keep a close eye on the situation and will provide another update by 6PM PST, or sooner if we have additional information to share.
AWS is reporting significant signs of recovery for specific APIs and requests, but several still remain to be mitigated and will likely take several hours more to complete, if not into tomorrow. Our teams have begun testing with this update but have reported we are still impacted at this time. We will continue to monitor the situation closely and will provide one more update this evening by 7PM PST, or sooner if we have additional information to share.
AWS continues to work on mitigation efforts, including stabilizing impacted APIs and implementing interim workarounds. Our teams are actively reviewing and validating these mitigations; however, this effort will continue into tomorrow. We will provide our next update at 7:00 AM PST, or sooner if additional information becomes available.
me-central-1 region is impacted more with two AZs out and the clusters in that region are unavailable. In addition to this, me-south-1 region is also experiencing an AZ outage, which causes degraded performance of clusters in me-south-1. We are closely tracking AWS's status. Impacted customers can track AWS status here https://health.aws.amazon.com/health/status
AWS me-central-1 region is impacted with two AZs out. Clusters in that region are currently unavailable. AWS me-south-1 region is also experiencing a single AZ outage, which causes degraded performance of clusters in that region. While no impact has been observed on GCP or Azure in those regions, customers with single-region deployments are advised to have their disaster recovery plans ready should the need to restore from a backup to another region be required. We are closely tracking AWS's status. Impacted customers can track AWS status here https://health.aws.amazon.com/health/status.
Beginning March 1, 2026, a power outage caused by physical damage to AWS infrastructure affected multiple Availability Zones in the AWS me-central-1 (UAE) region, resulting in the unavailability of CockroachDB Cloud clusters in that region. A single Availability Zone in me-south-1 was also impacted, causing degraded performance for clusters in that region. While AWS has not yet fully restored power to all affected Availability Zones, Cockroach Labs has directly engaged all customers impacted by this incident, providing resolution guidance and hands-on remediation assistance, including cross-region data restoration where applicable. As remediation steps have been completed or provided to all directly impacted customers, we are resolving this incident from Cockroach Labs' perspective. Cluster creation in the affected regions remains restricted pending full AWS recovery. Customers with outstanding questions or concerns are encouraged to contact Cockroach Labs Support.
We are investigating connectivity issues to Standard and Basic clusters on AWS.
We have resolved an internal misconfiguration, and see that Standard and Basic clusters are functioning normally again. We will continue to monitor to ensure that the error does not recur.
Standard and Basic clusters continue to function normally.
We have identified an issue that is causing Advanced cluster creation on AWS to take longer than usual. We are preparing a fix, and will update this status page with more information when we have an ETA for resolution.
We are preparing a fix for deployment, which should restore AWS cluster creation times to normal levels. We will update the status page when the fix is deployed, or by 14:30 US/Eastern with more information.
We are continuing to work on the fix for this issue. Next update will be at 15:00 US/Eastern.
We have deployed and verified a fix. Advanced clusters should now create within normal time frames.
We are investigating errors that are causing delays in creation of Basic and Standard clusters on AWS. Basic and Standard cluster creation on GCP is operating normally at this time.
We are investigating errors with Basic and Standard clusters on AWS. Cluster creation, certain cluster edit operations (such as adding a region), and certain CockroachDB Cloud Console functionality (such as listing databases or users) are also impacted. Basic and Standard clusters on GCP are operating normally at this time.
We have identified the cause of cluster creation failures impacting Basic and Standard AWS clusters. As a precautionary measure, while we apply a fix, we have disabled all cluster creation in Cockroach Cloud. We will restore cluster creation when it is safe to do so. There is no SQL availability impact to any existing clusters. We will update here by 3:00 PM US/Eastern with more information.
We are continuing to work on a fix for this issue. Next update will be by 4:00 PM US/Eastern time.
We are continuing to work on a fix for this issue. Next update will be by 4:30 PM US/Eastern time.
We have resolved the underlying issue that was preventing cluster creation and other cluster operations. We have re-enabled cluster creation on all clouds. We will monitor the situation to ensure that everything continues to operate normally.
All cluster operations are working normally.
We are investigating errors that cause creation for Advanced clusters to take longer than usual. Standard and Basic cluster creation is operating normally.
We have applied a mitigation and observe that cluster creations are proceeding again. We will monitor all impacted cluster creations to ensure they succeed.
Cluster creation is operating normally, and all delayed clusters have finished creation.
We’re currently impacted by an ongoing AWS outage in the us-east-1 region. Customers may experience failures or delays when creating new clusters, adding nodes, or running backups in this region. Existing clusters remain operational, but backup operations may be delayed or fail intermittently. We’re continuing to monitor AWS recovery efforts and will provide updates as more information becomes available.
AWS has resolved the underlying outage in the us-east-1 region, and we’re seeing recovery across CockroachDB Cloud operations. Cluster creation, scaling, and backups are now succeeding again. We’re continuing to monitor for full stability before marking this incident as resolved.
The earlier AWS outage in the us-east-1 region has been fully resolved. All CockroachDB Cloud operations, including cluster creation, scaling, and backups, are functioning normally again. We’ve verified recovery across affected systems and customer clusters, and normal operations have resumed.
We are currently experiencing an incident impacting certain cluster operations on Google Cloud Platform due to an known ongoing Google Kubernetes-related issue. Customers may encounter errors when creating or editing clusters. Our engineering teams are actively monitoring the situation and communicating with Google regarding resolution.
Google has reported the issue has been resolved and we've verified cluster operations are functional.
We are observing an incident impacting SQL availability across multiple Serverless clusters. We are currently investigating the issue.
Correction from the original status update. This issue appears to have been isolated to cluster crl-prod-6rw. We have identified the cause, mitigated and confirmed that cluster availability has returned. We are currently completing additional validations.
We have confirmed resolution of the incident impacting SQL availability for the affected cluster. SQL availability is restored and the associated cluster is now healthy.
We are aware of an ongoing Incident with dependent service "Let's Encrypt": https://letsencrypt.status.io/, which may impact cluster creation and add region operations for CockroachDB Cloud Advanced clusters. We will update this status page with more information as it becomes available.
LetsEncrypt has indicated the issue has been mitigated. We have validated cluster creation behaviour is now returning to normal.
We are investigating an issue that may prevent or delay cluster operations from succeeding.
We have corrected a misconfiguration which was preventing cluster operations from succeeding. We are monitoring the success of cluster operations already in progress and new operations.
We are investigating cluster operations, including creating or editing clusters, that are taking longer than usual.
We have identified the issue preventing cluster operations from proceeding, and we see that operations are moving forward. We will next update this incident by 16:00 EDT.
We continue to see cluster operations succeeding, and will actively monitor to ensure the issue does not reoccur.
All cluster operations are succeeding normally at this time.
