58 Cloudamqp incidents · 2024年12月 — official updates, affected components, duration and resolution details.
We're investigating connectivity issues affecting some servers in Azure West Europe. We'll post an update as soon as we know more.
Connectivity to servers in southeast Asia has been restored.
Google europe-west2 のサーバーに影響を与える接続の問題を調査しています。 弊社よりお知りになりましたら、早速アップデートをお送りいたします.
今後もこの問題について調査を続けてまいります.
事件は、EURPE-west2-bのサーバーに限定されています
GCE はまだ問題が報告されていますが、API 呼び出しが成功しています.
GCEは、EUPE-west2-bの停電が混乱を引き起こしたことを確認しました。 パワーの問題は緩和され、影響を受けたワークロードの完全回復は、今日13時UTCによって暫定的に期待されています。 回復を監視し続けます.
今後の問題がないか引き続き監視しています。
この事件は解決しました.
公式のインシデント更新を自動翻訳しています。
Azure West Europeのサーバーに影響する接続の問題について調査しています。 弊社よりお知りになりましたら、早速アップデートをお送りいたします.
Azureはこの問題を認識し、積極的に調査しています。 弊社よりお知りになりましたら、別のアップデートをお送りいたします.
Azureはこの問題を特定しました。 彼らは地域に影響を与えるストレージアクセシビリティの問題を狭くしました。 彼らのチームは、影響を受けるインフラの回復アクションを積極的に実行し、根本的な原因を調査し続けています。 回復の進捗を監視し、影響を受けたリソースを復元します.
Azure West Europe のインスタンスで通信を確立できるようになりました。 インスタンスのメジャー性をバックアップして実行します。 Azure はまだ解決する VM がいくつかあります.
すべてのCloudAMQPインスタンスがAzure West Europeで稼働しています.
**概要** CloudAMQPは10月9日のAzure西ヨーロッパ空室状況ゾーン2のクラスターのサブセットとの接続を失いました。 AZ2の単一ノードクラスターは、インシデントに最も影響を受けました。 **インパクト** お客様が、インシデント中に発生したデータやマージノードにアクセスできませんでした。 Azureからのイベントのタイムライン。 * 12:25 UTC に 09 10月 2026 – 私たちは、内部サービス監視アラートを介して、この問題を識別しました. * 12:29 UTC 上 09 10月 2026 – 顧客への影響が始まりました, 影響を受けた西欧ストレージスケールユニットで検出されたVMディスクの故障. * 2026年10月09日に2:30 UTC – 複数のストレージノードと障害ドメインを横断したハートビートの調査は、ストレージスケールユニットが歪みモードに入るようにしました。 * 15:32 UTC に 09 10月 2026 - ストレージ スケール ユニットを歪みモードから取り出し、衝撃されたストレージ サーバーにディスクを分離して残りのディスクの回復を有効にします。 この段階では、影響を受けた顧客のほとんどが回復を見始めました。 ※21:00 UTCは10月2026日(土)にデータセンターおよびストレージエンジニアが複数のストレージノードでリカバリプランを実行し続けました。 * 23:27 UTC 上 09 10月 2026 – 影響を受けたストレージノードを回復する努力が成功しました, そして、仮想ディスクを回復するために働き続け. サブシーケントリカバリアクションには、ノードレベルの回復と影響を受けたストレージリソースの復元が含まれます。 ※10月2026日(水)に3:00 UTC – サービスの機能性が回復し、顧客への影響が緩和されました。 **ルート原因** Azureは、西ヨーロッパ地域AZ 2のAzure Virtual Machineストレージディスクに影響をもたらすプラットフォームの問題を経験しました。 影響を受けたお客様は、地域に失敗した状態を入力する、接続障害、予期しない再起動、または仮想マシンを経験している可能性があります。 **決断** マシンアクセスはAzureで約3時間復元されました。 CloudAMQP は、インパクトのあるノードで通信を復元できるようになりました。 Azureからアップデートを監視し、ノードが影響しなくなったことを保証しました.
公式のインシデント更新を自動翻訳しています。
一部の非アジュールネットワークとAzure UAE North(azure-arm::uaenorth)のネットワーク接続を劣化させ、上流トランジットプロバイダでルーティングの問題を引き起こしています。 地域内のすべてのサーバーは、メッセージの処理、監視、メトリック、ログの処理が正常に行われています。 このルーティングの問題の影響を受けているネットワークからクライアントが接続する顧客は、接続タイムアウトを見ることができます。 提供者に問題が報告され、その周辺の管理トラフィックをルーティングしています.
ルーティングはノーマルに戻ります.
この事件は解決しました.
公式のインシデント更新を自動翻訳しています。
現在、この問題を調査中です.
問題が特定され、修正が実装されています.
修正を実装し、結果を監視しています.
この事件は解決しました.
公式のインシデント更新を自動翻訳しています。
DigitalOceanは、現在、APIとコントロールパネルの問題が発生しており、問題が主張している間、エラーとレイテンシが高まります
この事件は解決しました.
公式のインシデント更新を自動翻訳しています。
Azure West US にデプロイされた複数のサーバーについて、通知するさまざまなアラートを受け取りました。 明らかに、サーバーの25%に影響を与えるこの領域に固有の関係があるようです.
Microsoft Azureは、西米地域にネットワークの問題があることを確認しました。 現在、ほとんどのサーバーが運用中です。 下記の時間帯にモニタリングを続けます.
この事件は解決しました.
公式のインシデント更新を自動翻訳しています。
We are currently investigating this issue.
The issue has been identified and a fix is being implemented.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
**Summary** Between June 25 and June 26, our monitoring system incorrectly reported some dedicated servers as unreachable and sent "server down" notifications for servers that were in fact operating normally. This was a monitoring-side issue only — no customer servers or services were actually down, and no data or message delivery was affected. **Impact** Affected customers received one or more alerts indicating their dedicated server was down. These alerts were false positives. The servers themselves remained healthy and fully available throughout the incident. **Root cause** Before connecting to a server, our monitoring system performs a network authorization step. A defect in how one component handled a transient network hiccup could leave a monitoring process unable to complete that step, after which it failed to connect to the servers it was responsible for checking. The servers themselves stayed healthy and available throughout — the monitoring process had simply lost its ability to reach them, and reported them as down. **Resolution** We identified the affected monitoring processes, confirmed the root cause in production, and deployed a fix that makes this authorization step resilient to such transient failures and able to recover automatically. After deploying, we monitored the system to confirm that notifications returned to normal. We apologize for the confusion these false notifications may have caused.
Between 17:30 and 20:00 UTC on 2026-05-30, our server monitoring system generated a large number of false "server_unreachable" alerts. Clusters were operational and healthy throughout this period — no customer data or message delivery was affected. The alerts were caused by a rolling restart of our monitoring service, during which individual monitoring workers went offline mid-cycle and lost state. This caused the workers to report connection timeouts for servers they could no longer reach — even though those servers were fully healthy. We apologize for any concern this may have caused.
This incident has been resolved.
We are investigating connectivity issues affecting some CloudAMQP instances in Azure West Europe. Affected instances may see connection failures or unexpected restarts. The root cause is an ongoing Microsoft Azure platform incident in the West Europe region (started 14:31 UTC, 23 May 2026). Microsoft is investigating.
Resolved — Microsoft has confirmed that the Azure West Europe Virtual Machines incident is resolved. The impact window was 14:09–14:13 UTC on 23 May 2026, during which a limited number of customers may have experienced connection failures or unexpected Virtual Machine restarts. The Azure environment self-healed and the service has been confirmed restored. Affected CloudAMQP instances should now be operating normally.
Since around 8pm UTC last night, some metrics are not being forwarded to legacy integration. Prometheus integrations not affected. We are investingating.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
# Legacy host metrics integrations degraded Incident window: 2026-05-21 18:37 UTC – 2026-05-22 04:05 UTC \(9h 28m\) Affected service: Legacy metric integrations \(CloudWatch, Datadog, Librato, New Relic, Splunk, Stackdriver\). Host metrics affected, not broker metrics, nor was internal monitoring or console graphs. ## Summary For ~9.5 hours, the service that collects host-level metrics \(CPU, memory, disk, network\) for legacy third-party integrations entered a crash loop and stopped shipping data. ## Root Cause An internal credential-signing service was migrated to a new container runtime the afternoon before. Due to a bug with environment variables the collector could not renew it's credentials. The collector's existing credential remained valid for several hours, so the failure only surfaced when it tried to renew. ## Resolution On-call staff detected the failure early morning, developers helped restore the service and collector recovered at 04:05 UTC. ## Prevention * All services, both signing and collector report failed authentication attempts earlier and with higher severity * Metrics pipeline alarm thresholds was tightened so a similar drop in data triggers alarms faster.
We are currently experiencing issues with our backend services handling account and server creation. This does not affect any running customer servers but might delay provisioning new servers.
The issue has been identified and a fix is being implemented.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We are currently experiencing issues with our backend services handling account and server creation. This does not affect any running customer servers but might delay provisioning new servers.
A fix has been implemented and we are monitoring the results.
Still seeing slow requests, doing another investigation.
Looks like we're back at full performance now.
This incident has been resolved.
We are currently investigating this issue.
Servers in Azure West Europe can experience connection issues due to underlying issues at provider. Latest update from Azure "Current Status: We continue to investigate an availability issue caused by a subset of unhealthy backend storage infrastructure supporting Azure Virtual Machines and Azure Direct Drive in West Europe."
This incident has been resolved.
We are currently experiencing issues with our backend services handling account and server creation. This does not affect any running customer servers but might delay provisioning new servers.
The issue has been identified and a fix is being implemented.
A fix has been implemented and we are monitoring the results.
We had an abnormal increase in traffic causing timeouts and slow responses for customer.cloudamqp.com, we increased capacity and are back to normal. This did not affect any running clusters, just provisioning and configuration updates.
We are getting indications of instances being down in Azure region westus. We are investigating the issue.
This incident has been resolved.
Servers in Azure West US 3 can experience connection issues due to underlying issues at provider. We monitor and will update when we know more.
This incident has been resolved.
There appears to be an issue regarding sending metrics to all Legacy metrics integrations. Alarms based on broker metrics also got affected. We are currently investigating the issue Prometheus based metrics (Datadog v3 etc) not affected.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
Broker metrics and metrics integrations degradation Window: 2026-04-23 03:25 – 05:35 UTC \(2h 10m\) ## Impact Broker metrics \(connections, channels, queues, consumers, message rates, node and netsplit state\) were degraded, progressively dropping until recovery at 05:35 UTC. Downstream effects: * Queue, Consumer, Connection, Channel, Connection flow, and Netsplit alarms: threshold breaches during the window may not have triggered, or triggered late. Alarms already tripped before 03:25 UTC kept their state. * Metrics Integrations \(Datadog, CloudWatch, New Relic, Splunk, Dynatrace, etc.\): delivery of broker metrics was reduced for the duration of the window. Alarms configured on the receiving side may likewise have missed or lagged. Not affected: server metrics \(CPU, memory, disk\) and their corresponding alarms, Notice alarms, broker availability, and the metric graphs shown in the CloudAMQP console \(these use a separate data path and continued to update normally\). ## Timeline \(UTC\) * 03:25 — broker-metrics collection begins degrading * 03:35 — ~30% of broker-metric samples missing * 03:50 — ~50% plateau * 05:25 — collector workers cycled and re-initialised * 05:35 — full throughput restored ## Root cause Broker metrics are polled from each cluster's management HTTP API by a pool of collector workers and published to an internal message bus that both the Alarms service and Metrics Integrations consume from. At 03:25 UTC this service became unresponsive without raising an error. Affected workers silently stopped polling while remaining alive to the platform, so automatic restart did not trigger. On-call staff began investigating around 05:00 UTC; force-restarting the service restored metric flow. The exact trigger has not been identified. Our focus is on ensuring the condition is detected and handled promptly if it recurs. ## What we are changing * Added metrics and alarms specific to this service, with high-urgency paging for on-call. * Reliability improvements in the service itself to detect and restart stalled work. ## What you can do CloudAMQP offers a new generation of metrics integrations based on Prometheus, with a re-engineered pipeline that does not rely on these centralised services — each server forwards data directly to your endpoint. These have been running in production for some time and have proved very reliable. * [https://www.cloudamqp.com/blog/prometheus-metrics-integrations.html](https://www.cloudamqp.com/blog/prometheus-metrics-integrations.html) * [https://www.cloudamqp.com/docs/monitoring\_metrics\_datadog\_v3.html](https://www.cloudamqp.com/docs/monitoring_metrics_datadog_v3.html) * Background: [https://www.cloudamqp.com/blog/decentralized-observability-with-open-telemetry-part-1.html](https://www.cloudamqp.com/blog/decentralized-observability-with-open-telemetry-part-1.html)
One of our backend components suffered a small outage due to an unexpected full services restart. Connectivity with the servers were broken and service triggered alarms for many clusters. We apologize for the inconvenience. All clusters were operational and healthy within the incident/alarms timespan.
Customers are experiencing Management UI being unavailable with 502 Bad Gateway. Timeline: - 14:16 UTC -> The issue was identified - 14:30 UTC -> Recent code changes were reverted. No relationship with incident noted.
The Management UI and brokers are serving traffic normally. You can still use the Management UI by using the Broker URL and properly typing User and Password. The impact is related to our backend service logs into Management UI via CloudAMQP SSO.
We are continuing to work on a fix for this issue.
The Management UI SSO shortcut is back to service. Impacted Clusters: All customers with clusters created more than 6 months ago Cause: Some code changes unexpectedly changed the communication schema between our backend services and the SSO component on the clusters. The proxy configuration conflicted with those changes causing the 502 Bad Gateway error to show up. Impact: Only the SSO via CloudAMQP Console was impacted given it required communication with our backend components. The AMQP Clusters were healthy and servicing traffic via both AMQP and HTTP on Management UI and API. Logging into the Management UI via Credentials and all Authentications Backends was functioning normally.
