ingestion errors in mimir-prod-66
- identified
The issue has been identified, we are working on a fix
- identified
read path is also affected.
- resolved
the issue is resolved now, we're watching the cell
165 recorded Grafana incidents since March 2026, with official updates, affected components, duration and resolution information.
The issue has been identified, we are working on a fix
read path is also affected.
the issue is resolved now, we're watching the cell
Starting at approximately 12:37 UTC, a some Hosted Grafana instances across multiple regions have been experiencing availability issues. Our team has identified the root cause and a fix is currently in progress. Affected instances will recover automatically once the fix has been applied. No action is required on your end. We will provide further updates as they become available and apologize for any disruption this may cause.
We are continuing to work on a fix for this issue.
A fix as been implemented and Hosted Grafana instances are back to normal levels. We will continue to monitor the issue.
This incident has been resolved.
We are investigating elevated latency and intermittent errors when ingesting traces for Grafana Cloud Traces in AWS Germany. Customers may see delayed trace ingestion and failed trace writes. Next update by 10:30 UTC.
Between 09:31 and 10:13 UTC on 23 October 2026 (42 minutes), Grafana Cloud Traces in AWS Germany experienced elevated ingestion latency and intermittent trace write failures, affecting both direct trace ingestion and ingestion via the OpenTelemetry endpoint. All services are now operating normally.
We are currently investigating this issue impacting some GCK6 app project pages. We are in the process of providing a fix for this.
We are continuing to work on a fix for the issue.
This incident has been resolved.
Since approximately 20:30 UTC on October 20, we have observed an increase in alert rule evaluation errors affecting rules that query Grafana Cloud Prometheus (Mimir) data sources in Cluster AWS Germany. Our engineering team is actively investigating and working on remediation.
The root cause of the issue has been indentified and we are working on the remediation steps.
Remediation measures have been applied, and we are monitoring the recovery.
This incident has been resolved.
As of 21 Oct, 06:20 am UTC, we became aware of an issue with possible incorrect evaluation of Synthetic Monitoring alert rules in cluster prod-us-east-3. The issue has been ended 07:00 am UTC
We are currently investigating an issue in the prod-sa-east-1 region causing slow requests, or some stacks struggling to start. We will provide an update as soon as we have more information to share.
Synthetic Monitoring alerting is also impacted. We are continuing to investigate this issue.
Synthetic Monitoring alerting has recovered. We are continuing to monitor.
This incident has been resolved.
We are currently investigating this issue impacting some GCK6 app project pages. We will provide an update once we have more information.
This incident has been resolved.
We are currently investigating this issue.
We are continuing to investigate this issue.
Only the public API is affected for prod-eu-west-0
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
As of 06:30 UTC, we became aware of to an issue with Grafana Cloud unavailibity in the prod-gb-south-0 region. Engineering is actively engaged and assessing the issue. We will provide updates accordingly.
We are continuing to investigate this issue.
Between 06:15 and 12:15 UTC, Grafana Cloud Frontend Observability in GCP UK (prod-gb-south-0) experienced errors accepting incoming telemetry data. This has been resolved and Frontend Observability is operating normally in this region. We are continuing to investigate and monitor the remaining impacted services.
We are seeing errors in Frontend o11y again. We are continuing to investigate and monitor the impacted services.
This incident has been resolved.
We are currently investigating an issue causing elevated read and ruler errors. We will provide another update as soon as we have more to share.
Errors have stopped since 15:15 UTC. We are continuing to monitor.
This incident has been resolved.
We’re currently investigating an issue affecting metrics reads and rule evaluations in mimir-prod-66. Our team is actively working to identify the cause. Thank you for your patience.
We are continuing to investigate this issue.
A fix has been implemented and we are monitoring the results.
We are continuing to monitor for any further issues.
This incident has been resolved.
We are investigating Intermittent write failures in tempo-prod (prod-us-central-0) due to Google Cloud Storage internal errors.
Things are starting to recover, and we continue to monitor the issue.
This incident has been resolved.
We observed some spikes in latency starting at approximately 12:50 UTC. We are working on a resolution. Next update by 15:30 UTC
We are continuing to work on a fix for this issue. Next update by 16:30 UTC
Things are starting to stabilize as we continue to work on a mitigation.
This incident has been resolved.
We are investigating increased authentication errors affecting Grafana Cloud in the prod-us-west-0 region. Customers may experience intermittent failures for requests that require authentication. Our engineering team has identified database performance degradation affecting the authentication service and is investigating the underlying cause. We will provide another update as more information becomes available.
The issue has been identified, and we have applied mitigations on our end. We are currently observing the progress.
We have noticed improvements across services. We will now be applying a more permanent fix. Impacted users may experience a disruption in the auth-api service of up to 15min while we apply a DB patch that will improve performance.
The fix has been implemented, and we are monitoring the results.
This incident has been resolved.
A small number of customers with stacks in prod-us-central-4 are receiving "forbidden" errors when querying Prometheus data sources. We are investigating and will provide an update soon.
We are continuing to investigate this issue.
We have put some mitigations in place which seem to have helped with the errors. We continue to monitor the results.
This incident has been resolved. Thank you for your patience.
Phone call notifications to numbers in Russia and Kazakhstan are currently blocked. We are working to resolve the issue. As SMS support is limited in these countries, we recommend using mobile push notifications as an alternative. We do not currently have an estimated resolution time. We will provide further updates as more information becomes available.
Phone call notifications to numbers in Russia and Kazakhstan are still blocked. We are working to resolve the issue. As SMS support is limited in these countries, we recommend using mobile push notifications as an alternative. We do not currently have an estimated resolution time. We will provide further updates as more information becomes available.
Following further investigation, we've confirmed that SMS delivery to certain destinations is not currently supported. We've updated our documentation to clarify the supported destinations and available notification options. We recommend using mobile push notifications or phone calls as alternatives where SMS delivery is unavailable.
We are investigating an issue affecting metrics ingestion for Grafana Cloud Metrics customers in AWS US East (VA). Customers may see failed or delayed writes. Next update by 10:15 UTC.
We are continuing to investigate an issue affecting metrics ingestion for Grafana Cloud Metrics customers in AWS US East (VA). Customers may see failed or delayed writes. Next update by 11:00 UTC.
We are continuing to investigate an issue affecting metrics ingestion for Grafana Cloud Metrics customers in AWS US East (VA). Customers may see failed or delayed writes to Grafana Cloud Metrics, this may also impact Metrics written by Frontend Observability and/or Synthetic Monitoring. Next update by 12:00 UTC.
We have identified the cause of an issue affecting metrics ingestion for Grafana Cloud Metrics customers in AWS US East (VA) - prod-us-east-3. Customers may see failed or delayed writes to Grafana Cloud Metrics, this may also impact Metrics written by Frontend Observability and/or Synthetic Monitoring. We are continuing to work on mitigations and are monitoring the impact to Grafana Cloud Metrics. Next update by 13:00 UTC.
We have identified the cause of an issue affecting metrics ingestion for Grafana Cloud Metrics customers in AWS US East (VA) - prod-us-east-3. Customers may see failed or delayed writes to Grafana Cloud Metrics, this may also impact Metrics written by Frontend Observability and/or Synthetic Monitoring. We are continuing to work on mitigations and are monitoring the impact to Grafana Cloud Metrics. Next update by 14:00 UTC.
We have isolated infrastructure suspected of contributing to the metrics ingestion issue in AWS US East (VA) - prod-us-east-3. We are validating recovery and completing mitigation steps. Customers may continue to experience failed or delayed metric writes while this work continues.
From 09:45 UTC on 2026-10-12, customers in AWS US East (VA) experienced failed or delayed metric writes to Grafana Cloud Metrics, which may have also affected Frontend Observability and Synthetic Monitoring data in that region. This has been fully resolved and all services are operating normally again.
Between 14:40 and 15:00 UTC on 2026-10-12 (approximately 20 minutes), customers in AWS US East (OH) - prod-us-east-0, may have experienced failed metric writes to Grafana Cloud Metrics. This has been fully resolved and all services are operating normally.
Between 09:38 and 10:45 UTC on 9 October 2026, customers using Private Data Source Connect (PDC) in the AWS US East (OH) - prod-us-east-2 Grafana Cloud Region, experienced failed or timed-out data source queries. This has been fully resolved and all services are operating normally.