Signalfx Us1 outage history

We are investigating an issue with the AWS CloudWatch integration. Some customers may see missing or outdated resource tags on AWS metrics, and metric stream configuration changes may not be applied. Metric ingestion is not impacted. We will provide an update as soon as more information is available.

We have identified that the degraded AWS CloudWatch integration is caused by an ongoing AWS disruption in the me-central-1 (UAE) region, including AWS CloudWatch and the Resource Groups Tagging API. Customers with AWS resources in this region may see missing or outdated resource tags on AWS metrics, and metric stream configuration changes may not be applied. Metric ingestion is not impacted. AWS has not provided a recovery timeline and recommends migrating to another region.

Splunk has confirmed that the degraded AWS CloudWatch integration in the me-central-1 (UAE) region is caused by an ongoing AWS regional disruption outside of Splunk's control. AWS has not provided a recovery timeline and recommends migrating affected resources to another region. Metric ingestion is not impacted. We are identifying affected customers and will reach out individually to discuss next steps. If you have immediate concerns, please contact your account team.

A degradation in the performance of the Splunk APM data ingestion pipeline is causing the processing and storage of raw trace data to be delayed by more than five minutes. No data is being lost at this time and MetricSets are not impacted but the most recent data may not be available in trace search results.

We are continuing to work on a fix for this issue.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

We are investigating a degradation of the Splunk Observability Cloud web application for some customers. Data ingest is not impacted at this time. We will provide an update as soon as possible.

Our engineering teams are actively troubleshooting the identified performance problems in the Splunk Observability Web UI, we will share a new update as soon as possible.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

Charts for a subset of customers may not be loading. Datapoint ingest is not affected. We are investigating and will provide an update shortly.

Charts and dashboards are in a healthy state and we are monitoring the recovery.

This incident has been resolved.

We are investigating delays in processing Splunk Observability Synthetic Monitoring test results. Synthetic Monitoring tests are operational but metrics may be missing until the issue is resolved.

We are continuing to investigate this issue.

The issue has been identified and a fix is being implemented.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

Some customers may be experiencing delays in some charts and detectors. Datapoint ingest is not affected. We are investigating and will provide an update shortly.

We continue investigating the delays experienced in some charts and detectors. Our engineering teams continue troubleshooting and will share an update shortly.

A fix has been implemented and we are monitoring the results.

Service recovery has been confirmed. We will continue to monitor system performance closely and share updates accordingly.

Our engineering team has confirmed the fix deployed has stabilized impacted services across US1 region. This incident is now considered resolved.

From 5:45pm to 6:20pm PT, degradation in the performance of the Splunk APM trace processing pipeline was causing Troubleshooting MetricSets to be delayed by more than five minutes. As a result, the APM Troubleshooting experience, service maps and Tag Spotlight do not have access to the most recent data. The processing of metrics for Business Workflows, which also depends on this pipeline, are equally delayed. Trace data ingest is not impacted at this time; service-level and endpoint-level Monitoring MetricSets and the detectors built from them are also not impacted.

Some of the customer are seeing login failures due to certificate errors. Customers that are already logged are not impacted. The issue has been resolved.

Beginning at 5:50 AM PDT, datapoint ingestion for the archival tier was impacted, resulting in a small number of datapoints not being accepted. The impact was limited to the archival tier. Detectors and charts were not affected. We have implemented remediation and are actively monitoring the system to ensure continued stability. We will provide updates every 15 minutes.

As of 7:48 AM PDT, we are no longer observing any data being dropped. The system is continuing to recover, and we are actively monitoring it to ensure continued stability.

The system is continuing to recover, and we are actively monitoring it to ensure continued stability.

The system is showing steady progress in its recovery. Our teams remain actively engaged, closely observing system metrics to ensure continued stability

We have successfully restored datapoint ingestion and are seeing steady progress as the data backlog is processed. We are actively monitoring this process to ensure full service functionality returns as quickly as possible.

We continue monitoring to ensure full service functionality.

We are continuing to monitor the system to ensure ongoing stability. Our teams remain actively engaged in observing performance metrics to confirm that all operations are functioning as expected.

To accelerate the recovery process, we are currently adding additional capacity to our clusters. Our teams remain actively engaged and are continuing to monitor system performance as it stabilizes.

The system has recovered and we ingesting archived data with no delay.

Datapoint ingest is affected and we are dropping datapoints some datapoints from 8:52 PDT. We are investigating and will provide an update every 15 mins.

Between 8:52 PDT and 9:47 PDT, a small number of data points for new time series were not accepted. As a result, new time series may have experienced a delay before becoming available in charting and detectors. Remediation is in place, and we are continuing to monitor the system to ensure stability.

The issue has been resolved, and datapoint ingestion is operating normally. We have confirmed that the system remains stable following remediation. This incident is now resolved.

A degradation of the trace data ingest path for Splunk APM is causing some trace spans to be dropped and lost. We are investigating the issue and will provide an update as soon as possible.

This incident has been resolved.

We are investigating a major outage of the Splunk Observability AI Assistant.

The issue has been identified as an Azure OpenAI outage and we are proceeding with a workaround.

A fix has been implemented and we are monitoring the results.

Monitoring phase complete. The issue has been resolved.

From 08/25/2026, 11:38AM PDT, a degradation in the performance of the Splunk APM trace processing pipeline is causing Troubleshooting MetricSets to be delayed by more than five minutes. As a result, the APM Troubleshooting experience, service maps and Tag Spotlight do not have access to the most recent data. The processing of metrics for Business Workflows, which also depends on this pipeline, are equally delayed. Trace data ingest is not impacted at this time; service-level and endpoint-level Monitoring MetricSets and the detectors built from them are also not impacted.

We are continuing to investigate this issue.

The issue has been identified and a fix is being implemented.

A fix has been implemented and we are monitoring the results.

Our teams continue monitoring the situation to ensure full remediation.

This incident has been resolved.

Problems with processing alerts which are leading to delayed notifications. We are currently investigating the issue.

We are currently investigating elevated notification latency. Customers may experience delays in receiving alert notifications. We are actively working to resolve this issue and will provide updates as we make progress. We appreciate your patience.

We have identified and are actively resolving the issue causing delayed alert notifications. The fix has been implemented and is already showing significant improvement in notification delivery times. While some notifications that were queued before our fix may still experience minor delays as the system catches up, new notifications are being delivered normally. We appreciate your patience and will continue monitoring the situation closely.

We are actively working on resolving the issue. Customers may continue to experience delays in notification delivery. This includes newly generated notifications, which are being processed behind earlier events in the queue. Team is actively investigating the root cause and working to clear the backlog of in-queue notifications. As a temporary mitigation measure, we have disabled image previews in notifications to improve processing speed. Notifications will be text-only for now. We appreciate your patience and will provide further updates as progress is made.

We have implemented a fix to address the notification delays. Our monitoring shows significant improvement in notification delivery times, and we are closely tracking performance to ensure the issue is fully resolved. Thank you for your patience as we work to restore full service.

The notification delivery delays have been fully resolved. All queued notifications have been processed and the system is now operating normally.

A degradation in the performance of the Splunk APM metrics processing pipeline is causing Monitoring MetricSets to be delayed by more than five minutes. Trace data ingest is not impacted, but service, endpoint and workflow dashboards, and other charts and detectors built from Monitoring MetricSets are impacted.

We are continuing to investigate the issue.

We are continuing to investigate this issue.

The issue has been identified and a fix is being implemented.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

Datapoint ingest for new or reactivated metric time series was affected between 2:28 PM PST to 2:47 PM PST. There was partial data loss for new metric time series during that time period. This situation is now recovered.

Datapoint ingest for new metric time series was affected between 10:37 AM PST and 10:39 AM PST. There was partial data loss for new metric time series during that time period. This situation is now recovered.

Starting at 4a PT, we are investigating a major outage of the Splunk Observability Cloud web application. Data ingest is not impacted at this time. We will provide an update as soon as possible.

We are continuing to investigate the issue.

A fix has been implemented and we are monitoring the results.

This incident has been resolved. Please note that the Splunk Observability Cloud API was also affected during this incident.

Between 1a PDT and 2:50a PDT, any GCP integration that had the "Sync all (dynamically) listed projects" setting enabled had some of the Cloud Monitoring data from GCP dropped.  This issue has been resolved.  Customers may see gaps in charts and detectors that use affected GCP Cloud Monitoring metrics. AWS CloudWatch and Azure Monitor metrics are not impacted.

Customers may experience delays in seeing newly created metric time series. Datapoint ingest is not affected for existing time series. We are investigating and will provide an update shortly.

The issue causing delays in newly created metric time series has been resolved. Metric time series creation is operating normally. Datapoint ingestion for existing time series was not affected.