Signalfx Us1 outage history

We are investigating a major outage of the Splunk Observability AI Assistant.

The issue has been identified as an Azure OpenAI outage and we are proceeding with a workaround.

A fix has been implemented and we are monitoring the results.

Monitoring phase complete. The issue has been resolved.

From 08/25/2026, 11:38AM PDT, a degradation in the performance of the Splunk APM trace processing pipeline is causing Troubleshooting MetricSets to be delayed by more than five minutes. As a result, the APM Troubleshooting experience, service maps and Tag Spotlight do not have access to the most recent data. The processing of metrics for Business Workflows, which also depends on this pipeline, are equally delayed. Trace data ingest is not impacted at this time; service-level and endpoint-level Monitoring MetricSets and the detectors built from them are also not impacted.

We are continuing to investigate this issue.

The issue has been identified and a fix is being implemented.

A fix has been implemented and we are monitoring the results.

Our teams continue monitoring the situation to ensure full remediation.

This incident has been resolved.

Problems with processing alerts which are leading to delayed notifications. We are currently investigating the issue.

We are currently investigating elevated notification latency. Customers may experience delays in receiving alert notifications. We are actively working to resolve this issue and will provide updates as we make progress. We appreciate your patience.

We have identified and are actively resolving the issue causing delayed alert notifications. The fix has been implemented and is already showing significant improvement in notification delivery times. While some notifications that were queued before our fix may still experience minor delays as the system catches up, new notifications are being delivered normally. We appreciate your patience and will continue monitoring the situation closely.

We are actively working on resolving the issue. Customers may continue to experience delays in notification delivery. This includes newly generated notifications, which are being processed behind earlier events in the queue. Team is actively investigating the root cause and working to clear the backlog of in-queue notifications. As a temporary mitigation measure, we have disabled image previews in notifications to improve processing speed. Notifications will be text-only for now. We appreciate your patience and will provide further updates as progress is made.

We have implemented a fix to address the notification delays. Our monitoring shows significant improvement in notification delivery times, and we are closely tracking performance to ensure the issue is fully resolved. Thank you for your patience as we work to restore full service.

The notification delivery delays have been fully resolved. All queued notifications have been processed and the system is now operating normally.

A degradation in the performance of the Splunk APM metrics processing pipeline is causing Monitoring MetricSets to be delayed by more than five minutes. Trace data ingest is not impacted, but service, endpoint and workflow dashboards, and other charts and detectors built from Monitoring MetricSets are impacted.

We are continuing to investigate the issue.

We are continuing to investigate this issue.

The issue has been identified and a fix is being implemented.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

Datapoint ingest for new or reactivated metric time series was affected between 2:28 PM PST to 2:47 PM PST. There was partial data loss for new metric time series during that time period. This situation is now recovered.

Datapoint ingest for new metric time series was affected between 10:37 AM PST and 10:39 AM PST. There was partial data loss for new metric time series during that time period. This situation is now recovered.

Starting at 4a PT, we are investigating a major outage of the Splunk Observability Cloud web application. Data ingest is not impacted at this time. We will provide an update as soon as possible.

We are continuing to investigate the issue.

A fix has been implemented and we are monitoring the results.

This incident has been resolved. Please note that the Splunk Observability Cloud API was also affected during this incident.

Between 1a PDT and 2:50a PDT, any GCP integration that had the "Sync all (dynamically) listed projects" setting enabled had some of the Cloud Monitoring data from GCP dropped.  This issue has been resolved.  Customers may see gaps in charts and detectors that use affected GCP Cloud Monitoring metrics. AWS CloudWatch and Azure Monitor metrics are not impacted.

Customers may experience delays in seeing newly created metric time series. Datapoint ingest is not affected for existing time series. We are investigating and will provide an update shortly.

The issue causing delays in newly created metric time series has been resolved. Metric time series creation is operating normally. Datapoint ingestion for existing time series was not affected.

We have confirmed that the issue affecting Tag Spotlight has been resolved. Customers may have experienced limited functionality within Tag Spotlight beginning at approximately 4:30 PM PT on June 30. Full functionality was restored at 7:20 AM PT on July 1.

Due to a regression in a recent release, users navigating from Logs Related Content on the APM TraceView page to Log Observer were redirected back to APM TraceView instead. This impacted the Logs Correlation experience for customers across all realms, except US3 and Gov. The change has been reverted to restore the expected navigation experience. Product availability for APM and Log Observer was not impacted; the issue was limited to the Logs Correlation workflow from TraceView. Incident Start Time: 05:09 PDT, June 25, 2026 Incident End Time: 22:15 PDT, June 25, 2026

Rum API Endpoints were not available for one of our IPs, resulting in degraded service or a partial outage at first, and possibly a complete outage towards the end. This issue occurred from 4:22pm PST to 4:57pm PST

Datapoint ingest is affected , some datapoints could be dropped. Team is investigating and will provide an update shortly.

Our engineering team is currently implementing an impact mitigation configuration change. The internal investigation is still ongoing. We will share an additional update as soon as possible.

Our engineering team has completed impact mitigating configuration changes and confirmed services are recovered as of 4:55 PDT. We will continue monitoring and share an update as soon as possible.

Our engineering team has confirmed that all the services recovered as of 4:55 PM PDT and remained stable during monitoring. This incident is now resolved.

We are investigating delays in processing Splunk Observability Synthetic Monitoring test results for a small subset of customers. Tests are still running, but metrics may be delayed until the issue is resolved.

We have identified and remediated an issue affecting synthetic test result ingestion for a subset of customers. We apologize for any inconvenience this may have caused and thank you for your patience. Time range of impact: June 1 20:00 UTC - June 2 15:30 UTC

Charts for customers may not be loading. App and UI availability is also impacted. We are investigating and will provide an update shortly.

The issue has been identified and a fix is being implemented.

A fix has been implemented and we are monitoring the status.

This Incident has been resolved.

A degradation of the Splunk O11y Cloud ingest path is causing ~20% of AWS CloudWatch metrics to be dropped. We are investigating the issue and will provide an update as soon as possible.

This incident has been resolved. We have since confirmed that only AWS CloudWatch metrics were impacted during this incident. Azure and GCP metrics were not impacted.

We are currently investigating this issue.

We are continuing to investigate this issue.

The issue has been identified and a fix is being implemented.

This incident has been resolved.

Charts for all customers may not be loading. Datapoint ingest is not affected. We are investigating and will provide an update shortly.

Our teams continue working to identify the cause and implement a fix.

Our investigation remains ongoing, and there is no change in status at this time. We are continuing to prioritize this issue and will provide the next update in thirty minutes.

Our teams are continuing their efforts to identify the root cause of this issue. There is no change in status to report at this time, but we are dedicating all necessary resources to reach a resolution. We will provide another update shortly.

Update (8:50 AM PST): We have observed that charts are now loading successfully. While service is recovering, our teams remain on-site to investigate the root cause and ensure full system stability. We will continue to monitor performance closely.

We are continuing to monitor the environment. There is no change in status at this time; our teams remain focused on resolving the remaining isolated issues.

Our teams are continuing to monitor the environment and are actively working to fully resolve the issue.

Update 10:30 PST: We are also investigating reports of login issues. Users may experience difficulty authenticating or accessing their accounts at this time.

The issue has been identified and a fix is being implemented.

A fix has been implemented and we are monitoring the results.

We are continuing to monitor the environment to ensure full remediation.

This incident has been resolved.

We are investigating an issue affecting Real User Monitoring (RUM) metrics across all realms. Customers may experience custom metric MTS quota limits being exceeded, resulting in metric data being dropped. RUM metric dimensions that were previously available may be missing.

The issue affecting Real User Monitoring (RUM) metrics  has been identified. The fix is currently being implemented. We are actively working to restore full service and will provide further updates as progress continues.

We are continuing to work on the fix. We expect to have further updates within the next hour. We appreciate your patience as we work diligently to resolve this issue and restore full service.

We are currently continuing to test the fix for the Real User Monitoring (RUM) metrics issue. We are thoroughly validating the mitigation to ensure it resolves the issue effectively and maintains system stability. We appreciate your ongoing patience and will provide further updates as testing progresses.

We are currently continuing to test the fix for the Real User Monitoring (RUM) metrics issue. We will share an update once we have further information.

The outage impacting RUM MMS Data Drop has been resolved after internal engineering teams deployed a fix. All systems are operational and stable. Thank you for your patience and understanding.

We are currently investigating an issue where RUM (Real User Monitoring) MMS data is being dropped, which may result in missing or incomplete data in charts and dashboards related to real user monitoring. We are actively investigating and will provide updates as the situation develops.

We have identified the cause of the issue resulting in missing or incomplete data in Real User Monitoring charts and dashboards. We are implementing a fix and will provide an update within 15 minutes.

A solution to address the issue causing missing or incomplete data in Real User Monitoring (RUM) charts and dashboards is currently being implemented. Our teams are actively working on deploying the fix and will continue to provide updates as progress is made

We have identified an issue causing missing or incomplete data in Real User Monitoring (RUM) custom charts and dashboards across multiple realms. A fix is currently being rolled out and validated

A fix for the issue affecting Real User Monitoring (RUM) charts and dashboards has been deployed and is being actively monitored across all realms; customers may now see data restored in charts and dashboards

The issue affecting Real User Monitoring (RUM) custom charts and dashboards across multiple realms has been resolved. The fix has been deployed, and data is now flowing normally.