Signalfx outage history

From 3:08PM PDT, customers may experience delays in seeing newly created MetricTimeSeries. Datapoint ingest is not affected. We are investigating and will provide an update shortly.

We've identified that this incident was a false alarm due to rate limiting applied to a single tenant. We have reached out to the affected tenant. All other customers within US0 are not and have not been impacted by this incident.

This incident has been resolved.

A degradation in the performance of the Splunk APM metrics processing pipeline is causing Monitoring MetricSets to be dropped. Trace data ingest is not impacted, but service, endpoint and workflow dashboards, and other charts and detectors built from Monitoring MetricSets are impacted.

The issue has been identified and a fix is being implemented.

A fix has been implemented and we are monitoring the results.

The issue has been fixed.

Some customers may have experienced delays in alert notifications from 4 am ~ 6 am (PST), July 24th, 2026. Datapoint ingest is not affected. The issue is recovered.

We are currently investigating the issue. Certain APIs may be failing and active alert pages is loading slowly. No data is being dropped or lost.

The issue has been identified and the fix is being implemented.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

We are currently investigating the issue. Certain APIs may be failing and active alert pages is loading slowly. No data is being dropped or lost.

Customer impact began around 7:00 AM PDT. We have identified a fix, but a full recovery could take several hours

We are continuing to work on a fix for this issue. A full recovery could take several hours

A fix has been implemented and we are monitoring the results as backlog recovers. Alert states and event time series in the UI may be delayed while the backlog recovers.

This incident has been resolved.

Starting at 1a PDT, a small percentage of detectors may have missed firing. No data has been lost. We are investigating and will provide an update shortly.

A fix has been implemented and we are monitoring the results.

We are continuing to monitor for any further issues.

We have seen a return of a small percentage of detectors that may not be firing. No data is lost. We are continuing to investigate.

We continue to investigate an issue affecting a small percentage of detectors in US0 that may not have fired as expected. No data loss has been observed. Our engineering teams are actively investigating the issue, and we will provide another update as more information becomes available.

A mitigation has been applied for the issue affecting a small percentage of detectors in US0 that may not have fired as expected. We are observing continued recovery and are monitoring the environment to ensure the mitigation is fully effective. No data loss has been observed. We will provide another update once recovery is complete.

We are observing continued recovery and are monitoring the environment to ensure the mitigation is fully effective. No data loss has been observed. We will provide another update once recovery is complete.

We are observing continued recovery and are monitoring the environment to ensure the mitigation is fully effective. No data loss has been observed. We will provide another update once recovery is complete.

We have applied mitigations that reduced the impact, but recovery is not yet complete. A small percentage of detectors in US0 may still not be firing as expected due to delayed processing. No data loss has been observed. Engineering teams continue to investigate towards the additional mitigations. We will provide another update as more information becomes available.

We have identified that newly created time series in US0 may take up to 10 minutes to be included in detectors, which may delay detector evaluations for those new time series. Notification delay mean time is currently healthy, and no data loss has been observed. Engineering teams are continuing mitigation work and validating whether it clears the remaining delays. We will provide another update once validation is complete.

We continue to investigate delayed inclusion of newly created time series in detectors in US0. New time series may take up to 10 minutes to be included in detectors, which may delay detector evaluations for those time series. Notification delay mean time remains healthy, and no data loss has been observed. Engineering teams are performing a controlled rollout of affected backend components as part of ongoing mitigation work, and this is not expected to introduce additional customer impact. We will provide another update as validation progresses.

This incident has been resolved.

Between 4:45p - 6:45p PDT a very small percentage of metric data points were being dropped. This issue has been resolved.

Starting at 11:30a PDT, datapoint ingest is affected and we are dropping a small percentage of datapoints. We are investigating and will provide an update as progress is made.

We continue to investigate the issue affecting a small percentage of datapoint ingest in US0. Our engineering teams are actively working on the issue and evaluating potential mitigations. We will provide another update as more information becomes available.

We continue to investigate the issue affecting a small percentage of datapoint ingest in US0. Our engineering teams are actively working on the issue. We will provide another update as more information becomes available.

We have observed improvement in datapoint ingest in US0, and the rate of dropped datapoints has returned to expected levels. We are continuing to monitor the service closely to ensure stability. We will provide another update once monitoring is complete or if any additional issues are observed.

This incident has been resolved.

Starting at 6:48a PDT, Synthetic test runs are not able to be viewed in the user interface. Tests are still running and detectors are still working. We are investigating the issue and will send updates as progress is made.

The APM Profiling user interface is also unavailable at this time due to the same underlying issue. We are continuing to investigate and will provide updates as progress is made.

The data visible in the APM Profiling user interface is experiencing delays. Splunk Synthetics Monitoring is operational. We are continuing to investigate and will provide updates as progress is made.

This incident has been resolved.

Between 1a PDT and 2:50a PDT, any GCP integration that had the "Sync all (dynamically) listed projects" setting enabled had some of the Cloud Monitoring data from GCP dropped.  This issue has been resolved.  Customers may see gaps in charts and detectors that use affected GCP Cloud Monitoring metrics. AWS CloudWatch and Azure Monitor metrics are not impacted.

We are investigating a major service disruption affecting Splunk APM Trace Search and the Splunk RUM Session Details feature. Customers may be unable to search or view traces and may be unable to load session details. Data ingest is not impacted at this time. We will provide an update as soon as possible.

We are continuing to investigate this issue.

The issue has been identified and a fix is being implemented.

This incident has been resolved.

Customers may be experiencing delays in some detectors. Datapoint ingest is not affected. We are investigating and will provide an update shortly.

We have identified the source of the issue and have taken mitigation actions. Recovery is in progress, and we are seeing gradual improvement. Our teams are continuing to investigate the underlying cause while monitoring the recovery. We will provide another update as more information becomes available.

The issue causing delays in some detectors has been resolved. Systems have recovered, and detector processing has returned to expected levels. Thank you for your patience while we worked to resolve this issue.

A degradation in the performance of the Splunk APM metrics processing pipeline is causing Monitoring MetricSets to be delayed by more than five minutes. Trace data ingest is not impacted, but service, endpoint and workflow dashboards, and other charts and detectors built from Monitoring MetricSets are impacted.

We are continuing to investigate this issue.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

We have confirmed that the issue affecting Tag Spotlight has been resolved. Customers may have experienced limited functionality within Tag Spotlight beginning at approximately 4:30 PM PT on June 30. Full functionality was restored at 7:20 AM PT on July 1.

Due to a regression in a recent release, users navigating from Logs Related Content on the APM TraceView page to Log Observer were redirected back to APM TraceView instead. This impacted the Logs Correlation experience for customers across all realms, except US3 and Gov. The change has been reverted to restore the expected navigation experience. Product availability for APM and Log Observer was not impacted; the issue was limited to the Logs Correlation workflow from TraceView. Incident Start Time: 05:09 PDT, June 25, 2026 Incident End Time: 22:15 PDT, June 25, 2026

We are investigating why Trace Lookups are failing for traces created between approximately 5:30am PT - 7:30am PT. Monitoring MetricSets and Troubleshooting MetricSets are not impacted. Metrics are still available. We are investigating the issue.

We are still investigating the issue.

We are continuing to investigate this issue.

The issue affecting Trace Lookup searchability for traces generated between approximately 5:30 AM and 7:30 AM PT on June 17 has been identified. This occurred during an approved maintenance window associated with repartitioning and scaling activities. Customers may be unable to search for traces generated during this timeframe. Trace ingestion and processing are operating normally, and traces generated outside of the affected window remain searchable. Our team has developed a fix and is currently validating it through internal testing. We expect validation to complete within approximately two hours and will provide another update once the fix is ready for deployment or as additional information becomes available.

We continue to make progress toward resolution. The fix is currently moving through the implementation process and will be rolled out once all deployment steps have been completed. We will provide another update as progress continues.

The fix has been successfully deployed, and we have confirmed that the issue affecting Trace Lookup searchability for traces generated between approximately 5:30 AM and 7:30 AM PT on June 17 has been resolved. Customers should now be able to search affected traces as expected.

The Scheduled Maintenance for US0 Splunk APM Span Firehose Optimization has been extended while we continue to monitor the improvements.

We have completed our Scheduled Maintenance window. Thanks for your patience.

A degradation in the performance of the Splunk APM trace processing pipeline is causing Troubleshooting MetricSets and Monitoring MetricSets to be delayed by more than five minutes. As a result, the APM Troubleshooting experience, service maps and Tag Spotlight do not have access to the most recent data. The processing of metrics for Business Workflows, which also depends on this pipeline, are equally delayed. Trace data ingest is not impacted at this time.

Efforts to address the pipeline performance degradation are ongoing.

We are continuing to work on the performance degradation affecting the Splunk APM trace processing pipeline. Our teams are actively working to restore normal service, and we will provide further updates as more information becomes available.

There is no change in status. Our teams continue to work on resolving the performance degradation in the Splunk APM trace processing pipeline. Delays of over five minutes in Troubleshooting and Monitoring MetricSets persist. We will provide further updates as soon as more information is available.

We have identified the root cause of the performance degradation in the Splunk APM trace processing pipeline. Our engineering teams are currently planning the most efficient approach to implement a fix and restore normal service functionality. We will provide further updates as we move into the implementation phase.

Our teams are continuing to finalize the implementation plan for the fix to address the performance degradation in the Splunk APM trace processing pipeline. We will provide further updates as we move into the implementation phase.

There is no change in status to report. Our engineering teams are currently finalizing the remediation strategy to address the performance degradation within the Splunk APM trace processing pipeline. We are prioritizing a stable resolution.

Our engineering teams are working diligently to ensure the proposed fix for the Splunk APM trace processing pipeline is executed effectively to restore normal service.

Efforts to improve the pipeline performance, including monitoring metrics pipeline management capabilities such as aggregation, filtering, and archiving, are ongoing.

We are continuing to work on a fix for this issue.

This incident has been resolved as latency has dropped to a normal threshold.

A degradation in the performance of the Splunk APM trace processing pipeline is causing Troubleshooting MetricSets and Monitoring MetricSets to be delayed by more than five minutes. As a result, the APM Troubleshooting experience, service maps and Tag Spotlight do not have access to the most recent data and may fail or be slow to load. The processing of metrics for Business Workflows, which also depends on this pipeline, are equally delayed. Trace data ingest is not impacted at this time; service-level and endpoint-level Monitoring MetricSets and the detectors built from them are also not impacted.

We are continuing to investigate this issue.

A fix has been implemented and we are monitoring the results.

We are continuing to monitor for any further issues.

This incident has been resolved.

Customers may be experiencing delays in some charts and detectors. Datapoint ingest is not affected. We are investigating and will provide an update shortly.

Lagging continues and has caused some charts and dashboards to fail to load.

We have scaled up to reduce lag but are are still investigating the root cause.

A degradation in the performance of the Splunk APM metrics processing pipeline is causing Monitoring MetricSets and Troubleshooting MetricSets to be delayed by more than five minutes. Trace data ingest is not impacted, but service, endpoint and workflow dashboards, and other charts and detectors built from Monitoring MetricSets are impacted.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.