Sentry outage history

We are currently investigating an issue that affects the availability of our product and API

We are continuing to investigate an issue that affects the availability of our product and API

We are experiencing slower page loads and API responses due to a database issue in us

We have identified a potential cause of excess traffic and are working on a fix.

We have put a fix into place and are monitoring the situation

The issue has been resolved

We are investigating delays in error alerts in the US region, up to 10 minutes

We are continuing to investigate the issue

We have identified the issue and implemented a fix.

Error alerting is back to normal.

RCA complete, mitigating now

mitigated, gocd functional again

We are currently investigating a delay in evaluating alerts in the US region.

We have identified the cause and are working to determine a mitigation. This is only affecting US right now.

We have implemented a fix and the systems are recovering. We are monitoring to ensure that backlogs are proessed.

The problem has been resolved, all backlogs have cleared and everything is operational again.

We are currently investigating ingestion delays for spans and transactions in DE

We are currently investigating ingestion delays for spans and transactions in DE and US

We are currently investigating ingestion delays for spans, transactions and alerts in DE and US

We've identified the source of latency and are reverting that change.

The problematic change has been reverted and we're processing backlogs.

We've recovered all ingestion latency in DE. In US, errors have processed their backlog and we continue to make progress on spans in US.

We're continuing to make progress on the span ingestion backlog in US.

We continue to make progress on the spans backlog with an estimated completion time of 15 minutes.

All backlogs have been processed.

We are currently investigating error ingestion latency in US.

We've mitigated the source of ingestion latency and are processing the backlog.

Backlogs have been processed

We are currently investigating reports of API timeouts for the Sentry API in our US region.

API performance has returned to expected levels. We are continuing to monitor.

API performance has returned to expected levels.

We are actively investigating this issue.

We've identified the root cause and are actively working on mitigation.

A fix has been made and the backlog is being processed.

We're processing new events and alerts in realtime, and continue to process the tail of the backlog.

The backlog has been processed

We are currently investigating the issue

We've put a fix in place and are monitoring. We anticipate catching up to ingestion within 1 hour

Ingestion latency for spans, crons, and logs is back to normal levels.

The ingestion delay is resolved

We are actively investigating this issue.

We are still actively investigating this issue.

We've identified the issue causing delays in ingestion and have applied a fix. We'll update again when we return to real time processing.

We're processing new events in real time, and are working through a backlog of events now.

We've processed all backlogged events and have fully recovered.

We are experiencing issues with error and profile ingestion in our US region and are investigating the issue.

We have identified the issue and error and profile ingestion is back up in US. Expect delays in ingestion and alerting for errors and profiles.

We have resolved the incident and are monitoring our ingestion backlog.

The backlog has been cleared and ingestion is back to normal.

We are actively investigating this issue

We are still actively investigating this issue

We've found and fixed the root cause and are burning through our spans backlog. Ingestion should be back to normal in around an hour.

We're still burning through our spans backlog

We've burned through our spans backlog. Latency across span consumers are normal.

we are investigating an issue affecting errors ingestion in the DE/EU region

We have identified the issue causing errors, attachments, and feedback events to have high ingest latency in our EU region and are working on a fix.

We have identified the issue causing errors, attachments, and feedback events to have high ingest latency in our EU region and are continuing to work on a fix.

We have implemented a fix for errors and feedback ingestion in our EU region, and latency for those pipelines has returned to normal. We are still working on fixing our attachments ingestion pipeline in EU.

We have implemented a fix for our attachment ingestion issue in our EU region, and are monitoring while the backlog burns. We will update here when latency is back to normal.

The backlog on attachments in our EU region is fully burned, and latency is back to normal. While new error events are being processed with normal latency, we have a backlog of old error events that we are now burning through.

We are still burning the backlog of EU region error events received during the incident. We are around halfway through the backlog.

We have fully burned the backlog of EU error events ingested during the incident. Latency is normal across all event types.

The issue is mitigated and the webhook backlog is being processed. Seer code reviews will run and linked Github issues and PRs will update incrementally.

The webhook backlog is being processed, ETA ~3 hours until fully caught up. Seer code reviews will run and linked Github issues and PRs will update incrementally.

The webhook backlog is processing. Estimated recovery is now 2-2.5 hours.

The webhook backlog is processing.

The webhook backlog is now resolved.

We are currently investigating the issue.

We continue to investigate the issue.

We have identified the problem and are working on recovery.

We are still working on recovery. New errors should be processed in real time, and we are working on recovering historical data.

We are still working on recovery. New errors should be processed in real time, and we are working on recovering historical data. We are working to start allowing user feedback in DE.

We are processing errors in realtime, and error backlogs have been processed. We're continuing to monitor the situation before restoring feedback ingestion.

We are continuing to monitor ingestion, and a fix is being deployed to unblock feedback ingestion.

Feedback ingestion is back to realtime and the backlog has been processed.

We are investigating ingestion delays.

We identified the issue and restoring service.

Service has been restored and we are actively monitoring the situation

Service has been restored.

We are currently investigating the issue.

We have identified the problem and are working to restore ingestion

We have implemented a fix and are monitoring the situation. We expect ingestion to catch up in the next hour, approximately.

Ingestion has caught up and alerts have been restored.

We are currently investigating this issue.

We have identified the problem and are working on a fix, ingested errors are delayed and will be processed once the fix is in place.

We have identified the problem and are working on recovery. New errors are being processed in real time and we are in the process of consuming backlogged errors.

We are still working on recovery. New errors should be processed in real time, and we are working on recovering historical data.

We expect recovery of historical data to be complete within the next 2 hours.

We have processed the majority of our backlog. We are now recovering an additional 25M error events ranging from 22:20 to 23:20 UTC, which should complete in the next 2 hours

We have processed the majority of our backlog. We are now recovering the tail of the backlog.

We have processed the backlog.

We are currently investigating reports of inconsistent reported number of processed logs in Sentry for the last day

Between July 6th 20:25 UTC and July 7th 12:25 UTC, a wrong configuration change was deployed causing over-sampling of logs, spans and metrics. The configuration change is now reverted and the issue is fixed.