Historia awarii Scout

We are experiencing delays in our ingest pipeline

This incident has been resolved.

A subset of customers are experiencing ingest delays. Investigating.

This incident has been resolved.

Certain customers are unable to view the UI and metrics are delayed.

Metrics are recovering.

This incident has been resolved.

We are seeing problematic database behavior and working to mitigate it.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

We are working to address high load on database instances. We are still receiving data as normal.

A portion of customers are seeing UI unavailability and slow metric ingest times. We are still working towards a sustainable solution.

We have made some changes to mitigate the situation. Metrics for affected customers are recovering.

This incident has been resolved.

We are having some database issues affecting a portion of accounts. Applying a fix. Metrics are still being ingested.

Ingested metrics are catching up.

This incident has been resolved.

We are performing emergency maintenance on our database. Some customers are unfortunately being impacted.

A fix has been implemented and we are monitoring the results.

Some metrics are still delayed, but backfill is proceeding.

We are currently investigating this issue.

The issue has been identified and a fix is being implemented.

A fix has been implemented and we are monitoring the results.

Most customers are recovered but a portion are still impacted.

Remaining customers are now recovering.

This incident has been resolved.

We are seeing delays in processing metric information. We are investigating.

We are suffering some cascading effects that are impacting web portal availability as well.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

A subset of customers experienced a delay in metrics. All metrics have caught up at this point.

We are seeing a spike in metrics, causing delays

This incident has been resolved.

We are currently investigating this issue.

A fix has been implemented and we are monitoring the results.

The initial fix was unsuccessful, certain accounts are now substantially delayed.

An alternate approach has been applied, we are watching.

It has been a long day with kafka. We continue to experience instability, causing lag and dropped payloads.

Throughput has improved although behavior of individual partitions remains a problem and is still causing delays in some cases.

We are not to full resolution yet.

Zookeeper corruption has been rooted out. Things appear healthier and catching up in all cases.

This incident has been resolved.

Metric ingestion is being delayed for a set of users. We are working towards resolution.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

Ingested records are taking longer than usual to process. In some cases, this is affecting alerting.

We are continuing to investigate this issue.

We have identified the issue and are working on fixes.

Ingest is recovering. Some accounts will require additional backfill of data, which we are working on.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

Ingested data is suffering delays in processing. Our team is working on a fix.

The issue has been identified and a fix is being implemented.

Delays are recovering, we are monitoring.

This incident has been resolved.

We are currently investigating this issue.

We are continuing to investigate this issue.

A fix has been implemented and we are monitoring the results.

We are continuing to monitor for any further issues.

This incident has been resolved.

The issue has been identified and a fix is being implemented.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

The issue has been identified and a fix is being implemented.

This incident has been resolved.

We are currently investigating this issue.

The issue has been identified and a fix is being implemented.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.