50 recorded Anodot incidents since September 2020, with official updates, affected components, duration and resolution information.
We detected the problem, working on fixing it ASAP
A fix had been applies, we're monitoring the results
The incident had been resolved
Snowflake Streams have encountered difficulties in retrieving data samples over the last few hours. This issue is not localized to a single region but is impacting all of our clusters across different geographical areas. Our team is actively conducting a thorough investigation to identify the root cause of this disruption and to implement a resolution as quickly as possible
[CRUCIAL UPDATE] The following Data Collectors are currently affected and are not retreiving any samples: - Snowflake - S3 - Kinesis We are continuing to investigate the issue.
All data collectors are currently not working. We are working on restoring them.
We are experiencing issues in collecting data, and detecting and dispatching anomaly type alerts
Starting 7:00 AM UTC. Expected downtime for Frankfurt Customers: ~3 hours
Maintenance in Frankfurt Cluster is completed. Alerts are functional again, Data Collectors are still offline.
Starting 11:00 AM UTC. Expected downtime for US Customers: ~3 hours
Maintenance in US Cluster is completed. Alerts are functional again, Data Collectors are still offline
This incident had been resolved
Anomaly detection and anomaly alerting in EU (Frankfurt) region are current not working. We've identified the issue and are working to resolve it
A fix has been implemented and we are monitoring the results.
This incident had been resolved
We are currently investigating this issue.
We are continuing to investigate this issue.
This incident has been resolved.
Partial data collection issue for data collectors.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We has an issue that caused missing a 4 hours of data. It may have caused many false No-Data alerts. A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We're Investigating the issue
A fix has been implemented and we are monitoring the results.
Incident Resolved
This incident has been resolved.
We're experiencing an elevated level of data API errors and are currently looking into the issue.
We are currently investigating this issue.
This incident has been resolved.
Anomaly detection and anomaly alerting in US region are currently unstable, we've identified the issue and are working to resolve it
We are continuing to work on a fix for this issue.
A fix has been implemented and we are monitoring the results.
We are continuing to monitor for any further issues.
This incident has been resolved.
We're currently investigating an issue where small portion of metrics (the same constant metrics) are experiencing sample loss. As a result false-positive no-data alerts can be triggered and the metrics data shall not be shown in UI.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
During the last 24 hours some alerts may have been missed by the system, we're deploying a hotfix in an hour time
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We're experiencing connectivity issues since 5AM UTC time, this may result in degraded performance and reduced accuracy of alerts and composite calculations. There is also a minor chance of sample data loss. We have identified the problem and applied the necessary measures and are monitoring the system state.
This incident has been resolved.
Streams working with BigQuery does not retrieve new data, seems to be related to incompatible changes in BigQuery - Investigating. This is a cross region issue that impact all of our clusters.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
Around 05:00 UTC we detected errors in samples receiving pipeline, issue is under investigation. Please be advised that at least some of the incoming metrics data was lost and false-positive alerts can be triggered
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We are experiencing issues with the application effecting the ability to visualize metrics and anomalies.
This incident has been resolved.
We've identified the issue and working on a fix. Please note that alerts console is fully functional and no events are lost.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
The issue has been identified and a fix is being implemented.
A fix has been implemented and we are monitoring the results.
The incident was resolved. Alerts dispatching got delayed but no data was lost.
Our alert data processing infrastructure is running behind which is causing delays in sending alerts. No data has been lost and the system should be caught up shortly.
Alerts is now back to normal and we're investigating the root cause
We are continuing to monitor for any further issues.
This incident has been resolved.
We are working on a mitigation with an ETA of 1-2 hours.
Mitigation identified and is being rolled out to production clusters
A fix has been implemented and applied globally, additional monitoring has been applied and we'll be monitoring it over the next few days
We are continuing to monitor for any further issues.
This incident has been resolved.
