Störungsverlauf von Airbrake

Our dashboard graphs are currently not loading as expected. We are investigating the root cause and will share an update as soon as we have more information. All metrics may be affected including error counts in digest emails.

We have identified the issue affecting service availability and are working through recovery steps. We are continuing to work on restoring normal operation and will share another update as soon as service is stable.

The incident has been resolved and metrics are back online. There will be a dip in metric count graphs while the incident was experienced. Error event data was recorded normally and was unaffected. We apologize for the inconvenience. If you have any questions please reach us at [email protected].

Airbrake is currently experiencing an issue with email notification delivery. Errors are being processed as normal. We will update this incident as we work to resolve this issue.

We've resolved an issue with our email notifications. The issue was caused by email asset delivery which we've resolved. We apologize for any inconvenience. Please email support at [email protected] if you have any questions or concerns.

A subset of customers experienced an issue where the error dashboard list was not loading properly. This may have resulted in error filtering being faulty or the errors list being blank. The issue was escalated to our techops team which resolved the issue. We apologize for the inconvenience that this has caused. If you have any questions or concerns, please reach us at [email protected].

We are currently investigating an issue causing individual project dashboard and settings pages to be unavailable. We apologize for the inconvenience. Please email us at [email protected] if you have any questions.

The error dashboard and settings pages are available again. We're continuing to monitor and investigate the issue.

The issue causing intermittent dashboard loading issues has been resolved. If you have any questions, please reach us at [email protected]. We apologize for the inconvenience.

We are currently investigating an issue causing the login page and dashboard to fail loading due to an error. Please reach out to [email protected] if you have any questions. We apologize for the inconvenience.

The web application is back up and account API is no longer experiencing errors. We're continuing to monitor the resolution. Please reach us at [email protected] if you have any questions.

The web app is experiencing errors which is preventing users from accessing the error dashboard. This is an issue that resurfaced after being resolved earlier today. We will update this post after further investigation.

The web application is back up and account API is no longer experiencing errors. We're continuing to monitor the resolution again. Please reach us at [email protected] if you have any questions.

This incident has been resolved.

We're investigating an issue where the metric graphs for errors are not displaying properly. Please reach out to us at [email protected] if you have any questions. We apologize for the inconvenience. Error monitoring other than metrics is unaffected.

A resolution has been implemented for the metrics API issues that resulted in error graphs to not be visible on the dashboard. We're continuing to monitor the service. Please let us know if you have any questions at [email protected].

We have identified an issue affecting a certain subset of customers and are currently working on remediation. We'll keep this issue updated. Please reach out to [email protected] if you have further questions.

We have resolved the backend issue where a subset of Airbrake projects' experienced blank error dashboards. The backlog in error events have been processed. Please reach out to [email protected] if you have further questions.

Update: A subset of customers experienced service disruptions due to an unresponsive backend API instance. This was triggered by a log storage issue that correlated with a surge in error ingestion traffic. We are actively investigating automated solutions to prevent similar occurrences in the future. We apologize for any inconvenience this caused, please reach out with any questions to [email protected].

Our alerts notifications have been disabled. Digests and other notification types (like Slack) are unaffected. We are working with our email service provider to reenable our alerts domain and will follow up as there is more information. Reach us at [email protected] if you have any questions.

Our email provider has reenabled our alerts domain. All email notifications should be operational again. We still have our sign-up page disabled to implement an additional verification measure and will close this incident when the new sign-up page is live. If you have any questions, please reach us at [email protected].

Updating the status to "monitoring" as alert emails are fixed. We will update the status to "resolved" when the new sign-up page is live.

This issue has been resolved. Our emails have been flowing for the past day and our sign-up page is live again. We apologize for any inconvenience. If you have any questions, please reach us at [email protected].

We have identified an issue with delivering email alerts. We are investigating the cause of the disruption and will report more as we gather more info. Errors are still being recorded and are being delivered to your error dashboards. Integration alerts are also unaffected.

This issue has been resolved. Emails are being delivered again. Reach out to [email protected] if you have any questions.

We have identified an issue affecting a certain subset of customers and are currently working on remediation. You may notice a delay in Error processing. We'll keep this issue updated. Please reach out to [email protected] if you have further questions.

We apologize for any inconvenience this caused. The incident has been resolved and normal operations have resumed.

We are currently investigating a delay in error processing. Errors on your error dashboard and error email notifications might take longer to appear than usual. We will share updates as we make progress in addressing this issue. If you have any questions or concerns, please reach us at [email protected]. We apologize for any inconvenience.

We have identified the issue and are currently working to process the backlog of notifications. We will continue to monitor the situation and provide updates.

The backlog in error notifications has been processed. Users should get error emails and see errors show up on their dashboards without delays again. We apologize for any inconvenience. If you have any questions, please reach us at [email protected].

We are currently investigating increased errors.

This incident has been resolved. Services should be back to normal.

We've identified an issue with our queue. Notification delivery is currently delayed. Thanks for your patience and sorry for any inconvenience this causes. If you have any issues or questions please email [email protected]

We are continuing to work on a fix for this issue.

Queues are back to normal and notifications are once against being delivered.

This incident has been resolved.

Some users may not be able to access the Airbrake web app. We have all hands on deck troubleshooting this dashboard outage and look forward to getting it back online ASAP. Thanks for your patience and sorry for any inconvenience this causes. If you have any issues or questions please email [email protected]

This incident has been resolved, the Airbrake web app is now working as expected. If you have any issues or questions please email [email protected]

Some users may not be able to load error, performance, or deploy data in their dashboard. The API is accepting errors as usual and you should be able to continue troubleshooting errors as soon as the dashboard has been restored. Thanks for your patience and sorry for any inconvenience this causes. If you have any issues or questions please email [email protected].

The cause of the incident has been identified and resolved. Systems have returned to their operational states.

We are currently investigating degraded performance on a database shard. Some customers may experience delays processing error and APM data.

We are continuing to investigate this issue.

The issue has been identified and a fix is being implemented.

A fix has been implemented and we are monitoring while the backlog of notice data / APM events are being processed.

The backlog of notice and apm data has been processed and shard performance has stabilized. Please reach out to support ([email protected]) with further questions.

Some users may not be able to access the error dashboard. The API is accepting errors as usual and you should be able to continue troubleshooting your errors as soon as the dashboard has been restored. We have all hands on deck troubleshooting this dashboard outage and look forward to getting it back online ASAP. Thanks for your patience and sorry for any inconvenience this causes. If you have any issues or questions please email [email protected]

This incident has been resolved.

We are currently investigating API errors which are affecting some projects.

We are continuing to investigate an issue.

We have identified an issue where for some requests, our API is failing due to data caching errors. Affected users might experience their error dashboards failing to load and a delay in email notifications. If you have any questions or concerns, reach out to [email protected].

We are continuing to investigate the issue causing a delay in processing performance data. Customer's project dashboards and email notifications should be returning to normal. If you experience any issues or have any questions, reach us at [email protected]

We are continuing to work on a fix for this issue.

We've applied a change that relieved the load on our production database that was causing a delay in processing performance data. Delays in application performance should be resolving. If you continue to experience any issues, please let us know at [email protected].

We are currently investigating API errors which are affecting some projects.

We are continuing to investigate this issue.

We've received reports of issues with tracking deploys, we are investigating this and believe it to be a related symptom. If you are experiencing this issue please let us know [email protected], we apologize for any inconvenience this causes.

All systems are back to normal, we will continue to monitor the situation. If you experience any issues sending errors or tracking deploys for your project, please let us know [email protected]

The overloaded ingress server which caused issues with deploy and error tracking was scaled up and the issues have been resolved. If you have any questions, please reach out to us at [email protected].

We are currently investigating API errors and processing delays which are affecting some projects. Some errors will be delayed but not lost. Sorry for the inconvenience.

We have identified the issue causing API errors and processing delays which is affecting some projects. We are currently working to mitigate the outage in processing and are restarting core processing components. We apologize for the inconvenience. If you have any questions or concerns, please reach us at [email protected].

A fix has been implemented and we are monitoring the results. Our APIs are back to normal and there are no more processing delays. Please contact the support at [email protected] if you have any questions.

The issue causing the API errors and delay in error notice processing has been resolved. The backlog has been processed and systems are operating normally. We apologize for the inconvenience. Please reach us at [email protected] if you have any questions.