Dead Mans Snitch Status

Maintenance on our primary message bus is done and all systems are back to normal. This kind of maintenance shouldn't be causing timeouts or delayed processing and we will be digging into it to ensure it doesn't occur again.

This incident has been resolved.

The certificate for https://releases.deadmanssnitch.com expired early this morning (US Eastern) which left downloads unavailable for roughly 6 hours. We're sorry for the issue and any trouble it caused. The root cause was that we missed migrating a DNS record used to verify the certificate when migrating to a new DNS provider earlier this year.

Our primary message bus is still undergoing maintenance and we have implemented all the mitigations that we can without making larger changes. Taking that into consideration and the fact that we have not seen a disruption which would cause out of order processing within the last hour we've decided to close this incident.

Alerts and emails are once again being sent in a timely manor. We've identified the root cause as a job which had a missing timeout which was causing issues due to an upstream connection issue. This highlighted that the monitoring we have in place for our worker system needs to be reviewed as we didn't receive an alert that there was an issue.

This incident has been resolved.

At 12:49:15UTC the process responsible for periodically checking Snitch histories was shutdown during a deploy but was not restarted as part of the deploy process. Around 20:10UTC we were notified by a customer that a Snitch wasn't failing when it was expected to be and discovered the process was offline as part of our investigation. At 20:22UTC we restarted the process which then kicked off checks for all potentially missing Snitches. Unfortunately we did not receive an alert that the process was offline because, as we should have known, the alert would only have occurred if this process had been running. We're discussing fixes to ensure this kind of error doesn't happen again.

Between 9:15UTC and 11:10UTC a worked process became wedged which caused roughly 12% of alerts during that time to be delayed and the dashboard to be outdated. We've resolved the immediate issue, worked through pending alerts, and we are looking into the cause of the lock up.

Starting at 17:15 UTC until 18:45 UTC we had an issues where alerts for missing and errored Snitches were delayed. All history for Snitches is accurate despite the delay. We've identified the root cause as a bad deploy which caused issues connecting to our backend message bus. We were not initially alerted to the issue and we're investigating the gap in our alerting to make sure this doesn't happen again.

Error rates have returned to normal and systems are all green. We've identified the possible root cause as a bug in the version of the message bus client we were using that could cause a deadlock in some cases. We've updated to the latest version of the client and rolled out the update to production.

The scheduled maintenance has been completed.

The scheduled maintenance has been completed.

The scheduled maintenance has been completed.