Dead Mans Snitch 장애 기록

1. 2026년 10월 27일 오후 4:13 UTCinvestigating

   API에 503 응답을 올리고 원인을 조사합니다.

2. 2026년 10월 27일 오후 4:19 UTCmonitoring

   우리의 자동 확장 서비스는 과도하게 공격하고 그것 보다는 더 멀리 확장되었습니다. 우리는 우리의 윤곽을 새롭게 하고 더 문제점을 감시하고 있습니다.

3. 2026년 10월 27일 오후 5:00 UTCresolved

   이 사건은 해결되었습니다.

공식 인시던트 업데이트를 자동 번역했습니다.

1. 2025년 11월 22일 오후 5:12 UTCinvestigating

   We are experiencing spurious timeouts receiving check-ins for Snitches. The root cause is related to maintenance a service provider is performing on our backend message bus. You may see occasional timeouts though those check-ins should still be processed normally. We are currently investigating a way to work around the timeouts.

2. 2025년 11월 22일 오후 8:20 UTCmonitoring

   Maintenance on our primary message bus is ongoing with our service provider however timeouts have stabilized. We're continuing to monitor the system and get a better understanding of the problem. We're also discussing possible changes to mitigate similar issues in the future but we expect it will be a larger project to plan and implement.

3. 2025년 11월 23일 오전 12:03 UTCresolved

   Maintenance on our primary message bus is done and all systems are back to normal. This kind of maintenance shouldn't be causing timeouts or delayed processing and we will be digging into it to ensure it doesn't occur again.

1. 2025년 10월 20일 오후 5:19 UTCmonitoring

   Email alerts are currently backed up due to upstream issues with our service provider (Amazon SES). Integrations (such as Slack and Webhooks) are unaffected though depend on the state of those services.

2. 2025년 10월 20일 오후 6:06 UTCmonitoring

   We've worked through the backlog of email alerts as Amazon SES has recovered for us. We are still monitoring the situation as AWS continues to work through issues with multiple services.

3. 2025년 10월 20일 오후 7:18 UTCmonitoring

   We are continuing to monitor for any further issues.

4. 2025년 10월 20일 오후 7:18 UTCresolved

   This incident has been resolved.

1. 2025년 7월 29일 오전 11:12 UTCresolved

   The certificate for https://releases.deadmanssnitch.com expired early this morning (US Eastern) which left downloads unavailable for roughly 6 hours. We're sorry for the issue and any trouble it caused. The root cause was that we missed migrating a DNS record used to verify the certificate when migrating to a new DNS provider earlier this year.

1. 2025년 4월 8일 오전 12:12 UTCidentified

   We have received reports of false alerts being sent out for some short interval Snitches. The issue is related to maintenance an upstream provider is performing on our main message bus service that is causing some check-ins to be processed out of order. We've taken an initial step to mitigate the issue but have more work to do to fully address the issue.

2. 2025년 4월 8일 오전 1:02 UTCmonitoring

   We are continuing to monitor the situation as maintenance on our primary message bus continues at our upstream provider. We're currently discussing options for a longer term fix but we expect that fully solving the issue will take longer than it will take for the maintenance to complete.

3. 2025년 4월 8일 오전 3:26 UTCresolved

   Our primary message bus is still undergoing maintenance and we have implemented all the mitigations that we can without making larger changes. Taking that into consideration and the fact that we have not seen a disruption which would cause out of order processing within the last hour we've decided to close this incident.

1. 2025년 1월 28일 오후 2:39 UTCidentified

   The system we use to send alerts and emails is currently backlogged due an upstream service issue. We're

2. 2025년 1월 28일 오후 2:48 UTCidentified

   We believe we've identified the issue which has been causing one job (sending iOS push notifications) to block our queue. We're working to deploy a fix now.

3. 2025년 1월 28일 오후 2:57 UTCmonitoring

   We've deployed a fix for the issue and our worker system has worked through all of the queued jobs. We're monitoring to make sure the system is stable and will be reviewing steps to avoid this in the future.

4. 2025년 1월 28일 오후 3:39 UTCresolved

   Alerts and emails are once again being sent in a timely manor. We've identified the root cause as a job which had a missing timeout which was causing issues due to an upstream connection issue. This highlighted that the monitoring we have in place for our worker system needs to be reviewed as we didn't receive an alert that there was an issue.

1. 2024년 4월 16일 오후 8:44 UTCresolved

   At 12:49:15UTC the process responsible for periodically checking Snitch histories was shutdown during a deploy but was not restarted as part of the deploy process. Around 20:10UTC we were notified by a customer that a Snitch wasn't failing when it was expected to be and discovered the process was offline as part of our investigation. At 20:22UTC we restarted the process which then kicked off checks for all potentially missing Snitches. Unfortunately we did not receive an alert that the process was offline because, as we should have known, the alert would only have occurred if this process had been running. We're discussing fixes to ensure this kind of error doesn't happen again.

1. 2024년 4월 15일 오전 11:36 UTCresolved

   Between 9:15UTC and 11:10UTC a worked process became wedged which caused roughly 12% of alerts during that time to be delayed and the dashboard to be outdated. We've resolved the immediate issue, worked through pending alerts, and we are looking into the cause of the lock up.

1. 2023년 3월 27일 오후 7:21 UTCresolved

   Starting at 17:15 UTC until 18:45 UTC we had an issues where alerts for missing and errored Snitches were delayed. All history for Snitches is accurate despite the delay. We've identified the root cause as a bad deploy which caused issues connecting to our backend message bus. We were not initially alerted to the issue and we're investigating the gap in our alerting to make sure this doesn't happen again.

1. 2022년 6월 22일 오후 1:34 UTCinvestigating

   We're seeing an increase in timeouts for our check-in service. We've removed a pair of misbehaving nodes from our load balancer and we're investigating the root cause. Any check-in that received a 408 Request Timeout error was not processed.

2. 2022년 6월 22일 오후 2:07 UTCmonitoring

   We've replaced the affected nodes and error rates have gone back to normal levels

3. 2022년 6월 22일 오후 2:25 UTCresolved

   Error rates have returned to normal and systems are all green. We've identified the possible root cause as a bug in the version of the message bus client we were using that could cause a deadlock in some cases. We've updated to the latest version of the client and rolled out the update to production.

1. 2022년 2월 24일 오후 4:59 UTCidentified

   Our hosting provider is experiencing an outage with routing to applications. This is causing an outage for our API and management site. Check-in handling and alerting is not affected. https://status.heroku.com/incidents/2402

2. 2022년 2월 24일 오후 5:22 UTCresolved

   This incident has been resolved.

1. 2021년 12월 15일 오후 11:53 UTCinvestigating

   We're investigating an issue related to a backend migration that caused some previously paused Snitches to become unpaused and start alerting again. We're working to repause those Snitches.

2. 2021년 12월 16일 오전 12:20 UTCidentified

   We've identified that some Snitches that were pending (i.e. have never checked-in to Dead Man's Snitch) were alerting. We've reverted those Snitches which were pending and hadn't been paused to their original state. This should stop future emails for those Snitches.

3. 2021년 12월 16일 오전 12:39 UTCmonitoring

   Pending Snitches should all be fixed and we're seeing alert metrics return to normal. We're waiting until the top of the hour after our basic hourly Snitches alert before we consider the incident closed.

4. 2021년 12월 16일 오전 1:15 UTCresolved

   We're really sorry for the trouble and confusion this caused. The issue itself was related to migrating to a new alerting backend that's been in the works for quite a while. We had hoped for it to be seamless (and it almost was) but we failed to double check the state of Snitches that were waiting for their first check-in. We plan to take another pass at our verification process to make sure we haven't missed anything else.

1. 2021년 12월 7일 오후 4:47 UTCidentified

   Email check-ins for Snitches are currently returning "451 Temporary Service Failure" to at least one of our email receivers. We believe this may be related to issues in Amazon US-East-1 region. We're currently trying to work around the issue.

2. 2021년 12월 7일 오후 5:23 UTCidentified

   We've updated our inbound email servers in DNS to deprioritize inbound-smtp.us-east-1.amazonaws.com. Email receiving should start to recover as servers receive the new MX record and retry pending emails.

3. 2021년 12월 7일 오후 6:13 UTCresolved

   Email check-ins are back to normal levels. We're looking into why inbound servers didn't retry with our alternate server and ways to avoid this situation in the future.

1. 2021년 9월 27일 오전 8:04 UTCinvestigating

   We're currently investigating issue affecting check-in processing and dashboard availability. We believe these are related to two major issues affecting our hosting provider (Heroku) and are currently investigating. https://status.heroku.com/incidents/2361 https://status.heroku.com/incidents/2362

2. 2021년 9월 27일 오전 8:48 UTCidentified

   We've temporarily disabled alerting as we investigate a way to work around the upstream issues.

3. 2021년 9월 27일 오전 9:44 UTCidentified

   We've worked around the issues with check-in processing and are currently working through the backlog of pending check-ins in the queue. It doesn't appear our check-in receiver was impacted by the outage, just the workers that process the check-ins.

4. 2021년 9월 27일 오전 9:57 UTCmonitoring

   Our check-in workers have caught up on all pending check-ins and alerts should be accurate going forward. Our main goal has been to get alerting and check-in processing back online. Heroku continues to experience issues with dynos and routing requests. We've worked around the dyno issues by temporarily moving check-in processing to hosts on EC2. We monitoring check-in process and Heroku's status and will update once we consider the issue fully resolved.

5. 2021년 9월 27일 오후 12:10 UTCmonitoring

   Routing issues with the API and Dashboard appear to be mostly resolved. We are migrating some check-in processing back to Heroku and will continue to monitor the situation.

6. 2021년 9월 27일 오후 12:55 UTCresolved

   All systems are green. We've migrated all processing back to Heroku as they have resolved their upstream issue. Our processing system should have recovered more quickly than it did and we're investigating a possible fix for that.

1. 2021년 1월 4일 오후 12:14 UTCinvestigating

   We're investigating an issue with check-in processing starting around 11:45 UTC.

2. 2021년 1월 4일 오후 12:28 UTCmonitoring

   We've restarted the affected service and confirmed that it's processing correctly. It has now caught up on the backlog of pending check-ins. We're continuing to investigate the root cause.

3. 2021년 1월 4일 오후 12:28 UTCmonitoring

   We are continuing to monitor for any further issues.

4. 2021년 1월 4일 오후 3:54 UTCresolved

   The root cause has been tracked down to an timeout error during check-in processing that wasn't handled correctly and put the process into a bad state. We're working on a fix for the issue and should have it deployed shortly. Check-in processing stopped at 11:47 UTC but we weren't made aware of the issue until 11:57 UTC. In reviewing our metrics and alerting we've identified a better metric to be alerting on and will be working that into an update to our internal monitoring and alerting systems.

1. 2020년 10월 15일 오후 2:32 UTCinvestigating

   We're investigating an issue with a blocked worker queue that has caused some alerts to be delayed and some by several hours. Our workers have caught up and we're monitoring the situation. We're continuing to investigate the impact and root cause.

2. 2020년 10월 15일 오후 3:36 UTCresolved

   This incident has been resolved.

1. 2020년 8월 31일 오후 8:06 UTCresolved

   We had a DNS outage starting today at 19:37 UTC and lasting until 19:47 UTC. The outage was caused by our DNS provider having an expired card on file and alerts were going to an unmonitored email address.

1. 2020년 3월 2일 오후 2:40 UTCresolved

   At 12:14AM UTC our check-in processor had a slow down processing queued check-ins that caused some 15 minute jobs which check-in at the tail end of the interval to report as missing. We have an automated runbook that forced a restart at 12:20AM and the processor caught back up at 12:21AM UTC when we reported the jobs as being healthy. This caused a small number of false alerts. We know all too well the stress of being paged in the middle of the night and are working to improve the resiliency of our alerting backend in situations like this to avoid these types of false alerts in the future.

1. 2019년 9월 26일 오후 4:59 UTCresolved

   We're sorry for any false alerts you received today due to our maintenance. We know how disruptive and aggravating it can be and we work hard to make incidents like this as rare as possible. We performed a required database maintenance today at 16:16 UTC following our normal maintenance process. The maintenance finished at 16:29 UTC at which time we started processing our backlog of check-ins. At 16:30 UTC we believed we had caught up on queued check-ins so we reenabled our alerting process. Unfortunately we had not fully caught up and caused a number of false alerts. Again, we're really sorry for the false alerts. We are in the process of backend improvements that better handle cases where processing is delayed. We look forward to talking about those improvements soon.

1. 2018년 11월 20일 오후 2:07 UTCinvestigating

   We are investigating false alerts caused by check-ins not being processed. No check-ins were lost and we're in the process of catching up on any that were delayed.

2. 2018년 11월 20일 오후 2:54 UTCinvestigating

   All pending check-ins have been processed and the system appears to be stable. We are really sorry for any false alerts and the stress and confusion they may have caused. We are working on ways to make sure issues like this can't happen again.

3. 2018년 11월 20일 오후 2:56 UTCresolved

   This incident has been resolved.