45 Dead Mans Snitch incidents · август 2014 г. — official updates, affected components, duration and resolution details.
Мы видим повышенные 503 ответа на API и исследуем причину.
Наша служба автомасштабирования была чрезмерно агрессивной и масштабировалась дальше, чем следовало. Мы обновили нашу конфигурацию и отслеживаем дальнейшие проблемы.
Этот инцидент был урегулирован.
Автоматический перевод официального обновления инцидента.
We are experiencing spurious timeouts receiving check-ins for Snitches. The root cause is related to maintenance a service provider is performing on our backend message bus. You may see occasional timeouts though those check-ins should still be processed normally. We are currently investigating a way to work around the timeouts.
Maintenance on our primary message bus is ongoing with our service provider however timeouts have stabilized. We're continuing to monitor the system and get a better understanding of the problem. We're also discussing possible changes to mitigate similar issues in the future but we expect it will be a larger project to plan and implement.
Maintenance on our primary message bus is done and all systems are back to normal. This kind of maintenance shouldn't be causing timeouts or delayed processing and we will be digging into it to ensure it doesn't occur again.
Email alerts are currently backed up due to upstream issues with our service provider (Amazon SES). Integrations (such as Slack and Webhooks) are unaffected though depend on the state of those services.
We've worked through the backlog of email alerts as Amazon SES has recovered for us. We are still monitoring the situation as AWS continues to work through issues with multiple services.
We are continuing to monitor for any further issues.
This incident has been resolved.
The certificate for https://releases.deadmanssnitch.com expired early this morning (US Eastern) which left downloads unavailable for roughly 6 hours. We're sorry for the issue and any trouble it caused. The root cause was that we missed migrating a DNS record used to verify the certificate when migrating to a new DNS provider earlier this year.
We have received reports of false alerts being sent out for some short interval Snitches. The issue is related to maintenance an upstream provider is performing on our main message bus service that is causing some check-ins to be processed out of order. We've taken an initial step to mitigate the issue but have more work to do to fully address the issue.
We are continuing to monitor the situation as maintenance on our primary message bus continues at our upstream provider. We're currently discussing options for a longer term fix but we expect that fully solving the issue will take longer than it will take for the maintenance to complete.
Our primary message bus is still undergoing maintenance and we have implemented all the mitigations that we can without making larger changes. Taking that into consideration and the fact that we have not seen a disruption which would cause out of order processing within the last hour we've decided to close this incident.
The system we use to send alerts and emails is currently backlogged due an upstream service issue. We're
We believe we've identified the issue which has been causing one job (sending iOS push notifications) to block our queue. We're working to deploy a fix now.
We've deployed a fix for the issue and our worker system has worked through all of the queued jobs. We're monitoring to make sure the system is stable and will be reviewing steps to avoid this in the future.
Alerts and emails are once again being sent in a timely manor. We've identified the root cause as a job which had a missing timeout which was causing issues due to an upstream connection issue. This highlighted that the monitoring we have in place for our worker system needs to be reviewed as we didn't receive an alert that there was an issue.
At 12:49:15UTC the process responsible for periodically checking Snitch histories was shutdown during a deploy but was not restarted as part of the deploy process. Around 20:10UTC we were notified by a customer that a Snitch wasn't failing when it was expected to be and discovered the process was offline as part of our investigation. At 20:22UTC we restarted the process which then kicked off checks for all potentially missing Snitches. Unfortunately we did not receive an alert that the process was offline because, as we should have known, the alert would only have occurred if this process had been running. We're discussing fixes to ensure this kind of error doesn't happen again.
Between 9:15UTC and 11:10UTC a worked process became wedged which caused roughly 12% of alerts during that time to be delayed and the dashboard to be outdated. We've resolved the immediate issue, worked through pending alerts, and we are looking into the cause of the lock up.
Starting at 17:15 UTC until 18:45 UTC we had an issues where alerts for missing and errored Snitches were delayed. All history for Snitches is accurate despite the delay. We've identified the root cause as a bad deploy which caused issues connecting to our backend message bus. We were not initially alerted to the issue and we're investigating the gap in our alerting to make sure this doesn't happen again.
We're seeing an increase in timeouts for our check-in service. We've removed a pair of misbehaving nodes from our load balancer and we're investigating the root cause. Any check-in that received a 408 Request Timeout error was not processed.
We've replaced the affected nodes and error rates have gone back to normal levels
Error rates have returned to normal and systems are all green. We've identified the possible root cause as a bug in the version of the message bus client we were using that could cause a deadlock in some cases. We've updated to the latest version of the client and rolled out the update to production.
Our hosting provider is experiencing an outage with routing to applications. This is causing an outage for our API and management site. Check-in handling and alerting is not affected. https://status.heroku.com/incidents/2402
This incident has been resolved.
We're investigating an issue related to a backend migration that caused some previously paused Snitches to become unpaused and start alerting again. We're working to repause those Snitches.
We've identified that some Snitches that were pending (i.e. have never checked-in to Dead Man's Snitch) were alerting. We've reverted those Snitches which were pending and hadn't been paused to their original state. This should stop future emails for those Snitches.
Pending Snitches should all be fixed and we're seeing alert metrics return to normal. We're waiting until the top of the hour after our basic hourly Snitches alert before we consider the incident closed.
We're really sorry for the trouble and confusion this caused. The issue itself was related to migrating to a new alerting backend that's been in the works for quite a while. We had hoped for it to be seamless (and it almost was) but we failed to double check the state of Snitches that were waiting for their first check-in. We plan to take another pass at our verification process to make sure we haven't missed anything else.
Email check-ins for Snitches are currently returning "451 Temporary Service Failure" to at least one of our email receivers. We believe this may be related to issues in Amazon US-East-1 region. We're currently trying to work around the issue.
We've updated our inbound email servers in DNS to deprioritize inbound-smtp.us-east-1.amazonaws.com. Email receiving should start to recover as servers receive the new MX record and retry pending emails.
Email check-ins are back to normal levels. We're looking into why inbound servers didn't retry with our alternate server and ways to avoid this situation in the future.
We're currently investigating issue affecting check-in processing and dashboard availability. We believe these are related to two major issues affecting our hosting provider (Heroku) and are currently investigating. https://status.heroku.com/incidents/2361 https://status.heroku.com/incidents/2362
We've temporarily disabled alerting as we investigate a way to work around the upstream issues.
We've worked around the issues with check-in processing and are currently working through the backlog of pending check-ins in the queue. It doesn't appear our check-in receiver was impacted by the outage, just the workers that process the check-ins.
Our check-in workers have caught up on all pending check-ins and alerts should be accurate going forward. Our main goal has been to get alerting and check-in processing back online. Heroku continues to experience issues with dynos and routing requests. We've worked around the dyno issues by temporarily moving check-in processing to hosts on EC2. We monitoring check-in process and Heroku's status and will update once we consider the issue fully resolved.
Routing issues with the API and Dashboard appear to be mostly resolved. We are migrating some check-in processing back to Heroku and will continue to monitor the situation.
All systems are green. We've migrated all processing back to Heroku as they have resolved their upstream issue. Our processing system should have recovered more quickly than it did and we're investigating a possible fix for that.
We're investigating an issue with check-in processing starting around 11:45 UTC.
We've restarted the affected service and confirmed that it's processing correctly. It has now caught up on the backlog of pending check-ins. We're continuing to investigate the root cause.
We are continuing to monitor for any further issues.
The root cause has been tracked down to an timeout error during check-in processing that wasn't handled correctly and put the process into a bad state. We're working on a fix for the issue and should have it deployed shortly. Check-in processing stopped at 11:47 UTC but we weren't made aware of the issue until 11:57 UTC. In reviewing our metrics and alerting we've identified a better metric to be alerting on and will be working that into an update to our internal monitoring and alerting systems.
We're investigating an issue with a blocked worker queue that has caused some alerts to be delayed and some by several hours. Our workers have caught up and we're monitoring the situation. We're continuing to investigate the impact and root cause.
This incident has been resolved.
We had a DNS outage starting today at 19:37 UTC and lasting until 19:47 UTC. The outage was caused by our DNS provider having an expired card on file and alerts were going to an unmonitored email address.
At 12:14AM UTC our check-in processor had a slow down processing queued check-ins that caused some 15 minute jobs which check-in at the tail end of the interval to report as missing. We have an automated runbook that forced a restart at 12:20AM and the processor caught back up at 12:21AM UTC when we reported the jobs as being healthy. This caused a small number of false alerts. We know all too well the stress of being paged in the middle of the night and are working to improve the resiliency of our alerting backend in situations like this to avoid these types of false alerts in the future.
We're sorry for any false alerts you received today due to our maintenance. We know how disruptive and aggravating it can be and we work hard to make incidents like this as rare as possible. We performed a required database maintenance today at 16:16 UTC following our normal maintenance process. The maintenance finished at 16:29 UTC at which time we started processing our backlog of check-ins. At 16:30 UTC we believed we had caught up on queued check-ins so we reenabled our alerting process. Unfortunately we had not fully caught up and caused a number of false alerts. Again, we're really sorry for the false alerts. We are in the process of backend improvements that better handle cases where processing is delayed. We look forward to talking about those improvements soon.
We are investigating false alerts caused by check-ins not being processed. No check-ins were lost and we're in the process of catching up on any that were delayed.
All pending check-ins have been processed and the system appears to be stable. We are really sorry for any false alerts and the stress and confusion they may have caused. We are working on ways to make sure issues like this can't happen again.
This incident has been resolved.
