Histórico de interrupções do Honeybadger

1. 15 Eost 2026 da 20:48 UTCresolved

   Entre 19:54 e 20:07 UTC em 15 de agosto, pedidos para o nosso relatório de erro dos EUA API (api.honeybadger.io) experimentou uma taxa de falha elevada. Durante esta janela, a maioria dos pedidos para enviar erros e eventos cronometraram ou receberam respostas de 5xx. Relatório de erro do Browser (JavaScript), uploads de mapas de origem, aplicação web Honeybadger, painéis, monitoramento de tempo de serviço e nossa região da UE não foram afetados. O que aconteceu: A partir de 19:54 UTC, uma única fonte começou a enviar relatórios de erro em aproximadamente 150 vezes sua taxa normal, com cargas diversas vezes maiores do que a média. Isso mais do que dobrou o volume total de dados que chegam ao nosso nível de ingestão, que saturaram em um minuto. Porque o serviço saturado continuou aceitando conexões em vez de retornar erros de forma direta, nosso auto-escalamento e alerta — que observam as respostas de erro — não registrou o problema rapidamente, e a capacidade foi adicionada mais tarde do que deveria ter sido. Serviço recuperado às 20:07 UTC uma vez que a capacidade adicional veio online. A maioria dos relatórios afetados foram tentados com sucesso por nossas bibliotecas de clientes uma vez que a API recuperada, então a grande maioria dos dados enviados durante a janela foi finalmente recebida. Relatórios enviados por bibliotecas sem lógica de repetição não foram recebidos e não podem ser recuperados. O que fizemos: nosso nível de ingestão agora escala a latência da resposta além das taxas de erro, então esse tipo de saturação é detectada em um minuto ou dois ao invés de apenas uma vez que os erros aparecem. Nós também elevamos a capacidade de base do nível e ampliamos nossos alertas de paging para cobrir falhas que não aparecem como erros de servidor. Além disso, estamos adicionando limites em quantos dados uma única fonte pode enviar. Lamentamos pelo rompimento, e por quaisquer lacunas que isso causou em seus dados de erro durante a janela.

Traduzido automaticamente da atualização oficial do incidente.

1. 8 Gouere 2026 da 11:48 UTCresolved

   Entre 06:22 e 08:23 UTC em 8 de julho, pedidos para o nosso relatório de erro dos EUA API (api.honeybadger.io) experimentou uma taxa de falha altamente elevada. Durante esta janela, a maioria dos pedidos de envio de erros, eventos e check-ins recebeu respostas de 5xx ou cronometraram. Os relatórios de erros enviados por bibliotecas clientes sem lógica de reexperimentação durante este período não foram recebidos e não podem ser recuperados. A aplicação Web Honeybadger, painéis, monitorização em tempo útil e a nossa região da UE não foram afectadas. O que aconteceu: A partir das 06:18 UTC, começamos a receber relatórios de erro em aproximadamente triplicar nosso volume total normal da API. O nível de serviço que ingere esse tráfego estava no seu nível de capacidade overnight e ficou saturado. Nossa auto-escalagem respondeu, mas as métricas em que se baseou significativamente sub-medearam o tráfego falhado, então a capacidade foi adicionada muito lentamente — a recuperação total chegou às 08:23 UTC quando a escala pegou e absorveu a carga. Compondo o problema, os alertas PagerDuty não foram disparados para acordar um humano devido a uma má configuração. O que fizemos: Consertamos a fraqueza de escala subjacente: a capacidade da API agora também escala diretamente nas taxas de erro balanceador de carga (anteriormente, escalamos apenas nas taxas de solicitação, e essa escala não responde rapidamente o suficiente), e dobramos a capacidade máxima do nível de ingestão. Estas alterações já foram introduzidas nas regiões dos EUA e da UE. Também estamos avaliando a adição de taxa adaptativa por chave limitando ao serviço de ingestão. Lamentamos pelo rompimento, e por quaisquer lacunas que isso causou em seus dados de erro durante a janela.

Traduzido automaticamente da atualização oficial do incidente.

1. 12 Meurzh 2026 da 19:27 UTCresolved

   We had a backlog in our notifications queue that prevented timely delivery of alerts to channels such as Email, Slack, etc. The backlog has been processed as of 19:20 UTC. Our autoscaling rules for this service had a monitoring gap that prevented a scale-out to maintain a low latency on the queue. This monitoring gap has been resolved.

1. 14 Genver 2026 da 20:40 UTCidentified

   A recent deploy has resulted in our rejecting some check-in API requests. We are in the process of deploying a fix.

2. 14 Genver 2026 da 20:48 UTCresolved

   This incident has been resolved.

1. 22 Here 2025 da 19:46 UTCidentified

   The issue has been identified and a fix is being implemented.

2. 22 Here 2025 da 20:15 UTCresolved

   This incident has been resolved.

1. 22 Here 2025 da 13:09 UTCidentified

   API ingestion of errors has been interrupted. We are deploying a fix.

2. 22 Here 2025 da 13:11 UTCmonitoring

   The US API has been restored, and the EU API restoration is in process.

3. 22 Here 2025 da 13:16 UTCresolved

   The EU API has been restored.

1. 20 Here 2025 da 13:44 UTCidentified

   Our error processing pipeline has been impacted by the AWS outage earlier this morning. Errors are being processed, but the end-to-end latency is much higher than normal. We are working to restore the processing to full capacity.

2. 20 Here 2025 da 17:55 UTCidentified

   We're continuing to monitor the AWS outage. Errors are being processed, but may be delayed. Uptime checks are also delayed, so you may notice lag with uptime notifications and status page updates.

3. 20 Here 2025 da 19:58 UTCmonitoring

   Processing has been restored and all systems are normal. We'll continue to monitor as AWS continues to restore service. Thanks!

4. 20 Here 2025 da 19:58 UTCresolved

   This incident has been resolved.

1. 20 Here 2025 da 13:36 UTCresolved

   Our systems were affected by the DynamoDB outage in AWS' us-east-1 region. Impacts included not being able to log use the web app, uptime checks not being executed on time, and check-ins being falsely reported as missing. We are evaluating what steps we can take to mitigate these kinds of impacts from this kind of outage in the future. Additionally, our status page was not updated in a timely manner due to our not having configured various AWS services as dependent components of our status page. We have fixed that issue, so future AWS outages will result in our status page automatically being updated.

1. 12 Cʼhwevrer 2025 da 13:45 UTCresolved

   We had an interruption in processing of check-in alerts and in-app stats from about 12:30 to about 13:30 UTC. This may have resulted in a failure or a delay in reporting check-ins that went missing during that period.

1. 22 Here 2024 da 13:35 UTCmonitoring

   We have identified and corrected an issue in our data processing pipeline that may have caused delays for some customers. No data has been lost and the system should be caught up shortly.

2. 22 Here 2024 da 15:30 UTCresolved

   The backlog has been processed and all systems are running normally.

1. 23 Gwengolo 2024 da 10:37 UTCinvestigating

   We are looking into the cause of slowness and timeouts with the web app.

2. 23 Gwengolo 2024 da 11:48 UTCmonitoring

   We have identified the cause of the slowdown and performance is back to normal.

3. 23 Gwengolo 2024 da 11:57 UTCresolved

   This incident has been resolved.

1. 24 Gouere 2024 da 23:42 UTCidentified

   We have identified an issue with Insights event ingestion backlog. We are working on a fix.

2. 25 Gouere 2024 da 01:55 UTCmonitoring

   We have successfully pushed a fix and are waiting for the backlog to be processed. I also want to give a reminder that this only affects Insights events; error processing was unaffected.

3. 25 Gouere 2024 da 03:06 UTCresolved

   The backlog has been fully processed, so all Insights events should be ingested and available.

1. 12 Cʼhwevrer 2024 da 20:22 UTCinvestigating

   We are seeing an increased backlog in our check-in processing, which is causing false alarms to be reported. We are investigating the cause of the backlog.

2. 12 Cʼhwevrer 2024 da 20:39 UTCresolved

   We discovered and addressed a configuration issue that caused the delay in processing. Processing has returned to normal.

1. 6 Cʼhwevrer 2024 da 20:59 UTCinvestigating

   We are currently experiencing elevated response times and timeouts from our web application.

2. 6 Cʼhwevrer 2024 da 21:02 UTCinvestigating

   We are continuing to investigate this issue.

3. 6 Cʼhwevrer 2024 da 21:42 UTCidentified

   We've identified an issue with aggregate database queries and are working on a fix.

4. 6 Cʼhwevrer 2024 da 23:03 UTCmonitoring

   We've resolved an issue with query contention, and web application service has been restored. Our API pipeline was unaffected by this incident. Some charts on the reports tab may be outdated until we update project counts. We'll continue to monitor the situation. Sorry for the inconvenience!

5. 6 Cʼhwevrer 2024 da 23:19 UTCresolved

   This incident has been resolved.

1. 17 Genver 2024 da 19:19 UTCidentified

   We are working on restoring service

2. 17 Genver 2024 da 19:21 UTCresolved

   This incident has been resolved.

1. 10 Du 2023 da 19:44 UTCresolved

   We discovered this morning that our Logplex pipeline, which handles Heroku platform errors, had an interruption in processing, which caused platform errors not to be recorded. This has been corrected, the backlog is being processed, and additional monitoring has been added to avoid this issue in the future.

1. 13 Mezheven 2023 da 18:59 UTCidentified

   Issues with AWS Lambda are causing our uptime check reports to be delayed. Reports are queued up and will be delivered as AWS services recover.

2. 13 Mezheven 2023 da 19:34 UTCidentified

   Search indexing and notice timeline charts are also impacted by the failures with Lamba. We're continuing to queue updates for processing once Lambda resumes normal operations.

3. 13 Mezheven 2023 da 21:37 UTCresolved

   Lambda is back in business, and the backlog of events triggered during the outage is being processed.

1. 25 Meurzh 2023 da 01:06 UTCresolved

   Starting around 3:30pm PST, we had an incident where an unhealthy API server was in rotation in our load balancer. During that time requests routed to that server responded with a 502 which could have resulted in the clients dropping notices. At around 5:30pm PST we removed the unhealthy server from rotation. 7:22pm PST edit: Reworded message to describe that notices are not queued on the client, but dropped when the server responds in an error state

1. 24 Meurzh 2023 da 08:05 UTCinvestigating

   We currently are having issues processing our notice backlog. We are looking into the issue.

2. 24 Meurzh 2023 da 08:40 UTCidentified

   We have identified the issue and are working towards clearing the backlog.

3. 24 Meurzh 2023 da 09:01 UTCresolved

   We identified that our workers had an issue deploying. We deployed a fix and upped our capacity to deal with the current backlog. Everything is back to normal now. Turns out this issue was related GitHub rotating their public RSA key: https://github.blog/2023-03-23-we-updated-our-rsa-ssh-host-key/

1. 9 Meurzh 2023 da 09:46 UTCresolved

   We had an incident (from 1:00 to 1:27 PST) where our apps were unresponsive. They have since recovered.