Honeybadger driftsforstyrrelser

1. 15. august 2026 kl. 20.48 UTCresolved

   Mellem 19: 54 og 20: 07 UTC den 15. august, anmodninger til vores amerikanske fejlrapportering API (api.honeybadger.io) oplevede en forhøjet fejlprocent. I dette vindue, et flertal af anmodninger om at indsende fejl og begivenheder timet ud eller modtaget 5xx svar. Browser (JavaScript) fejlrapportering, kildekort uploads, Honeybadgers webapplikation, dashboards, uptime monitoring og vores EU-region blev ikke påvirket. Hvad skete der: Begynder på 19: 54 UTC, en enkelt kilde begyndte at indsende fejlrapporter på omkring 150 gange sin normale sats, med nyttelast flere gange større end gennemsnittet. Dette mere end fordoblet den samlede mængde data ankommer til vores indtagelse niveau, som mættet inden for et minut. Da den mættede tjeneste blev ved med at acceptere forbindelser i stedet for at returnere fejl direkte, registrerede vi ikke problemet hurtigt, og kapaciteten blev tilføjet senere, end det burde have været. Service inddrives på 20: 07 UTC, når yderligere kapacitet kom online. De fleste berørte rapporter blev genprøvet med succes af vores klient biblioteker, når API genvundet, så det store flertal af data indsendt i løbet af vinduet blev i sidste ende modtaget. Der blev ikke modtaget rapporter fra biblioteker uden genterisk logik, som ikke kan inddrives. Hvad vi har gjort: vores indtagelse niveau nu skalerer på respons latency ud over fejlrater, så denne form for mætning er opdaget inden for et minut eller to snarere end kun én gang fejl vises. Vi har også hævet niveauets basiskapacitet og udvidet vores personsøgningsindberetninger til at dække fejl, der ikke overflade som server fejl. Vi tilføjer yderligere grænser for, hvor meget data en enkelt kilde kan indsende. Vi beklager forstyrrelsen og eventuelle huller i dine fejldata under vinduet.

Automatisk oversat fra den officielle hændelsesopdatering.

1. 8. juli 2026 kl. 11.48 UTCresolved

   Mellem 06: 22 og 08: 23 UTC den 8. juli, anmodninger til vores amerikanske fejlrapportering API (api.honeybadger.io) oplevede en meget forhøjet fejlprocent. Under dette vindue, modtog et flertal af anmodninger om at indsende fejl, begivenheder og checkins 5xx svar eller timet ud. Fejlrapporter sendt af klientbiblioteker uden gentry logik i denne periode blev ikke modtaget og kan ikke inddrives. Webapplikationen Honeybadger, dashboards, overvågning af oppetid og vores EU-region blev ikke påvirket. Hvad skete der: Begynder på 06: 18 UTC, vi begyndte at modtage fejlrapporter på omtrent tredoble vores normale samlede API volumen. Det serviceniveau, der indtager denne trafik var på sin overnatning kapacitet niveau og blev mættet. Vores autoskalering reagerede, men de målinger, den baserede sig på, var betydeligt undermålte den fejlslagne trafik, så kapaciteten blev tilføjet alt for langsomt - fuld genopretning kom kl 08: 23 UTC, da skalering fanget og absorberet belastningen. PagerDuty blev ikke affyret for at vække et menneske på grund af forkert konfiguration. Hvad vi har gjort: Vi har rettet den underliggende skalering svaghed: API kapacitet nu også skaleres direkte på load- balancer fejlrater (tidligere vi skaleret kun på anmodning satser, og at skalering ikke reagerer hurtigt nok), og vi har fordoblet den maksimale kapacitet af indtagelse tier. Disse ændringer anvendes allerede i både USA og EU-regioner. Vi evaluerer også at tilføje adaptive per- key sats, der begrænser til indtagelse service. Vi beklager forstyrrelsen og eventuelle huller i dine fejldata under vinduet.

Automatisk oversat fra den officielle hændelsesopdatering.

1. 12. marts 2026 kl. 19.27 UTCresolved

   We had a backlog in our notifications queue that prevented timely delivery of alerts to channels such as Email, Slack, etc. The backlog has been processed as of 19:20 UTC. Our autoscaling rules for this service had a monitoring gap that prevented a scale-out to maintain a low latency on the queue. This monitoring gap has been resolved.

1. 14. januar 2026 kl. 20.40 UTCidentified

   A recent deploy has resulted in our rejecting some check-in API requests. We are in the process of deploying a fix.

2. 14. januar 2026 kl. 20.48 UTCresolved

   This incident has been resolved.

1. 22. oktober 2025 kl. 19.46 UTCidentified

   The issue has been identified and a fix is being implemented.

2. 22. oktober 2025 kl. 20.15 UTCresolved

   This incident has been resolved.

1. 22. oktober 2025 kl. 13.09 UTCidentified

   API ingestion of errors has been interrupted. We are deploying a fix.

2. 22. oktober 2025 kl. 13.11 UTCmonitoring

   The US API has been restored, and the EU API restoration is in process.

3. 22. oktober 2025 kl. 13.16 UTCresolved

   The EU API has been restored.

1. 20. oktober 2025 kl. 13.44 UTCidentified

   Our error processing pipeline has been impacted by the AWS outage earlier this morning. Errors are being processed, but the end-to-end latency is much higher than normal. We are working to restore the processing to full capacity.

2. 20. oktober 2025 kl. 17.55 UTCidentified

   We're continuing to monitor the AWS outage. Errors are being processed, but may be delayed. Uptime checks are also delayed, so you may notice lag with uptime notifications and status page updates.

3. 20. oktober 2025 kl. 19.58 UTCmonitoring

   Processing has been restored and all systems are normal. We'll continue to monitor as AWS continues to restore service. Thanks!

4. 20. oktober 2025 kl. 19.58 UTCresolved

   This incident has been resolved.

1. 20. oktober 2025 kl. 13.36 UTCresolved

   Our systems were affected by the DynamoDB outage in AWS' us-east-1 region. Impacts included not being able to log use the web app, uptime checks not being executed on time, and check-ins being falsely reported as missing. We are evaluating what steps we can take to mitigate these kinds of impacts from this kind of outage in the future. Additionally, our status page was not updated in a timely manner due to our not having configured various AWS services as dependent components of our status page. We have fixed that issue, so future AWS outages will result in our status page automatically being updated.

1. 12. februar 2025 kl. 13.45 UTCresolved

   We had an interruption in processing of check-in alerts and in-app stats from about 12:30 to about 13:30 UTC. This may have resulted in a failure or a delay in reporting check-ins that went missing during that period.

1. 22. oktober 2024 kl. 13.35 UTCmonitoring

   We have identified and corrected an issue in our data processing pipeline that may have caused delays for some customers. No data has been lost and the system should be caught up shortly.

2. 22. oktober 2024 kl. 15.30 UTCresolved

   The backlog has been processed and all systems are running normally.

1. 23. september 2024 kl. 10.37 UTCinvestigating

   We are looking into the cause of slowness and timeouts with the web app.

2. 23. september 2024 kl. 11.48 UTCmonitoring

   We have identified the cause of the slowdown and performance is back to normal.

3. 23. september 2024 kl. 11.57 UTCresolved

   This incident has been resolved.

1. 24. juli 2024 kl. 23.42 UTCidentified

   We have identified an issue with Insights event ingestion backlog. We are working on a fix.

2. 25. juli 2024 kl. 01.55 UTCmonitoring

   We have successfully pushed a fix and are waiting for the backlog to be processed. I also want to give a reminder that this only affects Insights events; error processing was unaffected.

3. 25. juli 2024 kl. 03.06 UTCresolved

   The backlog has been fully processed, so all Insights events should be ingested and available.

1. 12. februar 2024 kl. 20.22 UTCinvestigating

   We are seeing an increased backlog in our check-in processing, which is causing false alarms to be reported. We are investigating the cause of the backlog.

2. 12. februar 2024 kl. 20.39 UTCresolved

   We discovered and addressed a configuration issue that caused the delay in processing. Processing has returned to normal.

1. 6. februar 2024 kl. 20.59 UTCinvestigating

   We are currently experiencing elevated response times and timeouts from our web application.

2. 6. februar 2024 kl. 21.02 UTCinvestigating

   We are continuing to investigate this issue.

3. 6. februar 2024 kl. 21.42 UTCidentified

   We've identified an issue with aggregate database queries and are working on a fix.

4. 6. februar 2024 kl. 23.03 UTCmonitoring

   We've resolved an issue with query contention, and web application service has been restored. Our API pipeline was unaffected by this incident. Some charts on the reports tab may be outdated until we update project counts. We'll continue to monitor the situation. Sorry for the inconvenience!

5. 6. februar 2024 kl. 23.19 UTCresolved

   This incident has been resolved.

1. 17. januar 2024 kl. 19.19 UTCidentified

   We are working on restoring service

2. 17. januar 2024 kl. 19.21 UTCresolved

   This incident has been resolved.

1. 10. november 2023 kl. 19.44 UTCresolved

   We discovered this morning that our Logplex pipeline, which handles Heroku platform errors, had an interruption in processing, which caused platform errors not to be recorded. This has been corrected, the backlog is being processed, and additional monitoring has been added to avoid this issue in the future.

1. 13. juni 2023 kl. 18.59 UTCidentified

   Issues with AWS Lambda are causing our uptime check reports to be delayed. Reports are queued up and will be delivered as AWS services recover.

2. 13. juni 2023 kl. 19.34 UTCidentified

   Search indexing and notice timeline charts are also impacted by the failures with Lamba. We're continuing to queue updates for processing once Lambda resumes normal operations.

3. 13. juni 2023 kl. 21.37 UTCresolved

   Lambda is back in business, and the backlog of events triggered during the outage is being processed.

1. 25. marts 2023 kl. 01.06 UTCresolved

   Starting around 3:30pm PST, we had an incident where an unhealthy API server was in rotation in our load balancer. During that time requests routed to that server responded with a 502 which could have resulted in the clients dropping notices. At around 5:30pm PST we removed the unhealthy server from rotation. 7:22pm PST edit: Reworded message to describe that notices are not queued on the client, but dropped when the server responds in an error state

1. 24. marts 2023 kl. 08.05 UTCinvestigating

   We currently are having issues processing our notice backlog. We are looking into the issue.

2. 24. marts 2023 kl. 08.40 UTCidentified

   We have identified the issue and are working towards clearing the backlog.

3. 24. marts 2023 kl. 09.01 UTCresolved

   We identified that our workers had an issue deploying. We deployed a fix and upped our capacity to deal with the current backlog. Everything is back to normal now. Turns out this issue was related GitHub rotating their public RSA key: https://github.blog/2023-03-23-we-updated-our-rsa-ssh-host-key/

1. 9. marts 2023 kl. 09.46 UTCresolved

   We had an incident (from 1:00 to 1:27 PST) where our apps were unresponsive. They have since recovered.