Honeybadger 中断历史

1. 2026年8月15日 UTC 20:48resolved

   在协调世界时8月15日19:54至20:07之间,向美国错误报告API(api.honybadger.io)的请求失败率上升. 在这一窗口中,大多数提交错误和事件的请求被超时或收到5xx的答复。 浏览器(JavaScript)错误报告,源地图上传,Honeybadger网络应用程序,仪表板,上线监测,以及我们的欧盟区域没有受到影响. 发生的情况:从协调世界时19:54起,一个单一来源开始以大约是其正常速度的150倍提交出错报告,有效载荷比平均数大几倍。 这比达到摄入水平的数据总量翻了一番多, 摄入水平在一分钟内饱和。 由于饱和服务持续接受连接而不是直接返回出错,因此我们的自动缩放和警报——它监视错误反应——没有迅速登记出问题,能力被加起来的时间比它本该晚. 服务在协调世界时20:07恢复,一旦额外的容量上线. 多数受影响的报告在API恢复后被我们的客户端图书馆成功重审,所以在窗口期间提交的绝大多数数据最终都收到. 没有重试逻辑的图书馆发送的报告没有收到,无法收回。 我们所做的: 我们的摄入级现在除了错误率外, 还在反应延迟度上, 所以这种饱和度在一两分钟内被检测出来, 而不是只在错误出现之后。 我们也提高了等级的基线能力,并扩大了我们的呼号警报,以覆盖不作为服务器错误表面的故障. 我们正在额外增加限制 一个单一来源可以提交多少数据。 我们为中断而感到抱歉 也为您在窗口中错误数据 造成的任何漏洞而道歉.

自动翻译自官方事件更新。

1. 2026年7月8日 UTC 11:48resolved

   协调世界时7月8日06:22至08:23之间,向美国错误报告API(api.honybadger.io)的请求出现了高故障率. 在这一窗口中,大多数提交错误、事件和报到请求都收到了5xx回复或超时。 客户端库在这段时间里发送的没有重试逻辑的错误报告没有收到,无法恢复. Honeybadger网络应用程序、仪表板、上下班时间监测以及欧盟区域没有受到影响。 发生的事情:从协调世界时06:18开始, 我们开始收到错误报告, 大约是正常API总量的三倍。 吞吐这种流量的服务级别达到一夜之间的能力水平并变得饱和. 我们的自动缩放响应了,但是它所依赖的衡量标准对故障流量的测量严重不足,因此容量的加载速度太慢了——在协调世界时08:23,在缩放时完全恢复并吸收了负载. 使问题更加复杂的是,PagerDuty警报没有因为配置错误而发出来来唤醒一个人. 我们所做的:我们解决了潜在的缩放弱点:API能力现在也直接在负载-平衡器误差率上进行缩放(之前我们只根据请求率进行缩放,而缩放反应不够快),我们把摄取层的最大容量翻了一番. 这些变化已经部署在美国和欧盟地区。 我们还在评价 增加适应性的每个键率 限制摄入服务。 我们为中断而感到抱歉 也为您在窗口中错误数据 造成的任何漏洞而道歉.

自动翻译自官方事件更新。

1. 2026年3月12日 UTC 19:27resolved

   We had a backlog in our notifications queue that prevented timely delivery of alerts to channels such as Email, Slack, etc. The backlog has been processed as of 19:20 UTC. Our autoscaling rules for this service had a monitoring gap that prevented a scale-out to maintain a low latency on the queue. This monitoring gap has been resolved.

1. 2026年1月14日 UTC 20:40identified

   A recent deploy has resulted in our rejecting some check-in API requests. We are in the process of deploying a fix.

2. 2026年1月14日 UTC 20:48resolved

   This incident has been resolved.

1. 2025年10月22日 UTC 19:46identified

   The issue has been identified and a fix is being implemented.

2. 2025年10月22日 UTC 20:15resolved

   This incident has been resolved.

1. 2025年10月22日 UTC 13:09identified

   API ingestion of errors has been interrupted. We are deploying a fix.

2. 2025年10月22日 UTC 13:11monitoring

   The US API has been restored, and the EU API restoration is in process.

3. 2025年10月22日 UTC 13:16resolved

   The EU API has been restored.

1. 2025年10月20日 UTC 13:44identified

   Our error processing pipeline has been impacted by the AWS outage earlier this morning. Errors are being processed, but the end-to-end latency is much higher than normal. We are working to restore the processing to full capacity.

2. 2025年10月20日 UTC 17:55identified

   We're continuing to monitor the AWS outage. Errors are being processed, but may be delayed. Uptime checks are also delayed, so you may notice lag with uptime notifications and status page updates.

3. 2025年10月20日 UTC 19:58monitoring

   Processing has been restored and all systems are normal. We'll continue to monitor as AWS continues to restore service. Thanks!

4. 2025年10月20日 UTC 19:58resolved

   This incident has been resolved.

1. 2025年10月20日 UTC 13:36resolved

   Our systems were affected by the DynamoDB outage in AWS' us-east-1 region. Impacts included not being able to log use the web app, uptime checks not being executed on time, and check-ins being falsely reported as missing. We are evaluating what steps we can take to mitigate these kinds of impacts from this kind of outage in the future. Additionally, our status page was not updated in a timely manner due to our not having configured various AWS services as dependent components of our status page. We have fixed that issue, so future AWS outages will result in our status page automatically being updated.

1. 2025年2月12日 UTC 13:45resolved

   We had an interruption in processing of check-in alerts and in-app stats from about 12:30 to about 13:30 UTC. This may have resulted in a failure or a delay in reporting check-ins that went missing during that period.

1. 2024年10月22日 UTC 13:35monitoring

   We have identified and corrected an issue in our data processing pipeline that may have caused delays for some customers. No data has been lost and the system should be caught up shortly.

2. 2024年10月22日 UTC 15:30resolved

   The backlog has been processed and all systems are running normally.

1. 2024年9月23日 UTC 10:37investigating

   We are looking into the cause of slowness and timeouts with the web app.

2. 2024年9月23日 UTC 11:48monitoring

   We have identified the cause of the slowdown and performance is back to normal.

3. 2024年9月23日 UTC 11:57resolved

   This incident has been resolved.

1. 2024年7月24日 UTC 23:42identified

   We have identified an issue with Insights event ingestion backlog. We are working on a fix.

2. 2024年7月25日 UTC 01:55monitoring

   We have successfully pushed a fix and are waiting for the backlog to be processed. I also want to give a reminder that this only affects Insights events; error processing was unaffected.

3. 2024年7月25日 UTC 03:06resolved

   The backlog has been fully processed, so all Insights events should be ingested and available.

1. 2024年2月12日 UTC 20:22investigating

   We are seeing an increased backlog in our check-in processing, which is causing false alarms to be reported. We are investigating the cause of the backlog.

2. 2024年2月12日 UTC 20:39resolved

   We discovered and addressed a configuration issue that caused the delay in processing. Processing has returned to normal.

1. 2024年2月6日 UTC 20:59investigating

   We are currently experiencing elevated response times and timeouts from our web application.

2. 2024年2月6日 UTC 21:02investigating

   We are continuing to investigate this issue.

3. 2024年2月6日 UTC 21:42identified

   We've identified an issue with aggregate database queries and are working on a fix.

4. 2024年2月6日 UTC 23:03monitoring

   We've resolved an issue with query contention, and web application service has been restored. Our API pipeline was unaffected by this incident. Some charts on the reports tab may be outdated until we update project counts. We'll continue to monitor the situation. Sorry for the inconvenience!

5. 2024年2月6日 UTC 23:19resolved

   This incident has been resolved.

1. 2024年1月17日 UTC 19:19identified

   We are working on restoring service

2. 2024年1月17日 UTC 19:21resolved

   This incident has been resolved.

1. 2023年11月10日 UTC 19:44resolved

   We discovered this morning that our Logplex pipeline, which handles Heroku platform errors, had an interruption in processing, which caused platform errors not to be recorded. This has been corrected, the backlog is being processed, and additional monitoring has been added to avoid this issue in the future.

1. 2023年6月13日 UTC 18:59identified

   Issues with AWS Lambda are causing our uptime check reports to be delayed. Reports are queued up and will be delivered as AWS services recover.

2. 2023年6月13日 UTC 19:34identified

   Search indexing and notice timeline charts are also impacted by the failures with Lamba. We're continuing to queue updates for processing once Lambda resumes normal operations.

3. 2023年6月13日 UTC 21:37resolved

   Lambda is back in business, and the backlog of events triggered during the outage is being processed.

1. 2023年3月25日 UTC 01:06resolved

   Starting around 3:30pm PST, we had an incident where an unhealthy API server was in rotation in our load balancer. During that time requests routed to that server responded with a 502 which could have resulted in the clients dropping notices. At around 5:30pm PST we removed the unhealthy server from rotation. 7:22pm PST edit: Reworded message to describe that notices are not queued on the client, but dropped when the server responds in an error state

1. 2023年3月24日 UTC 08:05investigating

   We currently are having issues processing our notice backlog. We are looking into the issue.

2. 2023年3月24日 UTC 08:40identified

   We have identified the issue and are working towards clearing the backlog.

3. 2023年3月24日 UTC 09:01resolved

   We identified that our workers had an issue deploying. We deployed a fix and upped our capacity to deal with the current backlog. Everything is back to normal now. Turns out this issue was related GitHub rotating their public RSA key: https://github.blog/2023-03-23-we-updated-our-rsa-ssh-host-key/

1. 2023年3月9日 UTC 09:46resolved

   We had an incident (from 1:00 to 1:27 PST) where our apps were unresponsive. They have since recovered.