65 Honeycombio incidents · 2025年1月 — official updates, affected components, duration and resolution details.
我们发现了一个问题,即没有为SLO、Triggers和Anomaly Detection发送电子邮件。 我们正在努力解决.
PM UTC下午3:53至4:08, 共有508封邮件, 我们正在努力确保这种失败模式不再发生.
自动翻译自官方事件更新。
我们目前正在调查美国地区各种服务的退化情况.
我们已经确定和隔离了受影响服务 并且正在检查中恢复 SLO加工仍然退化.
SLO处理已经完全恢复. 所有服务现已投入使用.
10月20日,我们在美国地区经历了大约三个小时的退化服务. *从协调世界时13:30到14:40,查询,Honeycomb UI,管理API,并触发器经历了高纬度和出错率,影响最高的是协调世界时13:55到14:10. 在此期间,客户可能经历了缓慢或失败的查询和缓慢的UI,一些触发提示可能已经迟到或错过. * UTC从13:26到约16:10,SLO处理被延迟,因此SLO数据可能已经出现过时,而燃烧警报可能在这段时间被延迟或错过. * UTC从13:57到14:09,对摄入API的事件约一半的请求收到5xx错误,在UTC13:48到13:55\(1%出错率\)之间影响较小,在UTC14:15\(1.9%出错率\)大约一分钟. 在这些时期被拒绝的遥测学没有被摄入,除非发送者成功再试. 客户可能看到其数据在这些时期存在漏洞,包括触发器和覆盖其的SLO。 协调世界时13点43分 我们的待命工程师在接到警告后 接到了未准时发射的触发器 我们的调查发现,主数据库的CPU被大量长期读取的查询所饱和. 我们做了一个数据库故障 减轻负载。 这短暂地缓解了压力,但同样的询问很快又在新主机上重新出现,因此没有解决问题. 然后,我们停止了这些查询,暂时停止了SLO的处理,从而减少了数据库负荷,恢复了其他受影响的服务。 一旦对负责的查询进行了数据库边超时,我们恢复了SLO处理,然后赶上,我们修复了历史的SLO数据. 这些查询来自当Honeycomb环境的设置发生变化时 我们的SLO处理运行的检查. 在绝大多数情况下,它以几毫秒的速度完成. 然而,对于我们0.1%的环境, 拥有异常多的 SLO 和计算过的字段, 数据库运行它的方式非常低效, 这些查询在请求的服务超时后继续进行,多次尝试增加了几个服务共享的数据库的负荷. 为了防止这种情况再次发生,除了数据库边的超时,我们改变了数据库如何执行这样的环境查询,这使得其运行时间从分钟缩短到几毫秒. 我们还调整了警戒阈值,以确保随时待命的工程师能够迅速了解未来数据库负荷的变化.
自动翻译自官方事件更新。
在协调世界时13:57至14:09, Honeycomb的美国生产环境中摄入的事件因同时发生的一起事件而退化。 在这个窗口里,大约53%的入门活动都收到5xx错误.
自动翻译自官方事件更新。
Canvas不是给美国和欧盟地区的顾客加载的, 我们已经确定了我们所认为的原因.
我们已部署固定装置,正在恢复正常的Canvas功能.
这一事件已经得到解决.
自动翻译自官方事件更新。
从11:36UTC到12:43UTC,在US1中读取更古老数据的查询返回出错或超时. 对最近数据的查询没有受到影响,没有遥测。 问题已经确定,解决办法已经到位。 我们目前正在监测局势,以确认解决办法.
修复已经就绪,我们已确认分辨率. 查询和SLO处理正常.
自动翻译自官方事件更新。
10: 30到11: 00 AM EDT,我们经历了高水平的查询故障和缓慢. 影响已经减弱,我们目前正在调查和监测局势.
从上午9点50分开始 时间7月31日,我们收到警报 围绕询问健康和延迟。 原因最终被追踪到两个不同因素的冲突. 首先,我们在查询架构中引入了Lambda Conventions \ (LMI\),它在许多方面表现与经典的Lambda相似,并被引入作为查询性能改进工作的一部分,目的是让用户体验更好更快。 不幸的是,我们发现的经典羊肉和LMI的区别在于,在LMIs上不存在15分钟的经典羊肉硬站,这意味着我们查询系统中的某些假设不再存在. 在事件发生时,我们的新Canvas的功能 正在运行批次的查询 客户应该有 自我强制一分钟。 这份超时声明没有被适当传达回羊肉会,虽然与经典羊肉会的任务会在15分钟后自动被切断,但新的LMI并没有强制实施这种安全机制. 因此,一些大型查询运行了长达30分钟,使查询时间超乎寻常,导致用户试图运行新鲜查询出现出错. 系统恢复了10:10 AM CT,20分钟后,当大问讯清出后,我们继续调查事件原因. 一旦我们发现我们查询架构中的这一弱点,我们就开始努力纠正它,防止同一事件再次发生。 我们已经运送了所有 相关的事件后续.
自动翻译自官方事件更新。
我们正看到欧盟区域间歇性地失败。 我们正在积极调查原因.
我们继续看到有触发因素的问题,也注意到有疑问的问题。 我们正在调查这两个原因.
我们查明了问题的根源,实施了临时解决办法,并正在设法永久解决办法。 我们继续监测局势.
我们看到业绩出现倒退,触发器和查询器部分退化。 我们正在努力补救.
除专门接触的客户外,所有客户的部分退化问题均已得到解决。 我们正在研究减缓措施,以便我们完全恢复每个人的触发功能.
我们正在继续监测任何其他问题.
查询和触发的退化已经解决,我们恢复了完全的功能.
自动翻译自官方事件更新。
We are currently investigating this issue.
Clearing the CDN cache appears to have solved the issue.
This incident has been resolved.
Starting at 14:24 PT, there was an issue with the Activity Log causing events to stop being processed. You may notice a gap starting at that time. We are slowly backfilling these events and do not expect any data loss to occur. We expect the full data log to be restored around 17:30 PT.
Backfill of the Activity Log events has completed. All events during the affected period should be present.
We are investigating an issue with delayed ingest in the EU region. SLOs and Trigger evaluations may also be delayed.
We are monitoring the situation. Events may have been delayed between 17:53 and 18:04 UTC.
This incident has been resolved.
We are investigating an issue with delayed ingest in the US and EU region. Our engineers are rolling back a recent deploy. SLOs and Trigger evaluations may also be delayed.
We have rolled back the deploy and ingest service has recovered. There has been an ingest outage from 17:55 - 18:14 UTC. We have also observed an delay for Service Maps.
Triggers, SLOs and Service Maps are recovered. We are continuing to monitoring the Ingest Service
All services have been fully recovered. Impact window: 17:55–18:17 UTC Regions affected: Primary impact in US. The EU region was affected to a much lesser degree. During this period, the following effects may have occurred: Ingest: Some data was not ingested, resulting in complete or partial data loss for events sent during the window. Triggers: Triggers may have failed to fire within the impact window. SLOs: SLI values across the affected window are skewed by the missing data and may show artificial dips or accelerated budget burn. Service Maps: Maps covering the outage window are incomplete. Services and dependencies may be under-counted, as their traces were not ingested. We apologize for the disruption. Please reach out if you have any questions about how this may have affected your data.
On June 24, we experienced approximately 30 minutes of severe data ingestion degradation in Honeycomb’s US and EU instances. During this degradation, Honeycomb rejected a significant percentage of inbound telemetry, and customers would have seen up to a half-hour gap in ingested telemetry. Additionally, during this same time window, customers would have experienced data processing degradation and service disruption within Anomaly Detection and Service Maps. A deployment containing a change to how services retrieve dataset schemas into local caches caused a cascading set of failures, resulting in failed remote cache retrievals, and subsequently, a MySQL stampede due to several services falling back to retrieving schemas from the database. This stampede resulted in significant memory utilization growth, causing some services to crash loop with out-of-memory exceptions while retrieving schemas. Customers would have seen 5xx errors from our ingestion APIs when this occurred, because ingestion services were among those that were crash looping. This also included crash looping from services responsible for Anomaly Detection and Service Maps. Right after these service crash loops began, alerts fired for the crashing services, and procedures were run to pin all services back to the last known good build before the breaking change landed. Once services were running the previous deploy’s build, the 5xx error responses from ingestion APIs returned to nominal baseline levels, and services that were crash looping recovered and resumed normal operation. The underlying issue centered on how dataset schema cache payloads and metadata were serialized to memcached for remote cache retrieval, and how memcached clients deserialize that data before writing it to a local cache. The code change contained a feature flag to toggle the serialization behavior, and backwards-compatibility safeguards for clients deserializing different schema versions from memcached. However, services’ schema deserialization prior to the feature flag flip on did not behave as intended, treating each backwards-compatible schema read as a hard error rather than as a fallback, which subsequently caused every remote schema cache read to fall through to the database. Subsequent analysis and review of the code identified the issue and we have re-deployed this change with a fix along with additional tests. Services now correctly and successfully deserialize schemas from memcached in a safe, backwards-compatible manner.
We are aware of an issue affecting Honeycomb MCP. Write tool functionality may be degraded. Read tools are unaffected.
We are aware of an issue affecting Honeycomb MCP. Write tool functionality may be degraded. Read tools are unaffected.
MCP Write Tools have been restored. MCP users may need to reconnect to see all available tools.
Beginning June 17th, enhance-related requests were failing to complete. We've since identified and deployed a fix, and enhance is now fully operational. Thank you for your patience.
We are continuing to investigate intermittent slowness and query failures affecting our Production EU Region. We will provide an update as soon as we have more information.
Querying in our Production EU Region is fully operational. We will continue to monitor for any abnormal behavior.
We have identified the root cause of the intermittent slowness and query failures affecting our Production EU Region. The issue was triggered by a large volume of dataset deletions that caused elevated database load, resulting in query instability. No data was lost during this incident. Service has been restored. A fix has been applied to help prevent recurrence.
We are currently investigating the cause of slowness and query failures in our Production EU Region.
The issue is now resolved and querying is back to fully functional.
Requests to Honeycomb's Query Data and Management APIs in the production-eu1 environment encountered increased error rates between approximately 15:35 and 22:24 UTC. Event ingestion was not affected. The issue has been resolved.
We are investigating an issue with delayed ingest in the EU region. Received events are still being stored, but there may be a delay in event retrieval in queries. SLOs and Trigger evaluations may also be delayed.
Queries should no longer be delayed. SLOs and Trigger evaluations remain impacted.
The issue has been identified and a fix is being implemented.
All services are now healthy.
We are currently investigating this issue.
This incident has been resolved.
Canvas agent cannot load honeycomb internal tools.
This incident has been resolved.
We have identified and are working to resolve an issue that is causing query results to return inconsistent results.
Impact from this issue has concluded. For a period of time (different between US and EU instances, noted below), queries spanning data older than the most recent 2 hours with certain GROUP / WHERE clauses returned inconsistent results. Data was never lost, and reruns of the same queries will show the previously missing data. US impact times: 21:50 UTC - 23:20 UTC EU impact times: 20:50 UTC - 23:20 UTC
This is an administrative update, marking the incident as Resolved. There has been no further impact since 23:20UTC on May 21 2026.
