Elevated failure rates on the US error reporting API
- resolved
Between 19:54 and 20:07 UTC on August 15, requests to our US error reporting API (api.honeybadger.io) experienced an elevated failure rate. During this window, a majority of requests to submit errors and events timed out or received 5xx responses. Browser (JavaScript) error reporting, source map uploads, the Honeybadger web application, dashboards, uptime monitoring, and our EU region were not affected. What happened: Beginning at 19:54 UTC, a single source began submitting error reports at roughly 150 times its normal rate, with payloads several times larger than average. This more than doubled the total volume of data arriving at our ingestion tier, which saturated within a minute. Because the saturated service kept accepting connections rather than returning errors outright, our autoscaling and alerting — which watch for error responses — did not register the problem quickly, and capacity was added later than it should have been. Service recovered at 20:07 UTC once additional capacity came online. Most affected reports were retried successfully by our client libraries once the API recovered, so the large majority of data submitted during the window was ultimately received. Reports sent by libraries without retry logic were not received and cannot be recovered. What we've done: our ingestion tier now scales on response latency in addition to error rates, so this kind of saturation is detected within a minute or two rather than only once errors appear. We've also raised the tier's baseline capacity and widened our paging alerts to cover failures that don't surface as server errors. We're additionally adding limits on how much data a single source can submit. We're sorry for the disruption, and for any gaps this caused in your error data during the window.