Buildkite 장애 기록

We're seeing increased latency and error rates for a subset of our customers. We're currently investigating and will provide status updates as they become available.

We've identified the issue, we're continuing working on rolling out a fix. Impact is restricted to the builds create API for a subset of tenants.

A fix has been deployed, services are recovering.

We have seen a full recovery of services.

We're observing increased latency for all our customers. We're currently investigating and will provide status updates as they become available.

We've isolated the issue to elevated load on our REST API service and are working to mitigate. The agent and stacks API isn’t affected.

We've isolated the issue to elevated load on our REST API service and have mitigated the issue. The agent and stacks API isn’t affected.

The mitigation applied before the last update had the intended effect, and we have seen recovery in REST API latency.

We're observing increased latency and error rates for a subset of our customers on the Agent API. We're currently investigating and will provide status updates as they become available.

Between 00:05 - 00:34 UTC, a subset of customers experienced increased latency and timeout errors on the Agent API. This impacts job assignment. At peak impact, we saw an error rate of 1.3% of requests and job acceptance latency up to 53s.

We have received reports email deliveries have not been working, affecting signup and invite emails as well as build notification emails. This issue has now been resolved.

We are investigating delays to build and job notifications for a subset of customers.

We've spotted that something has gone wrong. We're currently investigating the issue, and will provide an update soon.

This incident has been resolved.

## Service Impact Customers experienced delayed Buildkite notification delivery. The customer impact varied depending on how those notifications are used. For some customers, delayed notifications also delayed downstream CI, merge, or deployment workflows. ## Incident Summary On 28 May, Buildkite experienced elevated notification delivery latency after part of our notification-processing infrastructure became underprovisioned. This happened because the Prometheus service used by our EKS autoscaling path ran out of storage, which meant some EKS-based workers could not autoscale correctly while queues were growing. We mitigated the incident by moving affected workloads back to our previous ECS-based infrastructure and manually increasing worker capacity. Recovery took longer than expected because the rollback path did not fully handle this scenario. ### Impact window 1 At 20:01 UTC, notification-processing workers became underprovisioned and notification delivery latency increased. We detected the issue through internal queue latency monitoring and began shifting affected workloads from EKS back to ECS. This rollback took longer than expected because the ECS services we were rolling back to were not ready to immediately take the full load. Engineers had to manually adjust scaling configuration and worker counts while the incident was active. Notification latency recovered for most customers by 21:00 UTC. ### Impact window 2 A second, shorter impact window occurred between 22:12 UTC and 22:40 UTC for a subset of customers. After the first recovery, some workloads were still running on EKS and had started autoscaling again after Prometheus recovered. We incorrectly believed those workloads were no longer serving traffic. When we reconciled our infrastructure configuration, those EKS workloads were scaled down before their ECS equivalents had been fully scaled up. This caused another period of underprovisioning for some notification-processing workers. We resolved it by completing the rollback and scaling the remaining affected ECS services. ### Customer Impact The impact was not identical for every customer. For customers who use Buildkite notifications as an input to other CI or deployment systems, notification latency can delay those downstream workflows. Some customers also experienced secondary or longer-running effects based on the specific notification types, retry behaviour, or integrations involved. We are following up directly with affected customers where their impact differed from the general incident. ## Changes we're making We have made the following immediate changes: * Increased Prometheus storage capacity and reconciled that change in infrastructure-as-code. * Added monitoring to alert before Prometheus storage exhaustion can affect autoscaling. * Moved affected notification-processing workloads back to known-good ECS capacity. * Fixed GitHub notification retry behaviour for a class of errors that could cause repeated retries and extend notification delays. We are also making the following reliability improvements: * Hardening the EKS-to-ECS rollback process so it verifies destination capacity, autoscaling configuration, and traffic movement before and during rollback. * Reviewing other EKS control-plane dependencies, including KEDA and Karpenter, to ensure their CPU, memory, and storage allocations are appropriate for production load. * Reassessing the order and pace of future EKS migrations so customer-critical workloads move more gradually and with clearer settling periods. * Improving customer-level monitoring for notification delivery latency, so we can detect customer-impacting regressions earlier. * Reviewing which notification types are on the scheduling or CI hot path for customers, and whether they need tighter latency expectations, separate queueing, or more specific alerting than general notification work. ## Areas we are improving: incident communication During this incident, our public status page did not reflect customer-visible impact as quickly or clearly as it should have. In particular, notification delivery latency can affect customers differently depending on how notifications are used in their CI and deployment workflows. We are improving how we communicate during notification latency incidents by: * Updating the status page earlier when notification latency is likely to affect customer workflows * Making status page updates clearer about the customer-visible impact, not just the affected internal service * Improving internal escalation paths for customers who report critical CI impact before the incident is fully understood * Using customer-level notification latency monitoring to help identify affected customers sooner

We're observing increased latency and error rates for a subset of our customers. We're currently remediating and will provide status updates as they become available.

We've identified the problem and have completed the remediation steps, we are now monitoring as service resumes.

We see processing time for all affected services has returned to normal as of 20 minutes ago.

We think the impact from the issue is over.

## Service Impact A subset of customers experienced elevated latency in notification delivery. ## Incident Summary While migrating a subset of our background processing services to Amazon EKS, we encountered an issue with delivery of internal metrics. The discovered issue did not impact performance or availability, but would have impaired our ability to detect such problems if they occurred. Out of an abundance of caution we decided to revert the migration, and moved those services back to the original infrastructure on AWS Fargate. When migrating to EKS, we scale down and disable automatic scaling on Fargate. This allows us to quickly migrate back by scaling up Fargate. When we moved the workloads back to Fargate to restore internal metrics, we missed the step to re-enable autoscaling. As a result, the affected services did not have sufficient capacity and could not keep up with incoming work. We re-enabled autoscaling promptly once the problem was discovered, and provisioned extra capacity for customers where a backlog of work had accumulated. Between 09:17 and 10:17 UTC, a small subset of our customers were impacted. Individual customers experienced a limited outage of notification services, which lasted between 35 and 58 minutes within this window, if there was any impact at all. The migration is performed in small batches, so not all customers experienced this incident. ## Changes we're making * We are simplifying the runbook used to rollback migrations in the event of incidents. * We are adding more verification steps to the migration process.

We are investigating delays to notifications across all customers

We have identified the issue and applied mitigations and are monitoring recovery We have determined that only a subset of customers are affected by the notification latency.

We are seeing recovery across affected customers and continue to monitor

The incident is resolved

## Service Impact A subset of our customers experienced elevated latency in our notification delivery, build dispatch and metrics services. ## Incident Summary We are in the process of migrating our underlying compute platform from AWS Fargate to AWS EKS for our production workloads. We are migrating our services in small batches so we can verify stability as we go. Between 15:42 and 17:33 our EKS Prometheus server began to need more memory than was available on the host where it was running. This was caused by autoscaling operations that increased the number of pods tracked by Prometheus, which in turn increased the Prometheus server's memory requirement. The host killed the Prometheus server process, which was restarted shortly after by the Kubernetes control plane. In the interim, the metrics used for application autoscaling were unavailable. The unavailable metrics meant that the affected services were not being triggered to scale up, resulting in the observed delays. Prometheus exceeded the host's available memory again soon after restarting, which caused the cycle to repeat. The on call team followed a prepared documentation to shift load on the affected services back to Fargate. The majority of customers saw complete recovery from 16:49. A handful of customers had developed such a large backlog during the period of higher latency, that they had to be manually scaled up further. All customers saw full recovery by 17:33. ## Changes we're making We have already made the following changes to our rollout of EKS for production workloads: * Upsized the underlying system nodes. * Set higher requests and limits for the Prometheus server so it can handle more product load. * Reviewed and set any missing requests and limits for all new EKS resources, ensuring that EKS has all the required information to prevent accidental resource contention. * Added more observability and monitors for EKS pod and node health to help us identify root causes quickly during future incidents. We have since migrated all these services back to EKS and observed successful scaling well beyond the limits we encountered during this incident.

Ingestion of Test Engine execution data from an internal queue to a data store stalled, has been resumed, and is working through the backlog. Visibility of test executions from the past hour hours will be delayed for approximately a further one hour. This has been a recurring issue; an architectural change is coming soon to eliminate this failure mode.

Processing of the backlog is complete.

We've spotted that something has gone wrong. We're currently investigating the issue, and will provide an update soon.

Additional capacity was added to our redis caches. This triggered a failover between UTC 15:10 - 15:14 and there was a spike of errors on the REST and GraphQL APIs. Customers would have seen some errors in the Buildkite UI during this period as well. We have been monitoring the situation since then and things have returned to baseline.

We are currently experiencing delayed processing of Test Engine data. We have identified and applied a fix for the issue but are expecting to continue to experience delays while we clear the ingestion backlog

The fix was successful and the backlog has now been cleared.

We are currently experiencing delayed processing of Test Engine data. We have identified and applied a fix for the issue but are expecting to continue to experience delays while we clear the ingestion backlog. At the current processing rate we expect the backlog to be cleared by approximately Sat 09 May 2026 00:00 UTC

The delayed backlog is now cleared and Test Engine ingestion is operating normally.

We are currently experiencing delayed processing of Test Engine data. We have identified and applied a fix for the issue but are expecting to continue to experience continued processing delays while we clear the ingestion backlog. At the current processing rate we expect the backlog to be cleared by approximately Fri 08 May 2026 13:30 UTC.

The Test Engine backlog is now cleared and operating normally.

We're aware that AWS is reporting availability zone failures in us-east-1. We are monitoring the situation but so far there is no customer impact.

A small subset of our customers are experiencing delayed notifications. We are actively provisioning additional capacity for these customers. Availability zone automatic failovers are occurring in response to the outage, and this is causing some brief error blips for some customers.

We are continuing to actively monitor the impacts of this availability zone outage for Buildkite customers. Some transient errors are visible due to availability zone failover events.

We have provisioned additional capacity in unaffected availability zones so that they are able to support the additional load. Automatic failovers continue to occur where necessary. Some latency and transient errors will be visible to customers.

We are actively moving resources out of us-east-1c. Similar brief latency and error blips will be visible to customers while these manual failovers occur.

We are continuing to move infrastructure resources out of the affected AWS Availability Zone. Brief latency and error blips will unfortunately continue while these manual failovers occur.

We are continuing to move infrastructure resources out of the affected AWS Availability Zone. Brief latency and error blips may continue while these manual failovers occur.

We are continuing to move infrastructure resources out of the affected AWS Availability Zone. Brief latency and error blips may continue while these manual failovers occur. (Apologies if you receive duplicated notifications for this update.)

Despite the ongoing AWS incident, our own services are now stable. We are continuing to monitor our services closely, and are ready for further action should the need arise. We are also watching AWS services closely as they recover.

The upstream AWS incident in us-east-1 has been resolved by AWS, and all Buildkite services are operating normally. No further customer impact is expected. We appreciate your patience during this incident.

## Service impact On 8th May 2026 UTC, between 00:00 and 07:30 UTC, some customers would have seen intermittent errors and latency spikes across many areas of the platform. ## Incident Summary The AWS availability zone incident in `use1-az4` triggered our automatic availability failover mechanisms on database and cache clusters, as per AZ-failure tolerant design. During the failover we saw some isolated request errors that were handled by client-side retries in the agent. Customer workloads were either entirely undisrupted or in the worst case saw elevated latency for a period of up to 5 minutes. Throughout the incident we monitored customer impact and prepared additional resources in a healthy availability zone to manually failover to if the automated systems proved insufficient. These were not necessary, and all our infrastructure self healed.

We've spotted that something has gone wrong. We're currently investigating the issue, and will provide an update soon.

We've identified the issue and are working on applying mitigations. At this time we can confirm inbound and outbound webhooks, and notifications are delayed.

We have now monitoring the incident. We are seeing most customers have recovered, and some showing signs of recovery.

All customer workloads have now recovered.

## Service Impact On 2026-05-07 between 22:22 UTC and 00:14 UTC the next day \(~1 hour 52 minutes\), Buildkite customers experienced delays across several Pipelines features: * Inbound webhook processing was delayed. * Outbound webhook delivery and build notifications were delayed. * Job dispatch was delayed for jobs queued during the window. No data was lost. Webhooks and notifications were queued for retry and delivered after the underlying database recovered. Customers with high webhook volumes or time-sensitive build dispatches were most affected. Some customers continued to see lingering latency for a short period after the underlying database recovered, while queued work drained. ## Incident Summary The Pipelines product depends on a shared database that brokers inbound API and webhook requests across all Pipelines shards. At 22:22 UTC, this database's writer instance began saturating under contention on internal locks and could no longer keep up with the volume of work directed at it. Because every Pipelines shard depends on this single shared database, the slowdown affected all customers. Contributing factors: * The shared database's writer was provisioned with very little spare capacity, leaving no headroom to absorb a load spike. * A specific category of background worker, the one that processes incoming webhooks, shares a queue with workers that dispatch builds, so back-pressure on webhook processing cascaded into delayed builds. * A small number of slow external webhook endpoints held worker capacity open inside long-running transactions, amplifying load on the shared database. This is a pattern we have seen before. * Our detection at the database tier was not specific enough to catch the saturation directly; we detected the problem ~12 minutes after impact began, via a downstream signal on Sidekiq queue latency. Mitigation: we provisioned a substantially larger database replica, failed over to it, and re-enabled processing shard-by-shard. We then resized a second replica to match, restoring our ability to fail over again if needed. We initially suspected the incident might be upstream of a separate availability-zone event experienced by our cloud provider. We have since ruled this out: the cloud provider's availability-zone event was declared after this incident was resolved, and the provider has confirmed the affected database instance was not impacted by that event. ## Changes we're making * **Decoupling inbound webhook ingestion from the shared database.** We are currently working through shard isolating inbound webhooks. The risk had already been identified and work had begun to mitigate the risk. In a stroke of bad fortune, it hadn’t been fully rolled out before we incurred the spike in load. Once this has been completed the underlying databases will be one step closer to be broken. * **Operator controls to pause queues during incidents.** We have shipped admin controls to pause individual Sidekiq sets. We’ve also improved our remediation tooling to give our on-callers more control over the ingestion pipeline for managing back-pressure on this specific shared database. * **Database-tier alerting.** We are adding alerts on database write latency and lock-wait activity, so we detect saturation at the source rather than via downstream queue latency. * **Capacity normalisation.** We are normalising autoscaling capacity across shards, so a spike to a single shard's queue can be absorbed by autoscaling rather than cascading.

We've spotted that something has gone wrong. We're currently investigating the issue with new builds not starting.

We've identified issue with job acquiring endpoint. We're rolling back now. We'll provide next update in ~20 minutes.

We're currently seeing recovery at 50% rate. We'll provide next update soon.

We've reverted a change that caused stale environment variables provided to acquire job used in Hosted Agents, agent-stack-k8s and other agent implementations using acquire job.

## Service Impact On May 7th between 08:14 and 09:25 UTC customers using [hosted agents](https://buildkite.com/docs/agent/buildkite-hosted), the [k8s stack](https://buildkite.com/docs/agent/self-hosted/agent-stack-k8s) and the Buildkite agent [acquire job feature](https://buildkite.com/docs/agent/cli/reference/start#run-a-single-job) experienced failures when starting a job, resulting in an error message `Missing agent. See: buildkite-agent bootstrap --help`. ## Incident Summary As part of our efforts to improve the performance of our platform we shipped a change to how database commits were grouped together. This change inadvertently caused certain job environment variables to be omitted when a job was assigned via the acquire method. These variables are key to our integration with hosted agents as well as customers using our Kubernetes stack and their omission caused any jobs launched via this method to fail. While we had test coverage that ensured these variables were populated, these tests did not exercise the code path used by acquire job effectively enough to indicate this problem before it was deployed. Furthermore, while parts of the change were placed behind a feature flag, the refactor of the code that caused this bug were not. Attempts to restore service by rolling back this change were hindered by the current revision being selected for deploy instead of the previous. The initial rollback was triggered at around 08:40 UTC, but it wasn’t until 09:05 we realised the mistake and began the rollback to the correct revision. The rollback started deploying at 09:16 and at 09:25 service was fully restored. ‌  ## Changes we're making We have added additional rollback gates making it easier to identify when the incorrect revisions has been selected. Our test suite will be expanded to include contract tests for APIs used by Hosted Agent and the Kubernetes stack. Additionally, we are configuring Hosted Agent synthetic tests to automatically page on-call engineers when failures occur, improving our response times.

A process writing test results to our Test Engine data store stalled, we've restarted the process and are seeing it catching up. We expect to be fully caught up on the backlog within the next couple of hours.

We've identified the issue and the system is currently processing the backlog of test executions

Processing of test execution ingestion data has successfully caught up.

We are investigating delays with build and job notifications for a subset of customers.

We've identified the source of the notification delays affecting a subset of our customers. Our engineers are applying remediations to reduce these delays.

Applied remediations have resolved the previous notification delays affecting a subset of our customers. We're continuing to monitor the affected services for stability.

Notifications continue to be delivered without delay for the previously affected subset of customers. This incident is resolved.

We're observing increased latency and error rates in the Agent API for a subset of our customers. We're currently investigating and will provide status updates as they become available.

An increase in requests has lead to the API service being temporarily saturated. We have updated rate limits to ensure this doesn't re-occur and will add further resources if necessary

We've identified a service change that is causing a service disruption. We are reverting this change.

We've corrected the issue that caused this disruption and normal service has been restored. We are monitoring the situation now.

This incident has been resolved.

### Service Impact Between 02:29 and 02:48 UTC on May 1st 2026, the Buildkite dashboard, Agent API, REST API and GraphQL were severely degraded for the majority of customers. During this period, customers were unable to reliably load pages in the dashboard, and agents experienced errors when communicating with the Agent API. Service was restored for all customers by 02:48 UTC, with a small number of follow-up support cases handled in the hours afterward. ### Incident Summary At 02:29 UTC, a database migration was applied that renamed a column on the users table in our foundation database, which is a core, unsharded table referenced across the Buildkite platform. The migration ran before the new application code was deployed. During the rollout window, the application code already running in production still expected the original column name. Because database migrations run before application code is deployed, queries that loaded a user record began failing immediately across the product, and continued to fail until the column was manually restored. The impact was detected at 02:29 UTC by an automated alert, and several customers reached out to support within minutes. Initial investigation focused on preparing an emergency rollback, but at approximately 02:41 UTC it was identified that the change was an unsafe migration and that a rollback would not restore service. We then moved to manually recreate the renamed column on the users table. This change was applied at approximately 02:48 UTC, at which point error rates returned to normal levels across the affected services. Total customer-facing impact was 19 minutes. ### What contributed to this happening Three things came together to allow this change to ship without the safety pre-steps that would normally precede a destructive migration: The migration was generated by an LLM-assisted workflow, which added the `safety_assured` annotation as part of the generated change. `safety_assured` is intended as an explicit engineer acknowledgement that the safety implications of a destructive migration have been considered and prepared for not as a routine flag added by tooling. We use Danger \([https://danger.systems/ruby/](https://danger.systems/ruby/)\) to call out risky database migrations with specific guidance for certain operations such as index changes — but there was no specific guidance for `rename_column`. The generic migration warning that _did_ exist had, after dozens of routine migrations, become background noise. The migration was bundled into the same PR as the application code that depended on the renamed column. This made the schema change easier to overlook in review, and meant the test suite for the PR only ever exercised the new code against the new schema, never the previously-deployed code against the new schema. This combination meant the unsafe sequence reached production with no automated test or human review specifically focused on whether the _currently-running_ application could tolerate the schema change. ### Changes we're making This incident was caused by a database migration that was not adequately prepared for safe deployment, and exposed gaps in both our review process and our use of code generation tooling. We're making the following improvements to prevent similar incidents: Database migrations that are unsafe to apply against the currently-running application code will be flagged and blocked before they can be merged. We are still working through the exact mechanism for this \(one option under active discussion is requiring schema migrations to ship in separate pull requests from any application code that depends on them\), but the intent is the same: the test suite on a migration's PR exercises the post-migration schema against the application code already running in production — the exact intermediate state that exists between the migration running and any new code shipping — so that backwards-incompatibility shows up as a failing test on the migration PR itself, where it can be caught by branch protection rather than by a human reading a routine review comment. We're extending Danger, the automated review tool that comments on every pull request that touches a database migration, to specifically highlight \`rename\_column\` amongst the other higher-risk operations it already highlights, so that they stand out from routine migration changes during code review. We're updating our internal engineering guidance \([AGENTS.md](http://agents.md/)\) to make clear that protective constructs such as safety\_assured \(from [https://github.com/ankane/strong\_migrations](https://github.com/ankane/strong_migrations)\) are intended as engineer-acknowledged sign-offs and should not be added by code-generation tooling on an engineer's behalf. Finally, we're reviewing how we surface and document the required pre-steps \(such as adding ignored\_columns\) for breaking schema changes, so that the safe rollout sequence is harder to skip, particularly for changes to unsharded core tables where there is no opportunity for a phased rollout. The changes described above protect against the _test-detectable_ class of schema/code mismatches. There remains a class of failure where a destructive migration to an unsharded core table affects every customer at once with no opportunity for a staged rollout. Our longer-term work on **cell-based deployments,** splitting customer data across multiple foundation databases, is what will give us the ability to roll out schema changes incrementally and detect issues against a small subset of customers before they reach the full fleet.

We're observing increased latency and error rates for a subset of our customers. We're currently investigating and will provide status updates as they become available.

We have identified and fixed the issue with the underlying database for a subset of customers. We are now monitoring the issue.

We have confirmed that latency and error rates have returned to normal for impacted customers.