Störungsverlauf von Websolr

Researchers recently discovered a sophisticated attempt to compromise XZ, a compression library that is widely used in Linux-based services across the world. It is suspected that if the compromise had been successful, state actors or other attackers would be able to remotely access many Linux-based machines on the Internet. Fortunately, the issue was discovered before the compromised version of the library made its way into mainline channels, so the impact is limited to versions 5.6.0 and 5.6.1. [CISA recommends](https://www.cisa.gov/news-events/alerts/2024/03/29/reported-supply-chain-compromise-affecting-xz-utils-data-compression-library-cve-2024-3094) users downgrade to XZ Utils 5.4.6 or earlier. Websolr's system maintenance policy is to use up to date stable and LTS versions of software, and this policy means that none of our systems are impacted by the compromise. Out of an abundance of caution, we audited every online server in our fleet and verified that none of them is running the compromised versions of XZ Utils. Websolr remains committed to the security and integrity of our systems and customers' data. Please direct any additional questions or concerns to support@websolr.com.

We are currently investigating this issue.

The issue has been identified and a fix is being implemented.

A fix has been implemented and we are monitoring the results.

This incident has been resolved. Heroku users will need to resend their provisioning requests. Heroku will also remove failed provisioning requests after 24 hours, so users will have to resend their provisioning requests.

We are currently investigating this issue.

The issue has been identified. Websolr.com is dependent on Heroku, and there is a incident for Heroku: https://status.heroku.com/

We are monitoring Heroku: https://status.heroku.com/

This incident has been resolved.

We are currently investigating this issue.

We have identified the root cause and are working on a fix.

A fix has been implemented and we are monitoring the results.

We continue to monitor the fix for any further issues. We will publish a postmortem once internal discussions are complete.

This incident has been resolved.

At approximately 06:00 UTC May 12th, users in the EU-West region began to receive HTTP 502 responses to all requests. This lasted until roughly 14:20 UTC.  In preparing for this postmortem, we identified two distinct problems: the disruption itself and the duration of time it took for a resolution. This postmortem will consider both. ‌ **Background** Websolr maintains a global fleet of servers, each of which is monitored by several different systems. These systems can identify problems and alert our operations team in the event of a problem. Part of the fleet is dedicated solely to acting as a proxy layer, capable of routing requests to specific Solr cores, enforce throttling, manage authentication, and more. The proxy layer is a distributed service, partitioned by geographic region. The decentralized nature of the proxy layer makes it resistant to node loss, as state is shared among all nodes in the layer. Thus, if a node fails for some reason, the load balancers will simply route traffic to the healthy nodes until the unhealthy node is fixed or replaced. This happens seamlessly so that users never even notice. ‌ **Proximate Cause of the Disruption** Nodes are periodically replaced automatically for a variety of reasons. Last week, one of the proxy layer’s nodes in EU-West-1 was replaced. The replacement came online and began to perform a bootstrapping process to install and configure all of the software needed to run an instance of the proxy service. However, the process failed due to a recent change in pip. This failure prevented the bootstrap process from installing _anything_, including the monitoring services. As a result, no alert was sent to the team. And it went unnoticed by users, as there was still a healthy node online serving requests. At 06:00 UTC May 12th, the healthy node was automatically replaced and failed to bootstrap as well. This meant that not only did the EU-West-1 region no longer have a functional routing layer, but that the monitoring services that otherwise would have indicated a problem were not present. ‌ **Proximate Cause of Delay in Resolution** Lacking alerts about the problem meant that it issue was not detected until our support team came online several hours later and noticed a massive number of support tickets. At that time, alarms were sounded and the problem was resolved within about 30 minutes. Up to that point there were a number of signals indicating that something was very wrong and should have warranted review. “Signals” in this context refer to such large departures from the norm that a reasonable person would know immediately that there is a problem. The most obvious signal was we began to receive an influx of support tickets. We received more tickets in the first few hours of the incident than we normally receive in 2 weeks, across all our products. We also received messages via social media regarding the disruption. Had anyone been monitoring these channels, it would have been obvious that there was a problem warranting investigation. Since our support team is US-based, the messages were arriving around 1:00AM our time, when everyone was asleep. Another signal was that our system for monitoring request metrics suddenly stopped receiving anything from the EU-West region. A sudden loss of traffic across an entire region is a clear signal that something is amiss. A third signal was that the load balancers experienced a massive increase in HTTP 5XX errors and almost 100% of requests were not successful. That itself should have raised alarms. ‌ **Root Cause of the Disruption & Delay** The aforementioned series of events are not the root cause of this particular incident. The root cause was the lack of sufficient alerting \(both direct and indirect\). Not only was a vital piece of infrastructure able to fail bootstrapping in a way that raised no alarm, but it remained undetected until the start of business hours because we lacked sufficient indirect monitoring. If the monitoring service wasn’t dependent on bootstrapping, or the bootstrapping process could page our team in the event of a problem, this incident would not have occurred. We would have been made aware of the problem last week and would have been able to fix it then, preventing it from recurring on May 12th. And if we’d had sufficient indirect monitoring \(identifying signals and thresholds that _imply_ a problem\), then the failed bootstrapping process wouldn’t have mattered as much, because we still would have been paged to examine the region anyway. ‌ **Resolution** When the technical problem was identified, we were able to fix the bootstrapping problem with a single line of code. The nodes were re-bootstrapped and the routing layer was repopulated with data. This fix was deployed in several other regions, with US-East-1 being the final region on \(scheduled for May 18th\). We have also upgraded the proxy service to be managed with our latest deploy tools, which are in heavy use with our hosted Elasticsearch product, Bonsai. Finally, we’re working on implementing some signal detection in our existing systems that would alert us about the high probability of a problem somewhere in our fleet. If you have any questions or concerns, please let us know at [support@websolr.com](mailto:support@websolr.com).

The issue has been identified and a fix is being implemented.

We are continuing to work on a fix for this issue.

A fix has been implemented and we are monitoring the results.

We have deployed an initial fix and have some data for affected indices currently backfilling.

We are still working on a resolution, but most indices metrics have been completely restored and are up to date. About 12% of indices are still backfilling historical metrics since the start of the incident. A small number of customers will have a small gap in their metrics data.

This incident has been resolved.

We are currently investigating the issue.

This incident has been resolved.

Earlier today, one of the nodes in our proxy layer was replaced by the Auto Scaling Group in US-West-1 region. The replacement provisioned successfully, and it did not throw any alerts. It successfully received state from other nodes in the proxy layer. However, due to an outdated CloudFormation template, the node was missing a key setting that was needed for connecting to the proper Solr nodes. This created a situation where the node reported healthy even while responding with 500-level errors to all requests. We were able to identify the issue and applied the missing setting to the new node, which resolved the issue. Our Operations team is currently investigating the root cause with our Platform team and will be taking appropriate measures to avoid a reoccurrence. If you have any questions or concerns, please let us know at [support@websolr.com](mailto:support@websolr.com).

We are investigating some issues affecting servers in a single availability zone. We are responding to issues as they arise, and are actively investigating the root cause. Automatic recovery is happening slowly, but steadily.

The issue was traced to a partial outage in AWS. The root cause was a malfunction within a single AZ, which caused a handful of instances to erroneously be replaced by the ASG. This manifested in Websolr as indices suddenly returning HTTP 500-level errors. Users without replication additionally experienced some data loss. Further questions should be directed to support@websolr.com.

We have identified an issue affecting performance for small percentage of users with indices in US-East-1. We are working to resolve.

We are continuing to work on a fix for this issue.

Impacted users should be seeing traffic and performance return to normal.

This incident has been resolved.

We have been alerted to an issue impacting one of our load balancers in the US-East-1 (Virginia) region. Our operations team has been notified and is currently working on a fix.

We have upgraded some infrastructure in our load balancer, which should offer a significant improvement in performance across the region. We will continue to monitor over the weekend.

We’re investigating reports of connectivity errors for some indices in the Virginia region.

The issue has been identified and a fix is being implemented.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

We've identified an issue in US-East which is causing a small amount of connection timeouts. Remediation is currently in progress.

The timeouts have been resolved.

We detected and resolved a networking issue which increased 503s for a subset of customers.

We have identified and fixed the problem.

Service has been restored.

We are observing an elevated error rate in US-East. The regression is limited in scope, affecting <0.1% of all requests. We have identified a root cause and are working on a fix.

This incident has been resolved.

We're detecting a network connectivity event affecting a single Availability Zone in our AWS Virginia region. General system redundancy is operating as designed, with no impact to customer traffic at this time. However we are standing by to intervene if necessary. https://status.aws.amazon.com/

From AWS: 4:58 PM PDT We can confirm that some instances are unreachable and some EBS volumes are experiencing degraded performance in a single Availability Zone in the US-EAST-1 Region. Engineers are engaged and we are working to resolve the issue. 5:05 PM PDT We have identified the root cause and are beginning to see recovery for instances and EBS volumes in the affected Availability Zone in the US-EAST-1 Region. We continue to work toward full resolution.

This incident has been resolved.

We are currently investigating this issue.

This issue has been resolved.

We are currently investigating this issue.

A fix has been implemented and we are monitoring the results.

Starting at 07:29 UTC, a server failure caused increased 503 errors for roughly 3% of indices in the US East region. The issue was detected and subsequently a fix was implemented at 08:04 UTC. Index traffic is now stable, and further follow-up maintenance is now being performed on those indices affected.

This incident has been resolved.

Our systems have detected an issue impacting about a dozen Cobalt and Staging indices in US East and we are investigating.

We have identified the issue and are working on a fix.

The incident has been resolved.

We have been automatically paged to respond to a server issue in the US-East region. Resolution will be forthcoming over the next several minutes.

Service has been restored to the impacted server.