Launchdarkly outage history

Server-side streaming requests experienced elevated error rates between approximately 12:00 p.m. and 12:12 p.m. PDT, and some SDKs establishing new streaming connections may have been unable to connect. Polling request error rates remained below 1% throughout the incident. No customer data was lost or compromised. We apologize for the disruption.

This incident has been resolved. Between 12:00 and 12:12 PM PDT on August 26, server-side SDKs establishing new streaming connections experienced elevated error rates. Existing streaming connections were not affected, and polling requests continued to be served with error rates below 1%. Some requests to the LaunchDarkly web application also experienced elevated errors between approximately 11:29 AM and 12:03 PM PDT. The issue was caused by an internal configuration change, which has been reverted. All systems have recovered and are operating normally. No customer data was lost or compromised.

Emails sent by LaunchDarkly — including sign-up verification, password reset, teammate invitation, and workflow notification emails — are currently failing to deliver due to an authentication issue with an upstream email delivery provider. We have identified the cause and are working to restore delivery. Feature flag delivery, SDKs, and the LaunchDarkly application itself are not affected. If you are trying to complete sign-up or reset a password, those emails will not arrive until this is resolved. Once delivery is restored, we will re-send affected verification and password reset emails. Next update within 2 hours.

Email delivery has been restored as of 2:32 PM UTC. Sign-up verification, password reset, teammate invitation, and notification emails are sending normally again. We are re-sending sign-up verification and password reset emails that failed to deliver during the incident window (August 22, 11:48 PM CDT through today). If you attempted to sign up or reset a password during that period and haven't received an email within the next few hours, please retry the action or contact support. We are continuing to monitor email delivery closely and will post a final update once we've confirmed sustained recovery.

We're experiencing a service degradation which can result in Flag updates via the Polling API to be delayed.

We've identified the root cause, our CDN provider had an issue on their end which is fixed now. Monitoring for a while to make sure things stabilise.

This incident has been resolved.

Flag Updates are currently delayed in the us-east-1 region. We are investigating the issue.

We investigated elevated error rates reported for mobile SDK flag delivery in US East. We've confirmed that flag delivery itself was not impacted. No customer action is required and we are monitoring the situation.

The issue is being marked as resolved. We're seeing improved latencies and our systems have recovered.

We are investigating elevated HTTP 500 errors affecting some server-side SDK streaming connections in our US commercial environment. Affected applications may fail to establish or may intermittently lose their streaming connection to LaunchDarkly.

We released several mitigations and we're noticing that error rates are dropping. We'll continue to monitor the situation.

We are continuing to investigate this issue.

SDKs restored to operational

All customers are experiencing missing Observability Traces starting July 30th at 5:16pm PT. We are investigating and will provide updates as they become available. Trace data is not lost and we are working to recover it.

Customers are experiencing missing Observability Traces ingested from July 30th 5:16pm until July 31st 10:59am PT. Observability Logs / Metrics are unaffected. We have identified the issue and are continuing our work to recover the missing Traces data.

The issue with Observability Traces has been identified and a fix has been implemented. We have recovered most trace data that was missing from July 30th 5:16pm to July 31st 10:59am PT. We will resolve the incident once we fully complete data recovery (expected to complete by 5pm PT).

The issue with Observability Traces has been resolved. Missing trace data has been recovered.

Customers are experiencing delayed session replay and error ingest as of 11:03am PT. We have identified the issue and are continuing our work to resolve it. Data loss is not expected.

The issue with session replay and error ingest has been identified and a fix has been implemented. We expect to catch up on all backlogged data ingest in ~15 minutes. No data has been lost.

The issue with session replay and error ingest has been resolved. We've caught up on all backlogged data as of 1:01pm PT.

Between 11:32 and 11:48 PT, customers experienced increased latency and elevated error rates on API endpoints. Latency and error rates have returned to normal, and the incident has been resolved.

The LaunchDarkly web application (app.launchdarkly.com) was unavailable and is actively recovering at the moment. The Flag Delivery network was also impacted, causing an elevated failure rate with retrieving flag values. Customers using client-side or server-side SDKs should continue to see the last known flag values if a local cache exists, or fall back to in-code values. Customers using our Relay Proxy should continue to see last known flag values if a local cache exists. Customers using our Edge SDKs should continue to see last known flag values. Some event data loss may have occurred.

We have identified the root cause and are implementing a fix.

The web application availability has recovered. Flag delivery and event ingestion continue to be impacted and are actively recovering. Our engineering team continues to work on remediation.

Production service impact has been restored. However, customers might need to restart their applications to fully reconnect to LaunchDarkly's services. Our engineering team is working on guidance that can be provided to customers.

All service availability has been fully restored. Customers using server-side SDKs might need to restart their applications to re-enable the SDKs' connection to LaunchDarkly. Customers using the Relay Proxy may also need to restart their Relay Proxy; in this case, the Relay Proxy should be restarted before restarting their applications that connect to it. To determine if an application needs to be restarted, customers can look at their application logs. Affected SDKs will emit (depending on language) logs containing one of "giving up permanently", "Invalid SDK key", "unauthorized", or "not authorized", or "401". Services which emitted logs such as these before 9:13AM PT may be unable to receive flag updates and will be unable to send events to LaunchDarkly until they are restarted.

We are currently investigating this issue

We are continuing to investigate this issue.

The issue has been identified and a fix has been implemented

We are continuing to work on a fix for this issue.

LaunchDarkly UI has recovered. We will continue to monitor Flag Delivery and event ingest here: https://status.launchdarkly.com/incidents/chywwz01ptb0.

We are investigating continued impact to customers' mobile and server SDK caused by the previous incident (https://status.launchdarkly.com/incidents/chywwz01ptb0). If customers are receiving SDK or Relay Proxy 401s please re-establish connections.

Our Flag Delivery services have recovered and we're observing an increase in customer traffic successfully reestablish connections to our streaming services. Customers who continue to encounter 401 connection errors in their SDKs or Relay Proxy instances are advised to restart their applications and/or Relay Proxy instances.

Customers continue to successfully reestablish connections to our streaming services. We recommend reviewing your application logs for the incident window. Affected SDKs emit messages—varying by language—that contain any of the following: "giving up permanently," "Invalid SDK key," "unauthorized," "not authorized," or "401." Any service that logged these messages may be unable to receive flag updates or send event data until it is restarted. Here are the recommended actions for each affected application: - If you use server-side SDKs, please restart the application to re-establish its connection to LaunchDarkly. - If re-establishing connection to the server-side SDK does not work, please create and use a second SDK key. - If you use the Relay Proxy, please restart the Relay Proxy to re-establish connections. It should not be necessary to restart the applications behind it. - Client-side SDKs will reconnect automatically as they restart. - If this does not resolve the issue for an affected application (e.g. you still see the log messages from above), please log a support ticket and we will follow up with you directly to assist with remediation.

We are continuing to monitor for any further issues, see our previous update for recovery steps.

All LaunchDarkly services are operational, though some customers need to take action to restore their connections to our Flag Delivery Network. Customers continue to successfully reestablish connections by completing the action guidance below and we continue to support customers working through the residual impact. SDKs and Relay Proxy instances which attempted to establish a connection to LaunchDarkly on July 10 between 09:11 and 11:42 AM PT were intermittently affected. This includes situations where the host application was restarted or redeployed during this time. SDKs and Relay Proxy instances which did not attempt to establish a connection during this time were unaffected. We recommend reviewing your application logs for the incident window. Affected SDKs emit messages—varying by language—that contain any of the following: "giving up permanently," "Invalid SDK key," "unauthorized," "not authorized," or "401." Any service that logged these messages may be unable to receive flag updates or send event data until it is restarted. Here are the recommended actions for each affected application: - If you use the Relay Proxy, please restart the Relay Proxy to re-establish connections. It should not be necessary to restart the applications behind it, but we recommend reviewing your application logs to confirm if restarting your applications is necessary. - If you use server-side SDKs without the Relay Proxy, please restart the application to re-establish its connection to LaunchDarkly. - Client-side and Mobile SDKs will reconnect automatically as their applications attempt to reconnect to LaunchDarkly under normal circumstances, for example when the application loads or the device regains network connectivity. - If this does not resolve the issue for an affected application (e.g. you still see the log messages from above), please log a support ticket and we will follow up with you directly to assist with remediation.

This incident has been resolved.

We are currently investigating an issue where experiments using frequentist analysis cannot be saved or started.

We have identified the issue and are working towards releasing a fix.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

Event ingestion is operating normally. Experimentation results, Guarded Rollouts, and in-product data charts were delayed ~5 min between 7:13–7:53 AM PT. No data was lost. We apologize for any inconvenience.

At 13:21 Pacific Time, we experienced a high volume of error alerts affecting users' ability to interact with the LaunchDarkly application including logging in and UI / API responsiveness. We quickly identified the root cause relating to flag delivery. We are expecting errors decreasing now.

We are continuing to investigate this issue.

We have rolled out a fix and error rates are returning to normal

All error rates have normalized.

Some customers are not receiving account verification emails. Our team is actively working to mitigate this issue.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

We are investigating elevated error rates on our event data ingestion endpoints affecting production customers, between approximately 7:00 AM PDT and 7:15 AM PDT. During this window, a portion of feature flag, experimentation, and AI/agent event data may have been rejected or delayed. Flag delivery and evaluation are not affected. SDKs continue to serve flags normally. Observability ingest is not affected.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

LaunchDarkly web application and flag delivery unaffected, but documentation and LaunchDarkly.com are unavailable for some users. A fix is on the way.

A fix has been implemented and monitoring the results.

Incident seems to be resolved for the majority of users, however, we are continuing to see some issues for in the San Jose and Los Angeles region. We're working to remediate for these regions.

We've identified the source of the problem for areas that were still seeing issues with the LaunchDarkly.com and docs and have a solution in place. We are now monitoring to ensure everything is resolved.

We're now seeing restored access in the affected regions.

Some customers are experiencing delayed Observability data ingest for OpenTelemetry data. We have identified the issue and are continuing our work to resolve it. Observability signals (errors / logs / traces / OTel metrics) are currently delayed by ~6 minutes on average. Data ingest delays began at 10:02am and are improving as of 10:16am PT.

The issue with Observability data ingest for OpenTelemetry data has been identified and a fix has been implemented. We are continuing to monitor the performance of impacted services. We will continue to update this page until it is resolved. Observability signals (errors / logs / traces / OTel metrics) are currently delayed by <1 minutes on average.

The issue with Observability data ingest for OpenTelemetry data has been resolved. Impacted services have returned to normal operation.

We identified a brief period of elevated errors affecting some LaunchDarkly application requests between 7:30-7:37pm PT. The issue has recovered, and we are continuing to monitor.

Monitoring has confirmed no further impact outside of the initially reported 7:30-7:37pm PT timeframe

We are currently experiencing delays in event ingestion impacting multiple product areas starting at 12:40pm PT. We have identified the issue and are working on a fix. No data has been lost.

A fix has been implemented and we are monitoring the results.

We are monitoring the status of our services.

We have made partial progress. Some features may show data gaps while we complete recovery. We are continuing to monitor.

We are continuing to monitor for any further issues.

We have made partial progress. We have identified a brief period of event data loss, and some event-based features may show incomplete data while we continue recovery. We are continuing to monitor.

We have made partial progress. We identified that a small percentage of event data was lost during two windows: June 9, 11:41 - 11:59 UTC and June 10, 23:55 UTC - June 11, 00:21 UTC. Some event-based features may show incomplete data while backfills and recovery continue. We are continuing to monitor and will provide another update when recovery is complete.

We have made major progress. We identified that a small percentage of event data was lost during two windows: June 9, 11:41 - 11:59 UTC and June 10, 23:55 UTC - June 11, 00:21 UTC. Some event-based features may show incomplete data while backfills and recovery continue. We are continuing to monitor and will provide another update when recovery is complete.

This incident has been resolved.