Historia awarii Flagsmith

Environment documents were being rendered with numeric values cast as numbers in the JSON responses, instead of the previous behaviour where the values were returned as strings. This caused issues for statically typed languages in particular.

### What Happened On Monday 3rd March at around 17:00 UTC, we deployed a release to the Edge API that inadvertently changed the JSON types returned for integer-typed feature state values. Values that had previously been returned as JSON strings \(e.g., `"300"`\) began returning as JSON numbers \(e.g., `300`\). The change affected the `/api/v1/environment-document/` endpoint used by SDKs operating in local evaluation mode. Features stored as strings, booleans, or null values were not affected, and the Core API was not affected. Our existing test suite did not catch the type change before it reached production. The issue was reported at 18:12 UTC, classified as P0 at 18:27 UTC, and we completed a rollback to the previous stable release by 18:36 UTC, restoring the expected behaviour. ### What's Next? We are fixing the underlying serialisation issue at its source before the next production deployment. The next release will preserve the expected string-typed behaviour for numeric features — no customer action is required. To prevent this from happening again, we are adding integration tests that compare Core and Edge response types for the `/api/v1/environment-document/` endpoint, introducing a stricter review process for any changes that affect serialisation or data types, and requiring that test failures are investigated at their root cause before a fix is applied.

We are receiving reports of failing Edge API requests for a small subset of Flagsmith environments. We have identified the problem and the fix is underway.

A fix has been deployed to all regions. We have resolved the failures and are monitoring further impact.

The applied fix has not resolved the issue for all customers. New version being rolled out now with a fix for all customers.

The latest update has resolved the issue in all regions. We are continuing to monitor.

This incident has been resolved.

We are currently investigating an issue with the Flagsmith asynchronous task processor, impacting flag change events, webhooks and audit logs in the Flagsmith UI.

We have resolved the issue with the task processor and are no longer seeing issues with the Flagsmith UI. We are continuing to monitor performance of the task processor.

All issues with the asynchronous task processor have been resolved and all operations in the Flagsmith UI are working as expected.

We're seeing a delay in processing tasks by our asynchronous task processor, meaning that changes made in the dashboard may take longer to reflect in the API.

The issue was caused by a spike of unusual traffic. This traffic has been mitigated and the issue is resolved.

We are currently investigating alerts that app.flagsmith.com is not loading.

It appears that the issue is related to our third party hosting provider for the Flagsmith dashboard. We are following up with them.

It looks like the issue has been resolved on the provider's side. We are monitoring and validating.

We are continuing to monitor for any further issues.

The incident on the third party system was resolved at 15.11 UTC.

Our website was down momentarily this morning, but it did not affect our main application (app.flagsmith.com). Users were unable to view our marketing page due to an operational issue with management of our content management system. We'll share a post mortem in 24 hours.

We leverage an outside provider as a content management system for our public website. We had an outage for two hours due to an internal operational error with how we managed our account. We have since updated our processes with how we manage our account to prevent the issue from occurring moving forward

We are currently investigating the Edge API sporadically timing out some requests or refusing connections.

The issue seems to only be affecting clients connecting to the us-east-2 region. We're currently redirecting traffic away from the region.

We have migrated all traffic away from us-east-2 and are monitoring the impact.

We have had further reports of similar issues following the migration of traffic away from us-east-2, we are continuing to investigate.

We have passed on all of the troubleshooting information to our infrastructure provider and are awaiting further information from them on this issue. If you are affected by this issue, please get in touch with us at [email protected] with any information that you can share so we can identify this intermittent issue.

The issue has been isolated to a specific component in our hosting provider's network infrastructure. We are working with them, and escalating to get the issue resolved.

Our infrastructure provider has confirmed they were intermittently unavailable between December 14, 01:00 UTC and December 17, 22:50 UTC. We are working with affected customers to confirm that this issue is resolved.

We have received confirmation from multiple customers in the US east region now that the issue has been resolved.

## What Happened On Monday 16th December at around 14:25 GMT, we started receiving reports from customers that they were seeing elevated errors in their monitoring regarding connections to the Flagsmith API. Our investigation suggested that there were no application level issues in the Flagsmith platform. Since all customer reports were coming from Eastern US regions, we moved traffic away from the region, redirecting to our US west region. By this time, we were also able to carry out our own testing using infrastructure set up in US east. Our testing, and feedback from our customers showed that moving the traffic away did not resolve the issue. At 19:15 GMT, we opened a ticket with our infrastructure partner. Their investigations also confirmed that there were no issues with the Flagsmith platform itself. At 21:53 GMT, we created a ticket directly with our infrastructure provider \(AWS\) for them to investigate. At around 23:30 GMT on 16th December, we implemented a work around by creating a new DNS record to point directly to our infrastructure in US east, bypassing the latency based routing \(as provided by AWS Global Accelerator\). We shared this with customers experiencing the issues, and they confirmed that it resolved the issue. From here, we continued to investigate the issue with AWS support, providing them with additional information based on our testing, and reports from our customers. On 18th December at 17:58, we received the following information from AWS confirming that there had been an issue with Global Accelerator in the US East region. > The team confirmed that between December 13 5:00 PM PST and December 17 2:50 PM PST, AWS Global Accelerator experienced intermittent connection failures for client traffic served by the Ashburn, Virginia edge location. The issue has been resolved and the service is operating normally. Following this, we were able to confirm that the issue was no longer reproducible for us, or for our customers. ## What’s Next? We have requested, and are currently waiting for, a full post-mortem from the AWS team which may affect our next steps. In the meantime, we have begun looking at alternative solutions for the Global Accelerator that we may be able to keep as a cold standby in case of similar issues in the future.

Resolved at: 15:22 UTC A bug in the validation logic traits meant that identities with traits that have a value of `""` would cause an exception, resulting in the identity not being stored correctly when updates were made. This issue was resolved at 15:22 UTC, any new requests to retrieve an identity's flags, or store their traits will correctly store those traits.

We are currently investigating this issue.

The cause is due to a migration applied to the production database taking more resources than we anticipated. The migration has been completed and the load is normalising. We are monitoring and managing the load. Currently, while this issue is ongoing, it is not possible to create new environments.

API latency is back to normal. We are continuing to monitor.

Environment creation is functional again now.

This incident has been resolved.

Core API and admin dashboard outage on 10th July 2024.

At around 23:35 UTC, July 9th we received an alert that our Core API was not responding. This resulted in our SaaS customers not being able to use the Flagsmith dashboard \([app.flagsmith.com](http://app.flagsmith.com)\). Customers SDK’s serving flags were not impacted for those using the Edge API. Please note, any customers still using our Core API to serve flags were also impacted. This number is limited as we have advised customers to migration to the Edge API starting in June 2022. Our team resolved the issue at 3:06 UTC, July 10th and the Core API was fully responsive. The root cause of the issue was a database running at maximum CPU caused by requests to an end point that triggered an inefficient query. We also had our load balancer consistently recycling unhealthy API tasks that also strained the system due to unnecessary database connections. These two items combined, resulted in the core API being unresponsive. We recovered the database by dropping all traffic and terminating all open connections. This allowed the database to be recovered and process traffic normally. We are mitigating future issues like this by doing the following: * Optimizing the query that was triggered that used too much CPU capacity \(note that this has been completed and deployed to our production SaaS environment\) * Add better alerting when inefficient queries are identified in the application * Improving our internal tools \(e.g. PagerDuty\) to improve response time of issue identification triggered by some team members being out of office

Our Core API was overwhelmed by massive traffic spike, causing the core SQL database to become extremely slow. This led to ECS tasks failing the health checks, prompting the load balancer to start and stop new tasks, which in turn added more load to the already maxed-out database. We tried several approaches to rate limiting the source of the traffic, but eventually had to temporarily stop traffic at the load balancer for 2 minutes in order to stabilise the system. We are working on implementing AWS API Gateway to include rate limiting at the gateway level to avoid these sort of incidents in the future.

We have been alerted to an issue with our asynchronous task processor which handles replicating flag updates across our network. We are currently investigating.

We have rolled back a recent change which has cleared out the backlog of tasks. Updates to flags should be propagated as normal.

We are continuing to monitor for any further issues.

This incident has been resolved.

We are currently investigating this issue.

The issue has been identified and a fix is being implemented.

A fix has been implemented and we are monitoring the results.

Issues are still persisting for integrations using the Go client. We are investigating further.

We are continuing to investigate this issue.

We have identified the remaining issue and are implementing a fix. ETA for full resolution: 15 minutes.

We have rolled back in certain affected regions and have completed the work for the permanent fix. This is in the final stages of testing now and will be rolled out imminently.

We are continuing to work on a fix for this issue.

We are deploying the permanent fix now.

This incident has been resolved.

### Timeline At around 13.45 today, we deployed a change to resolve a validation issue that had been introduced in a release earlier today. This validation issue affected only requests which provided a numeric value for the identity identifier. The new validation which was added, however, caused an issue for certain integrations since it also added a requirement for the traits key to be provided \(and not omitted\) which is not the case in some of our clients \(the Go client for example omits the traits key if the list is empty\). This meant that valid requests from these clients for identities with no traits were being incorrectly rejected as invalid. Once we received alerts for this from our monitoring and some of our affected customers we began investigating. At 14:54 we deployed a change which resolved the validation issue for certain cases, however not all. As such, at 15:06 we made the decision to roll back the affected regions, and at 15:48 we deployed a permanent fix for this including additional test cases to cover this behaviour. ### Impact Since the requests that were affected by this issue were those that had no traits, the impact was fairly limited and no trait data has been lost. Some identities will not have been created during this period, however, due to the nature of the Flagsmith integration, subsequent calls to identify those users will create them. ### Next Steps We have been working hard already on improving our release process for the Edge API. The first step of this, which is due to be released next week, is to improve our automated releases to rollback based on a number of additional alerting factors, including more granular looks at our error rates. This will ensure that, in future, a small subset of errors like this will trigger an immediate automated rollback. The next step after this is to create a more comprehensive end to end testing suite which exercises each of our SDKs to verify that the integrations are all compatible with any new changes.

We are currently investigating reports of a service outage on all of our infrastructure.

The incident was related to an erroneous DNS change. This has now been reverted and service should be back up and running. There may be a period of time where failures are still seen as we wait for DNS caching to be propagated.

We are currently investigating this issue.

A fix has been implemented and we are monitoring the results.

We are continuing to monitor for any further issues.

This incident has been resolved.

## **Summary** On September 5th at 09:45 UTC, we initiated a release that included a database migration aimed at introducing a new constraint to the table containing information related to flags. According to our pre-live tests, this task should not have taken more than 50 milliseconds. Unfortunately, during the release to production, due to the high throughput on a particular table that it needed to acquire a temporary lock on, this caused a backlog of blocked connections waiting on the migration to complete. This caused a knock on effect that exhausted the connections on the database and a full restart was necessary. Once the restart was complete, the connections were restored and service was resumed. This happened at 10:20 UTC. ## **Next Steps** We have researched the cause of the issue and we do still have further research to understand certain aspects. Our current plan in the meantime is to implement certain safeguards as can be found in the following links to the Postgres documentation which should help reduce any impact in the future. [https://www.postgresql.org/docs/11/runtime-config-client.html](https://www.postgresql.org/docs/11/runtime-config-client.html) [https://www.postgresql.org/docs/11/runtime-config-logging.html](https://www.postgresql.org/docs/11/runtime-config-logging.html) \(`log_lock_waits`\)

We are currently investigating this issue.

We have identified a database migration that has failed as part of a new release. We are working to re-apply the migration.

A fix has been implemented and we are monitoring the results.

We are continuing to monitor for any further issues.

This incident has been resolved. A postmortem will follow.

**Summary** On September 5th at 09:45 UTC, we initiated a release that included a database migration aimed at introducing a new constraint to the table containing information related to flags. According to our pre-live tests, this task should not have taken more than 50 milliseconds. Unfortunately, during the release to production, due to the high throughput on a particular table that it needed to acquire a temporary lock on, this caused a backlog of blocked connections waiting on the migration to complete. This caused a knock on effect that exhausted the connections on the database and a full restart was necessary. Once the restart was complete, the connections were restored and service was resumed. This happened at 10:20 UTC. **Next Steps** We have researched the cause of the issue and we do still have further research to understand certain aspects. Our current plan in the meantime is to implement certain safeguards as can be found in the following links to the Postgres documentation which should help reduce any impact in the future. [https://www.postgresql.org/docs/11/runtime-config-client.html](https://www.postgresql.org/docs/11/runtime-config-client.html) [https://www.postgresql.org/docs/11/runtime-config-logging.html](https://www.postgresql.org/docs/11/runtime-config-logging.html) \(`log_lock_waits`\)

We were alerted to a slow DB query in newly released code at 13:09 BST. We are reverting the code and expect to be back to normal latency in the next 10 minutes. Edge API is not affected.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

At the moment, we are conducting an investigation, which indicates that any flag changes made in the last approximately two hours may not be visible to the client.

We are continuing to investigate this issue with the utmost priority.

We have identified a database lock that has caused this issue with the task processor. We are working on an interim fix as we identify the root cause.

We have deployed an update which has resumed consumption of the task queue. We are now processing the task queue and expect to be caught up in the next hour.

This incident has been resolved. We will publish a full post-mortem imminently.

## Timeline We were alerted at 23:39 UTC on 18/07/2023 that the queue for our asynchronous task processor was above the acceptable threshold. Once our team was online in India at 2:59am UTC, the status page was updated. By this time the task processor queue had backed up and the application was not able to write flag change events to the datastore which powers the Edge API. We investigated multiple avenues to determine the cause of the issues but there were multiple ‘symptoms’ that made determining the root cause very difficult. One specific issue, which turned out to be a red herring, related to the functionality to forward core API requests to the Edge API. This process seemed to be taking much longer than expected. Much of the investigation was spent restricting the usage of this functionality. At around 9:30am UTC, the cause was attributed to a particular set of tasks in the queue which were causing the processor units to run out of memory. Once it was determined to be safe to do so, these tasks were removed from the queue. At 10:19 UTC the issue had been resolved and the queue had returned to normal, meaning that flag change events were being written to the Edge API datastore again. Any changes that were not processed at the time were also re-run to ensure that the state was consistent with the expected changes that had been made in the database. ## Issue Details The issue was caused by an environment in the Flagsmith platform that included 400 segments and nearly 5000 segment overrides. This meant that the environment document which is generated to power the Edge API was larger than it was possible for the task processor instances to load into memory, and subsequently write to the Edge API datastore. To compound the issue, these changes were made via the Flagsmith API which resulted in 1000s of tasks being generated to update the document in the Edge API datastore in a short space of time. Each of these needed to load the offending environment, causing the task processor instances to fall into a cycle of running out of memory. These tasks were slowly being blocked from being picked up again by the processors but the quantity meant that there were always new versions of the same \(or very similar\) tasks to pick up. ## Next Steps * Implement limits on the size of the environment document * This will primarily consist of implementing limits on the number of segments and features in a given projects, as well as limiting the total number of segment overrides in a given project. * Deprecate the functionality to forward requests from the Core API to the Edge API. All projects using the Edge API will need to ensure that all connected SDKs are using the Edge API only.

We are investigating an issue where project admin users do not receive the inherited permissions on each environment in the project.

We are continuing to investigate this issue.

The issue has been identified and a fix is being implemented.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

We are currently investigating this issue.

We have identified an issue caused by a downstream service which is causing a knock on effect to our performance. We are currently deploying a change to remove the downstream service.

The downstream service has been successfully removed. Response times have returned to normal. We are continuing to monitor the situation.

This incident has been resolved.

## Timeline At 12:15pm UTC, we were notified of increased response times on a number of our Edge API endpoints. Investigation showed nothing immediately obvious but we suspected that it could be caused by Sentry, our APM tool. We set about removing the Sentry initialisation from our code and deployed it as soon as we could. At 12:48pm UTC, this change was deployed and we observed the response times decrease immediately. At 12:52pm UTC our monitoring confirmed that the average response time had returned to normal. ## Next Steps * Look into improvements to reduce / remove the impact of Sentry issues on our Edge API. * Decrease the shutdown timeout of the Sentry SDK. * Look at using [Sentry relay](https://docs.sentry.io/product/relay) to remove the impact on core Edge API services. ‌ * Add integration tests to simulate performance degradation / outages from all downstream services.