Historial de interrupciones de Flagsmith

1. 10 de septiembre de 2026 a las 14:43 UTCinvestigating

   Estamos viendo una gran cantidad de tareas que consumen nuestro procesador de tareas asincrónico, dando como resultado retrasos en el procesamiento de cambios de bandera.

2. 10 de septiembre de 2026 a las 14:44 UTCinvestigating

   Seguimos investigando esta cuestión.

3. 10 de septiembre de 2026 a las 15:09 UTCmonitoring

   Se ha aplicado una solución y estamos monitoreando los resultados.

4. 10 de septiembre de 2026 a las 16:10 UTCresolved

   Este incidente ha sido resuelto.

Traducido automáticamente desde la actualización oficial del incidente.

1. 4 de marzo de 2026 a las 16:26 UTCresolved

   Environment documents were being rendered with numeric values cast as numbers in the JSON responses, instead of the previous behaviour where the values were returned as strings. This caused issues for statically typed languages in particular.

2. 4 de marzo de 2026 a las 16:27 UTCpostmortem

   ### What Happened On Monday 3rd March at around 17:00 UTC, we deployed a release to the Edge API that inadvertently changed the JSON types returned for integer-typed feature state values. Values that had previously been returned as JSON strings \(e.g., `"300"`\) began returning as JSON numbers \(e.g., `300`\). The change affected the `/api/v1/environment-document/` endpoint used by SDKs operating in local evaluation mode. Features stored as strings, booleans, or null values were not affected, and the Core API was not affected. Our existing test suite did not catch the type change before it reached production. The issue was reported at 18:12 UTC, classified as P0 at 18:27 UTC, and we completed a rollback to the previous stable release by 18:36 UTC, restoring the expected behaviour. ### What's Next? We are fixing the underlying serialisation issue at its source before the next production deployment. The next release will preserve the expected string-typed behaviour for numeric features — no customer action is required. To prevent this from happening again, we are adding integration tests that compare Core and Edge response types for the `/api/v1/environment-document/` endpoint, introducing a stricter review process for any changes that affect serialisation or data types, and requiring that test failures are investigated at their root cause before a fix is applied.

1. 22 de enero de 2026 a las 16:56 UTCidentified

   We are receiving reports of failing Edge API requests for a small subset of Flagsmith environments. We have identified the problem and the fix is underway.

2. 22 de enero de 2026 a las 17:30 UTCmonitoring

   A fix has been deployed to all regions. We have resolved the failures and are monitoring further impact.

3. 22 de enero de 2026 a las 17:51 UTCidentified

   The applied fix has not resolved the issue for all customers. New version being rolled out now with a fix for all customers.

4. 22 de enero de 2026 a las 18:02 UTCmonitoring

   The latest update has resolved the issue in all regions. We are continuing to monitor.

5. 22 de enero de 2026 a las 18:40 UTCresolved

   This incident has been resolved.

1. 23 de octubre de 2025 a las 10:08 UTCinvestigating

   We are currently investigating an issue with the Flagsmith asynchronous task processor, impacting flag change events, webhooks and audit logs in the Flagsmith UI.

2. 23 de octubre de 2025 a las 10:26 UTCmonitoring

   We have resolved the issue with the task processor and are no longer seeing issues with the Flagsmith UI. We are continuing to monitor performance of the task processor.

3. 23 de octubre de 2025 a las 14:49 UTCresolved

   All issues with the asynchronous task processor have been resolved and all operations in the Flagsmith UI are working as expected.

1. 23 de julio de 2025 a las 21:01 UTCinvestigating

   We're seeing a delay in processing tasks by our asynchronous task processor, meaning that changes made in the dashboard may take longer to reflect in the API.

2. 23 de julio de 2025 a las 21:17 UTCresolved

   The issue was caused by a spike of unusual traffic. This traffic has been mitigated and the issue is resolved.

1. 18 de junio de 2025 a las 13:46 UTCinvestigating

   We are currently investigating alerts that app.flagsmith.com is not loading.

2. 18 de junio de 2025 a las 13:47 UTCinvestigating

   It appears that the issue is related to our third party hosting provider for the Flagsmith dashboard. We are following up with them.

3. 18 de junio de 2025 a las 13:52 UTCmonitoring

   It looks like the issue has been resolved on the provider's side. We are monitoring and validating.

4. 18 de junio de 2025 a las 13:52 UTCmonitoring

   We are continuing to monitor for any further issues.

5. 18 de junio de 2025 a las 19:43 UTCresolved

   The incident on the third party system was resolved at 15.11 UTC.

1. 9 de junio de 2025 a las 10:13 UTCresolved

   Our website was down momentarily this morning, but it did not affect our main application (app.flagsmith.com). Users were unable to view our marketing page due to an operational issue with management of our content management system. We'll share a post mortem in 24 hours.

2. 11 de junio de 2025 a las 8:06 UTCpostmortem

   We leverage an outside provider as a content management system for our public website. We had an outage for two hours due to an internal operational error with how we managed our account. We have since updated our processes with how we manage our account to prevent the issue from occurring moving forward

1. 16 de diciembre de 2024 a las 19:07 UTCinvestigating

   We are currently investigating the Edge API sporadically timing out some requests or refusing connections.

2. 16 de diciembre de 2024 a las 19:10 UTCinvestigating

   The issue seems to only be affecting clients connecting to the us-east-2 region. We're currently redirecting traffic away from the region.

3. 16 de diciembre de 2024 a las 20:17 UTCmonitoring

   We have migrated all traffic away from us-east-2 and are monitoring the impact.

4. 16 de diciembre de 2024 a las 21:59 UTCinvestigating

   We have had further reports of similar issues following the migration of traffic away from us-east-2, we are continuing to investigate.

5. 16 de diciembre de 2024 a las 23:43 UTCinvestigating

   We have passed on all of the troubleshooting information to our infrastructure provider and are awaiting further information from them on this issue. If you are affected by this issue, please get in touch with us at [email protected] with any information that you can share so we can identify this intermittent issue.

6. 17 de diciembre de 2024 a las 15:49 UTCidentified

   The issue has been isolated to a specific component in our hosting provider's network infrastructure. We are working with them, and escalating to get the issue resolved.

7. 18 de diciembre de 2024 a las 18:56 UTCmonitoring

   Our infrastructure provider has confirmed they were intermittently unavailable between December 14, 01:00 UTC and December 17, 22:50 UTC. We are working with affected customers to confirm that this issue is resolved.

8. 19 de diciembre de 2024 a las 14:04 UTCresolved

   We have received confirmation from multiple customers in the US east region now that the issue has been resolved.

9. 20 de diciembre de 2024 a las 17:20 UTCpostmortem

   ## What Happened On Monday 16th December at around 14:25 GMT, we started receiving reports from customers that they were seeing elevated errors in their monitoring regarding connections to the Flagsmith API. Our investigation suggested that there were no application level issues in the Flagsmith platform. Since all customer reports were coming from Eastern US regions, we moved traffic away from the region, redirecting to our US west region. By this time, we were also able to carry out our own testing using infrastructure set up in US east. Our testing, and feedback from our customers showed that moving the traffic away did not resolve the issue. At 19:15 GMT, we opened a ticket with our infrastructure partner. Their investigations also confirmed that there were no issues with the Flagsmith platform itself. At 21:53 GMT, we created a ticket directly with our infrastructure provider \(AWS\) for them to investigate. At around 23:30 GMT on 16th December, we implemented a work around by creating a new DNS record to point directly to our infrastructure in US east, bypassing the latency based routing \(as provided by AWS Global Accelerator\). We shared this with customers experiencing the issues, and they confirmed that it resolved the issue. From here, we continued to investigate the issue with AWS support, providing them with additional information based on our testing, and reports from our customers. On 18th December at 17:58, we received the following information from AWS confirming that there had been an issue with Global Accelerator in the US East region. > The team confirmed that between December 13 5:00 PM PST and December 17 2:50 PM PST, AWS Global Accelerator experienced intermittent connection failures for client traffic served by the Ashburn, Virginia edge location. The issue has been resolved and the service is operating normally. Following this, we were able to confirm that the issue was no longer reproducible for us, or for our customers. ## What’s Next? We have requested, and are currently waiting for, a full post-mortem from the AWS team which may affect our next steps. In the meantime, we have begun looking at alternative solutions for the Global Accelerator that we may be able to keep as a cold standby in case of similar issues in the future.

1. 4 de diciembre de 2024 a las 15:55 UTCresolved

   Resolved at: 15:22 UTC A bug in the validation logic traits meant that identities with traits that have a value of `""` would cause an exception, resulting in the identity not being stored correctly when updates were made. This issue was resolved at 15:22 UTC, any new requests to retrieve an identity's flags, or store their traits will correctly store those traits.

1. 5 de noviembre de 2024 a las 10:19 UTCinvestigating

   We are currently investigating this issue.

2. 5 de noviembre de 2024 a las 10:25 UTCidentified

   The cause is due to a migration applied to the production database taking more resources than we anticipated. The migration has been completed and the load is normalising. We are monitoring and managing the load. Currently, while this issue is ongoing, it is not possible to create new environments.

3. 5 de noviembre de 2024 a las 10:32 UTCmonitoring

   API latency is back to normal. We are continuing to monitor.

4. 5 de noviembre de 2024 a las 10:37 UTCmonitoring

   Environment creation is functional again now.

5. 5 de noviembre de 2024 a las 13:28 UTCresolved

   This incident has been resolved.

1. 10 de julio de 2024 a las 13:31 UTCresolved

   Core API and admin dashboard outage on 10th July 2024.

2. 10 de julio de 2024 a las 13:31 UTCpostmortem

   At around 23:35 UTC, July 9th we received an alert that our Core API was not responding. This resulted in our SaaS customers not being able to use the Flagsmith dashboard \([app.flagsmith.com](http://app.flagsmith.com)\). Customers SDK’s serving flags were not impacted for those using the Edge API. Please note, any customers still using our Core API to serve flags were also impacted. This number is limited as we have advised customers to migration to the Edge API starting in June 2022. Our team resolved the issue at 3:06 UTC, July 10th and the Core API was fully responsive. The root cause of the issue was a database running at maximum CPU caused by requests to an end point that triggered an inefficient query. We also had our load balancer consistently recycling unhealthy API tasks that also strained the system due to unnecessary database connections. These two items combined, resulted in the core API being unresponsive. We recovered the database by dropping all traffic and terminating all open connections. This allowed the database to be recovered and process traffic normally. We are mitigating future issues like this by doing the following: * Optimizing the query that was triggered that used too much CPU capacity \(note that this has been completed and deployed to our production SaaS environment\) * Add better alerting when inefficient queries are identified in the application * Improving our internal tools \(e.g. PagerDuty\) to improve response time of issue identification triggered by some team members being out of office

1. 4 de julio de 2024 a las 8:36 UTCresolved

   Our Core API was overwhelmed by massive traffic spike, causing the core SQL database to become extremely slow. This led to ECS tasks failing the health checks, prompting the load balancer to start and stop new tasks, which in turn added more load to the already maxed-out database. We tried several approaches to rate limiting the source of the traffic, but eventually had to temporarily stop traffic at the load balancer for 2 minutes in order to stabilise the system. We are working on implementing AWS API Gateway to include rate limiting at the gateway level to avoid these sort of incidents in the future.

1. 5 de marzo de 2024 a las 10:27 UTCinvestigating

   We have been alerted to an issue with our asynchronous task processor which handles replicating flag updates across our network. We are currently investigating.

2. 5 de marzo de 2024 a las 10:29 UTCmonitoring

   We have rolled back a recent change which has cleared out the backlog of tasks. Updates to flags should be propagated as normal.

3. 5 de marzo de 2024 a las 10:30 UTCmonitoring

   We are continuing to monitor for any further issues.

4. 5 de marzo de 2024 a las 13:38 UTCresolved

   This incident has been resolved.

1. 18 de enero de 2024 a las 14:40 UTCinvestigating

   We are currently investigating this issue.

2. 18 de enero de 2024 a las 14:44 UTCidentified

   The issue has been identified and a fix is being implemented.

3. 18 de enero de 2024 a las 14:52 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

4. 18 de enero de 2024 a las 15:06 UTCinvestigating

   Issues are still persisting for integrations using the Go client. We are investigating further.

5. 18 de enero de 2024 a las 15:06 UTCinvestigating

   We are continuing to investigate this issue.

6. 18 de enero de 2024 a las 15:15 UTCidentified

   We have identified the remaining issue and are implementing a fix. ETA for full resolution: 15 minutes.

7. 18 de enero de 2024 a las 15:34 UTCidentified

   We have rolled back in certain affected regions and have completed the work for the permanent fix. This is in the final stages of testing now and will be rolled out imminently.

8. 18 de enero de 2024 a las 15:36 UTCidentified

   We are continuing to work on a fix for this issue.

9. 18 de enero de 2024 a las 15:43 UTCidentified

   We are deploying the permanent fix now.

10. 18 de enero de 2024 a las 17:33 UTCresolved

    This incident has been resolved.

11. 18 de enero de 2024 a las 17:34 UTCpostmortem

    ### Timeline At around 13.45 today, we deployed a change to resolve a validation issue that had been introduced in a release earlier today. This validation issue affected only requests which provided a numeric value for the identity identifier. The new validation which was added, however, caused an issue for certain integrations since it also added a requirement for the traits key to be provided \(and not omitted\) which is not the case in some of our clients \(the Go client for example omits the traits key if the list is empty\). This meant that valid requests from these clients for identities with no traits were being incorrectly rejected as invalid. Once we received alerts for this from our monitoring and some of our affected customers we began investigating. At 14:54 we deployed a change which resolved the validation issue for certain cases, however not all. As such, at 15:06 we made the decision to roll back the affected regions, and at 15:48 we deployed a permanent fix for this including additional test cases to cover this behaviour. ### Impact Since the requests that were affected by this issue were those that had no traits, the impact was fairly limited and no trait data has been lost. Some identities will not have been created during this period, however, due to the nature of the Flagsmith integration, subsequent calls to identify those users will create them. ### Next Steps We have been working hard already on improving our release process for the Edge API. The first step of this, which is due to be released next week, is to improve our automated releases to rollback based on a number of additional alerting factors, including more granular looks at our error rates. This will ensure that, in future, a small subset of errors like this will trigger an immediate automated rollback. The next step after this is to create a more comprehensive end to end testing suite which exercises each of our SDKs to verify that the integrations are all compatible with any new changes.

1. 1 de diciembre de 2023 a las 15:10 UTCinvestigating

   We are currently investigating reports of a service outage on all of our infrastructure.

2. 1 de diciembre de 2023 a las 15:29 UTCresolved

   The incident was related to an erroneous DNS change. This has now been reverted and service should be back up and running. There may be a period of time where failures are still seen as we wait for DNS caching to be propagated.

1. 12 de septiembre de 2023 a las 11:14 UTCinvestigating

   We are currently investigating this issue.

2. 12 de septiembre de 2023 a las 11:26 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

3. 12 de septiembre de 2023 a las 11:35 UTCmonitoring

   We are continuing to monitor for any further issues.

4. 12 de septiembre de 2023 a las 12:15 UTCresolved

   This incident has been resolved.

5. 13 de septiembre de 2023 a las 8:43 UTCpostmortem

   ## **Summary** On September 5th at 09:45 UTC, we initiated a release that included a database migration aimed at introducing a new constraint to the table containing information related to flags. According to our pre-live tests, this task should not have taken more than 50 milliseconds. Unfortunately, during the release to production, due to the high throughput on a particular table that it needed to acquire a temporary lock on, this caused a backlog of blocked connections waiting on the migration to complete. This caused a knock on effect that exhausted the connections on the database and a full restart was necessary. Once the restart was complete, the connections were restored and service was resumed. This happened at 10:20 UTC. ## **Next Steps** We have researched the cause of the issue and we do still have further research to understand certain aspects. Our current plan in the meantime is to implement certain safeguards as can be found in the following links to the Postgres documentation which should help reduce any impact in the future. [https://www.postgresql.org/docs/11/runtime-config-client.html](https://www.postgresql.org/docs/11/runtime-config-client.html) [https://www.postgresql.org/docs/11/runtime-config-logging.html](https://www.postgresql.org/docs/11/runtime-config-logging.html) \(`log_lock_waits`\)

1. 5 de septiembre de 2023 a las 9:45 UTCinvestigating

   We are currently investigating this issue.

2. 5 de septiembre de 2023 a las 9:51 UTCidentified

   We have identified a database migration that has failed as part of a new release. We are working to re-apply the migration.

3. 5 de septiembre de 2023 a las 10:25 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

4. 5 de septiembre de 2023 a las 10:25 UTCmonitoring

   We are continuing to monitor for any further issues.

5. 5 de septiembre de 2023 a las 12:01 UTCresolved

   This incident has been resolved. A postmortem will follow.

6. 12 de septiembre de 2023 a las 16:50 UTCpostmortem

   **Summary** On September 5th at 09:45 UTC, we initiated a release that included a database migration aimed at introducing a new constraint to the table containing information related to flags. According to our pre-live tests, this task should not have taken more than 50 milliseconds. Unfortunately, during the release to production, due to the high throughput on a particular table that it needed to acquire a temporary lock on, this caused a backlog of blocked connections waiting on the migration to complete. This caused a knock on effect that exhausted the connections on the database and a full restart was necessary. Once the restart was complete, the connections were restored and service was resumed. This happened at 10:20 UTC. **Next Steps** We have researched the cause of the issue and we do still have further research to understand certain aspects. Our current plan in the meantime is to implement certain safeguards as can be found in the following links to the Postgres documentation which should help reduce any impact in the future. [https://www.postgresql.org/docs/11/runtime-config-client.html](https://www.postgresql.org/docs/11/runtime-config-client.html) [https://www.postgresql.org/docs/11/runtime-config-logging.html](https://www.postgresql.org/docs/11/runtime-config-logging.html) \(`log_lock_waits`\)

1. 10 de agosto de 2023 a las 12:15 UTCidentified

   We were alerted to a slow DB query in newly released code at 13:09 BST. We are reverting the code and expect to be back to normal latency in the next 10 minutes. Edge API is not affected.

2. 10 de agosto de 2023 a las 12:26 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

3. 10 de agosto de 2023 a las 12:54 UTCresolved

   This incident has been resolved.

1. 19 de julio de 2023 a las 2:58 UTCinvestigating

   At the moment, we are conducting an investigation, which indicates that any flag changes made in the last approximately two hours may not be visible to the client.

2. 19 de julio de 2023 a las 6:35 UTCinvestigating

   We are continuing to investigate this issue with the utmost priority.

3. 19 de julio de 2023 a las 8:02 UTCidentified

   We have identified a database lock that has caused this issue with the task processor. We are working on an interim fix as we identify the root cause.

4. 19 de julio de 2023 a las 9:12 UTCmonitoring

   We have deployed an update which has resumed consumption of the task queue. We are now processing the task queue and expect to be caught up in the next hour.

5. 19 de julio de 2023 a las 10:19 UTCresolved

   This incident has been resolved. We will publish a full post-mortem imminently.

6. 19 de julio de 2023 a las 16:24 UTCpostmortem

   ## Timeline We were alerted at 23:39 UTC on 18/07/2023 that the queue for our asynchronous task processor was above the acceptable threshold. Once our team was online in India at 2:59am UTC, the status page was updated. By this time the task processor queue had backed up and the application was not able to write flag change events to the datastore which powers the Edge API. We investigated multiple avenues to determine the cause of the issues but there were multiple ‘symptoms’ that made determining the root cause very difficult. One specific issue, which turned out to be a red herring, related to the functionality to forward core API requests to the Edge API. This process seemed to be taking much longer than expected. Much of the investigation was spent restricting the usage of this functionality. At around 9:30am UTC, the cause was attributed to a particular set of tasks in the queue which were causing the processor units to run out of memory. Once it was determined to be safe to do so, these tasks were removed from the queue. At 10:19 UTC the issue had been resolved and the queue had returned to normal, meaning that flag change events were being written to the Edge API datastore again. Any changes that were not processed at the time were also re-run to ensure that the state was consistent with the expected changes that had been made in the database. ## Issue Details The issue was caused by an environment in the Flagsmith platform that included 400 segments and nearly 5000 segment overrides. This meant that the environment document which is generated to power the Edge API was larger than it was possible for the task processor instances to load into memory, and subsequently write to the Edge API datastore. To compound the issue, these changes were made via the Flagsmith API which resulted in 1000s of tasks being generated to update the document in the Edge API datastore in a short space of time. Each of these needed to load the offending environment, causing the task processor instances to fall into a cycle of running out of memory. These tasks were slowly being blocked from being picked up again by the processors but the quantity meant that there were always new versions of the same \(or very similar\) tasks to pick up. ## Next Steps * Implement limits on the size of the environment document * This will primarily consist of implementing limits on the number of segments and features in a given projects, as well as limiting the total number of segment overrides in a given project. * Deprecate the functionality to forward requests from the Core API to the Edge API. All projects using the Edge API will need to ensure that all connected SDKs are using the Edge API only.

1. 12 de julio de 2023 a las 13:46 UTCinvestigating

   We are investigating an issue where project admin users do not receive the inherited permissions on each environment in the project.

2. 12 de julio de 2023 a las 13:47 UTCinvestigating

   We are continuing to investigate this issue.

3. 12 de julio de 2023 a las 14:25 UTCidentified

   The issue has been identified and a fix is being implemented.

4. 12 de julio de 2023 a las 15:56 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

5. 12 de julio de 2023 a las 16:17 UTCresolved

   This incident has been resolved.