Historial de interrupciones de Lokalise

1. 14 de septiembre de 2026 a las 12:22 UTCinvestigating

   Actualmente estamos investigando esta cuestión.

2. 14 de septiembre de 2026 a las 13:28 UTCidentified

   Se ha determinado la cuestión y se está aplicando una solución.

3. 14 de septiembre de 2026 a las 15:14 UTCmonitoring

   Se ha aplicado una solución y estamos monitoreando los resultados.

4. 14 de septiembre de 2026 a las 15:14 UTCmonitoring

   Continuamos monitoreando cualquier otro problema.

5. 14 de septiembre de 2026 a las 15:22 UTCresolved

   Este incidente ha sido resuelto.

Traducido automáticamente desde la actualización oficial del incidente.

1. 17 de agosto de 2026 a las 3:06 UTCinvestigating

   Actualmente estamos investigando esta cuestión.

2. 17 de agosto de 2026 a las 3:48 UTCresolved

   Este incidente ha sido resuelto.

Traducido automáticamente desde la actualización oficial del incidente.

1. 4 de febrero de 2026 a las 16:23 UTCinvestigating

   We are currently investigating this issue.

2. 4 de febrero de 2026 a las 16:33 UTCidentified

   The issue has been identified and a fix is being implemented. Application and API are now operational with degraded performance.

3. 4 de febrero de 2026 a las 16:57 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

4. 4 de febrero de 2026 a las 17:07 UTCresolved

   This incident has been resolved.

5. 13 de febrero de 2026 a las 15:49 UTCpostmortem

   On February 4, 2026, Lokalise experienced a brief service interruption where both the application and API were unavailable for approximately 10 minutes. The incident began at 16:18 UTC and was fully resolved within 12 minutes as our engineering team restored service functionality. **What happened?** **The Cause:** A configuration update intended to enable new application metrics unexpectedly triggered a series of intensive, unoptimized database scans. This resulted in an immediate exhaustion of our backend processing capacity, preventing the web server from fulfilling incoming requests. **The Fix:** Our team identified the faulty configuration release and promptly initiated a rollback to the last known stable state. Once service was restored, a permanent fix was deployed to remove the problematic configuration entirely. **Timeline \(UTC\):** * **16:18:** Automated alerts reported service unavailability; investigation initiated. * **16:20:** The root cause was identified as a recent configuration deployment. * **16:27:** Rollback to a stable version was initiated. * **16:30:** Application and API functionality were fully restored. * **17:07:** System verified as stable after a final hotfix deployment. **What we are doing to prevent this in the future** * **Enhancing pre-deployment validation:** We are updating our automated review agents to flag potential performance issues in legacy code or dormant features before they are activated via configuration changes. * **Optimizing metric design:** We are transitioning to a system that pre-computes application metrics in isolated processes to ensure that monitoring activities never compete with core application resources. * **Refactoring database interactions:** Our team is reviewing and refactoring resource-intensive database queries to improve overall platform stability and response times. * **Consolidating internal alerting:** We are refining our monitoring workflows to consolidate overlapping alerts, allowing our engineers to focus even more quickly on resolution during critical events. We sincerely apologize for the disruption this incident caused to your work and your automated workflows. We recognize the trust you place in Lokalise for your daily operations, and we are committed to enhancing our processes to prevent similar occurrences. If you have any questions or require further assistance regarding this incident, please reach out to us at [**[email protected]**](mailto:[email protected]).

1. 26 de enero de 2026 a las 15:56 UTCinvestigating

   We are currently investigating this issue.

2. 26 de enero de 2026 a las 16:06 UTCinvestigating

   We are continuing to investigate this issue.

3. 26 de enero de 2026 a las 16:32 UTCresolved

   This incident has been resolved.

4. 2 de febrero de 2026 a las 16:59 UTCpostmortem

   On January 26, 2026, Lokalise experienced a service outage and performance degradation between 15:52 UTC and 16:32 UTC. During this time, the Lokalise application was unavailable, and users of the API experienced intermittent connectivity and high latency. **What happened?** **The Cause:** The incident was triggered during a migration of our monitoring systems. A configuration mismatch caused a high volume of internal network requests to fail, which subsequently overwhelmed our internal DNS services. This prevented various parts of our infrastructure from communicating with one another, including our primary databases and application services. **The Fix:** Our engineering team identified the source of the traffic and disabled the legacy monitoring configuration. We also rotated affected infrastructure nodes and adjusted our service scaling to alleviate pressure on our databases. These actions restored normal communication between our services, bringing the platform back to full operational status. **Timeline \(UTC\):** * **15:52:** Service degradation and unavailability detected and investigation initiated. * **16:24:** Root cause identified as internal network congestion impacting service connectivity. * **16:27:** Corrective actions implemented; service begins to stabilize. * **16:32:** Full service restored and performance monitored for stability. **What we are doing to prevent this in the future** * **Enhancing scaling capabilities:** We are upgrading our internal DNS services to scale horizontally, ensuring they can handle unexpected spikes in traffic without impacting the wider platform. * **Improving resource monitoring:** We are implementing additional alerting for resource exhaustion to identify and mitigate infrastructure bottlenecks before they impact service availability. * **Refining deployment procedures:** We are updating our internal documentation and validation steps to ensure infrastructure dependencies are strictly coordinated during system migrations. We sincerely apologize for the disruption this incident caused to your workflow. We understand how critical Lokalise is to your operations and are committed to improving our system's resilience to prevent a recurrence. If you have any questions or require further assistance, please contact us at [[email protected]](mailto:[email protected]).

1. 27 de noviembre de 2025 a las 8:30 UTCinvestigating

   We are currently investigating this issue.

2. 27 de noviembre de 2025 a las 8:37 UTCidentified

   The issue has been identified and a fix is being implemented.

3. 27 de noviembre de 2025 a las 8:41 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

4. 27 de noviembre de 2025 a las 8:43 UTCresolved

   This incident has been resolved.

1. 23 de octubre de 2025 a las 9:11 UTCinvestigating

   We are currently investigating this issue.

2. 23 de octubre de 2025 a las 9:45 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

3. 23 de octubre de 2025 a las 10:00 UTCresolved

   This incident has been resolved.

1. 7 de octubre de 2025 a las 13:13 UTCinvestigating

   We are currently investigating this issue.

2. 7 de octubre de 2025 a las 13:22 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

3. 7 de octubre de 2025 a las 13:29 UTCresolved

   This incident has been resolved.

1. 17 de septiembre de 2025 a las 15:20 UTCinvestigating

   We are currently investigating this issue.

2. 17 de septiembre de 2025 a las 15:43 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

3. 17 de septiembre de 2025 a las 16:14 UTCresolved

   This incident has been resolved.

1. 8 de septiembre de 2025 a las 10:08 UTCinvestigating

   We are currently investigating this issue.

2. 8 de septiembre de 2025 a las 10:10 UTCinvestigating

   We are continuing to investigate this issue.

3. 8 de septiembre de 2025 a las 10:12 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

4. 8 de septiembre de 2025 a las 10:18 UTCresolved

   This incident has been resolved.

5. 26 de septiembre de 2025 a las 11:16 UTCpostmortem

   **What happened** On Sep 8, 2025 at 09:59 UTC, a change introduced in our system unexpectedly missed a Feature Toggle condition, which caused a spike in the number of requests to one of our internal services. This led to degraded performance and temporary unavailability of both the Lokalise App and API. Our monitoring and alerting systems immediately triggered the alerts, allowing us to quickly identify and restore the service. **Impact** The Lokalise App was unavailable for approximately **9 minutes**, and the API experienced downtime for about **6 minutes** within that window. No customer data was lost or corrupted. **What we are doing to prevent this in the future** We are implementing stricter safeguards around feature rollout validation, improving how our services handle spikes in traffic, and adding additional error handling to reduce the risk of cascading failures. We sincerely apologize for the disruption this caused. Thank you for your patience and continued trust in Lokalise. If you have any questions, feel free to contact us at [[email protected]](mailto:[email protected]).

1. 30 de julio de 2025 a las 12:02 UTCinvestigating

   We are currently investigating this issue.

2. 30 de julio de 2025 a las 12:13 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

3. 30 de julio de 2025 a las 12:30 UTCresolved

   This incident has been resolved.

4. 13 de agosto de 2025 a las 15:29 UTCpostmortem

   **Summary** On July 30, 2025, Lokalise became unavailable and unstable at 11:55 UTC due to a new feature release. The new feature caused an unpredictable usage pattern with excessive memory usage and resulted in an ingress crash-looping. This was rectified and by 12:10 UTC, and Lokalise became fully operational again. **What happened?** During the update to one of our services, a configuration mismatch was introduced. The change, intended to increase the maximum allowed size, was not fully propagated across our infrastructure. This discrepancy between the service and the infrastructure layer led to excessive memory consumption, ultimately causing the service to become unstable and fully unavailable. Our team identified the problem quickly, the expected configuration was correctly propagated and by 12:10 UTC the problem was resolved. **Impact** 11:55 UTC: Lokalise's performance starts to degrade. 11:58 UTC: Lokalise is not available and an investigation is started. 12:05 UTC: Problem is identified and solution is applied. 12:10 UTC: Lokalise's performance is restored. 12:10 - 12:30 UTC: Performance is monitored and eventually the problem incident is resolved. ‌ We sincerely apologize for the disruption this caused. Thank you for your patience and continued trust in Lokalise. If you have any questions, feel free to contact us at [[email protected]](mailto:[email protected]).

1. 18 de julio de 2025 a las 12:39 UTCinvestigating

   We are currently investigating this issue.

2. 18 de julio de 2025 a las 13:24 UTCidentified

   We have identified the root cause. We are actively working to resolve it.

3. 18 de julio de 2025 a las 13:29 UTCmonitoring

   Problem resolved. We are monitoring the system.

4. 18 de julio de 2025 a las 13:33 UTCresolved

   Issue is resolved.

5. 31 de julio de 2025 a las 7:34 UTCpostmortem

   On July 18, 2025, Lokalise became unavailable at 12:31 UTC due to a maintenance error that disabled an encryption key needed for data access. We fully restored service by 13:32 UTC with no loss of customer data. **What happened?** During a periodic scheduled clean up procedure, an encryption key presumed to be unused was disabled. This key, in fact, was still in use by one of the auxiliary databases. With the key disabled, the application was unable to access data and this led to a disruption of service. Our team identified the problem quickly and worked closely with our cloud provider’s support team to resolve the issue. The encryption key was re-enabled and a point-in-time recovery was performed to restore the service. The main application was back online by 13:25 UTC, and all services were fully operational by 13:32 UTC. The root cause of the incident was that the encryption key was mistakingly referenced in two terraform state files. One of the state files was no longer associated with any infrastructure, the encryption key therefore was deemed unused, and scheduled for removal. **Impact** * 12:31 – 13:25 UTC: Lokalise application was unavailable. * 13:25 – 13:32 UTC: Some background services were still recovering. **What we are doing to prevent this in the future** We take reliability very seriously and have performed a thorough analysis of this incident. We are going to make a number of improvements to prevent this type of incident from happening in the future. These include: * improve procedure and automation around disabling and decommissioning of encryption keys to cross-check all data storages to ensure the key is not used anymore * improve monitoring to detect the databases going into grace period when an encryption key is disabled, but the database is still functional; this will allow us to re-enable the key without the need in point-in-time recovery and service interruption * review terraform state files for any keys and other resources referenced in multiple state files and address findings if any We sincerely apologize for the disruption this caused. Thank you for your patience and continued trust in Lokalise. If you have any questions, feel free to contact us at [[email protected]](mailto:[email protected]).

1. 23 de mayo de 2025 a las 5:57 UTCinvestigating

   We are currently investigating this issue.

2. 23 de mayo de 2025 a las 7:17 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

3. 23 de mayo de 2025 a las 7:38 UTCresolved

   This incident has been resolved.

1. 8 de mayo de 2025 a las 7:13 UTCinvestigating

   We are currently investigating this issue.

2. 8 de mayo de 2025 a las 8:23 UTCidentified

   We have identified the issue, and new async exports are now processing fine. We are implementing a fix to allow stuck processes to finish.

3. 8 de mayo de 2025 a las 13:27 UTCresolved

   This incident has been resolved.

1. 7 de mayo de 2025 a las 13:18 UTCresolved

   Async export processes that have been triggered between 07:15 and 08:15 UTC time today got stuck and didn’t complete. We will share more details in the postmortem message.

1. 17 de marzo de 2025 a las 13:33 UTCinvestigating

   Lokalise application is experiencing issues with processing search, filtering and statistics in the UI. We are currently investigating this issue.

2. 17 de marzo de 2025 a las 14:29 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

3. 17 de marzo de 2025 a las 14:45 UTCresolved

   This incident has been resolved.

1. 10 de marzo de 2025 a las 15:42 UTCinvestigating

   Lokalise is experiencing issues with AI translation tasks and MT orders processing. We are currently investigating this issue.

2. 10 de marzo de 2025 a las 19:01 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

3. 10 de marzo de 2025 a las 19:18 UTCresolved

   This incident has been resolved.

1. 28 de febrero de 2025 a las 17:23 UTCinvestigating

   We are currently experiencing issues with UI import and bulk actions. We are investigating the issue.

2. 28 de febrero de 2025 a las 18:01 UTCidentified

   The issue has been identified and a fix is being implemented.

3. 28 de febrero de 2025 a las 18:12 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

4. 28 de febrero de 2025 a las 18:40 UTCresolved

   This incident has been resolved.

5. 11 de marzo de 2025 a las 11:07 UTCpostmortem

   Between February 27 and March 1, 2025, our service experienced performance degradation in background tasks that impacted essential processes including file imports, synchronization with content integrations, and some bulk actions. **What happened?** A faulty release unintentionally caused our background tasks to stop working as expected. Although the issue did not appear immediately, it became noticeable the next evening. Our team promptly identified the problem, initiated an investigation, and implemented a corrective patch, ensuring full service restoration. **Impact** * **February 27th, 2025, 20:00 UTC to March 1, 2025, 12:02 UTC**: Customers experienced slower performance. * **March 1, 2025, 14:30 UTC**: Corrective patch deployed. **What we are doing to prevent this in the future** We recognize the importance of this event, and are taking further steps to ensure it does not happen again. Our key actions include: * Additional monitoring of background tasks to quickly spot issues. * Exploring options for our system to recover automatically from minor errors. * Enhancing our test coverage to help identify potential issues earlier in the release process. We sincerely apologize for the inconvenience this incident caused. Thank you for your patience and understanding as we continue to improve Lokalise’s resilience and reliability. If you have any questions or require further assistance, please contact [[email protected]](mailto:[email protected]).

1. 19 de febrero de 2025 a las 8:58 UTCinvestigating

   Some actions like file import, bulk actions are not being processed. We are investigating this issue.

2. 19 de febrero de 2025 a las 9:13 UTCinvestigating

   All functionality that is relying on background processes is affected. We are continuing to investigate this issue.

3. 19 de febrero de 2025 a las 9:22 UTCinvestigating

   We have restored the functionality to unblock customers. We are continuing to investigate the root cause.

4. 19 de febrero de 2025 a las 10:16 UTCidentified

   The issue has been identified and a fix is being implemented.

5. 19 de febrero de 2025 a las 10:17 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

6. 19 de febrero de 2025 a las 10:56 UTCresolved

   This incident has been resolved.

7. 27 de febrero de 2025 a las 12:48 UTCpostmortem

   On February 19, 2025, our service experienced performance degradation in background tasks that impacted essential processes including file imports, synchronization with content integrations, and some bulk actions. **What happened?** A faulty release unintentionally caused our background tasks to stop working as expected. Although the issue did not appear immediately, it became noticeable the next morning. Our team promptly investigated and mitigated the problem, quickly restoring full service. **Impact** * 08:54 UTC to 09:23 UTC: Customers experienced slower performance. **What we are doing to prevent this in the future** We recognize the importance of this event, and we are taking steps to ensure it does not happen again. Our key actions include: * Additional monitoring of background tasks to quickly spot issues * Exploring options for our system to recover automatically from minor errors. We sincerely apologize for the inconvenience this incident caused. Thank you for your patience and understanding as we continue to improve Lokalise’s resilience and reliability. If you have any questions or require further assistance, please contact [[email protected]](mailto:[email protected]).

1. 10 de febrero de 2025 a las 12:38 UTCinvestigating

   We are currently investigating this issue.

2. 10 de febrero de 2025 a las 12:40 UTCinvestigating

   We are continuing to investigate this issue.

3. 10 de febrero de 2025 a las 12:50 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

4. 10 de febrero de 2025 a las 12:52 UTCmonitoring

   We are continuing to monitor for any further issues.

5. 10 de febrero de 2025 a las 13:13 UTCmonitoring

   We are investigating the slowness of application and Lokalise OTA service unavailable.

6. 10 de febrero de 2025 a las 13:20 UTCmonitoring

   We have applied the fix and monitoring for issues.

7. 10 de febrero de 2025 a las 13:29 UTCresolved

   This incident has been resolved.

8. 14 de febrero de 2025 a las 8:59 UTCpostmortem

   On February 10, 2025, our service experienced a 16-minute outage followed by 39 minutes of degraded functionality due to an issue with a system configuration. **What happened?** While improving our disaster recovery process, a misconfiguration in the system was introduced unintentionally. Initially, this did not cause issues, but when we attempted to roll back the change, it led to unexpected complications. As a result, our service platform became temporarily unavailable, requiring the reconstruction of certain system components to restore full functionality. **Impact** * **12:30 – 12:46 UTC**: Service outage. * **12:46 – 13:25 UTC**: Service degradation—APP and API were operational. Some services remained unavailable \(OTA, Workflows, Connectors, Review Center\). **What we are doing to prevent this in the future** We recognize the importance of this event, and we have taken steps to ensure it does not happen again. Our key actions include: * Improving validation and monitoring processes to identify configuration issues before deployment. * Enhancing our tools and automations for faster services restoration. * Implementing blue-green deployment techniques, or equivalent, for seamless system upgrades. We sincerely apologize for the disruption this caused and appreciate your patience as we work to make our systems more resilient. If you have any questions, please reach out to [[email protected]](mailto:[email protected]).

1. 5 de febrero de 2025 a las 13:05 UTCinvestigating

   We are seeing issues with import and export functionality of the Lokalise app. We are investigating the issue.

2. 5 de febrero de 2025 a las 13:18 UTCidentified

   The issue has been identified and a fix is being implemented.

3. 5 de febrero de 2025 a las 13:28 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

4. 5 de febrero de 2025 a las 14:44 UTCresolved

   This incident has been resolved.

5. 10 de febrero de 2025 a las 8:19 UTCpostmortem

   On February 5, 2025, we experienced a major service degradation affecting our Import functionality. The incident began at 11:00 UTC and during this time, customers were unable to perform imports in the UI or utilize Git integrations, which may have disrupted their workflows. Services have been fully restored by 13:26 UTC. The outage was caused by a misconfiguration during our migration to a new infrastructure system. To resolve the issue, our team rolled back to the previous system. In response to this incident, we are implementing several measures to prevent similar occurrences in the future: * We will enhance our deployment health checks to cover a previously missing set of dependencies, so that we can catch misconfiguration before newly deployed instances start to be used. * Additional monitoring and alerting systems will be established to quickly identify and address issues as they arise. * We are increasing coverage of our automated tests to ensure all the critical functionality is covered. We sincerely apologize for the inconvenience this incident caused to our customers. We understand how crucial these services are for your operations, and we appreciate your patience as we work to improve our systems and ensure reliable service in the future. Your trust is important to us, and we are committed to enhancing our processes to prevent such occurrences. Thank you for your understanding.