Memsource Status

We are continuing to investigate this issue.

This incident has been resolved.

This incident has been resolved.

Hello everyone, this is a another test. Please kindly ignore this,

Test: Our team is still investigating this test incident.

Hello, this is a test. Kindly ignore this.

Hello everyone, this is a test so kindly ignore this notification.

### Introduction We would like to share more details about a service disruption that affected Phrase TMS on June 3, 2026. Between 16:39 and 17:02 CEST, customers experienced errors and degraded performance when accessing Phrase TMS in both the EU and US regions. The engineering team identified and resolved the issue within approximately 23 minutes. We apologize for the disruption and are committed to preventing similar incidents in the future. ## Timeline **Jun 3, 2026 @ 16:39 CEST** – Phrase TMS began returning errors for customers in both EU and US regions. Our monitoring/alerting detected this and on-call engineers were alerted and began investigation immediately. **Jun 3, 2026 @ 16:42 CEST** – The root cause was identified as a backwards-incompatible database schema change applied during a recent deployment. A status page update was posted to inform customers that the team was investigating. **Jun 3, 2026 @ 16:53 CEST** – A remediation was prepared to revert the incompatible schema change directly in the database. **Jun 3, 2026 @ 17:00 CEST** – The schema change was successfully reverted in the EU region. Phrase TMS in the EU region recovered and began serving requests normally. **Jun 3, 2026 @ 17:02 CEST** – The same remediation was applied in the US region. Full service was restored across both regions. **Jun 3, 2026 @ 17:48 CEST** – The status page incident was marked as resolved after continued monitoring confirmed stable operation. ## Root Cause During a planned deployment, a database schema change was applied that renamed a column in an internal database table. This change was not backwards-compatible with the version of the application currently running in production. Our production deployment process vs the QA one applies database schema changes separately before rolling out updated application instances — a standard and normally safe approach for additive changes. However, a column rename immediately invalidates queries from the currently running application that reference the original column name. As this did not cause any issues in QA the problem was not immediately caught. As a result, during the window between the schema change being applied and the new application version completing its rollout, all running application instances executed database queries referencing a column that no longer existed, causing all affected requests to fail with errors. ## Actions to Prevent Recurrence The root cause was a backwards-incompatible schema change applied in a single step during a rolling deployment. The following actions have been taken to prevent a recurrence: * **Code review guidelines updated:** Engineering review guidelines now require reviewers to raise a blocking finding for any database schema migration that is not backwards-compatible with the previously deployed application version. This explicitly covers column renames, column drops, table renames, and constraint additions that would cause running application instances to fail. * **Safe schema change pattern enforced:** The engineering team is required to use an incremental, multi-step approach for destructive schema changes: the new structure is added alongside the existing one in a first deployment, the application transitions across subsequent deployments, and the old structure is removed only once no running application instance references it.

This incident has been resolved.

### Introduction We would like to share details about an incident that affected **Phrase Analytics** and Phrase Data between May 4, 2026 and Jun 1, 2026. During this period, segment-based analytics metrics - including character and word counts \(completed, confirmed and locked\), words processed with CJK source language - displayed as zero for data created or updated within the affected window. This post-mortem outlines what happened, when it was resolved, and the steps we are taking to prevent a recurrence. ### Timeline May 4, 2026 - A configuration change to our analytics data ingestion pipeline unintentionally caused it to re-process all historical data rather than only new changes. This placed significant additional load on our message streaming infrastructure, where current changes were competing with past changes and serialised at the end of processing queue. May 7, 2026 - The increased load led to storage pressure within the streaming cluster, causing analytics processing to slow down. May 14, 2026 - We were continuously fixing capacity and work assignments in the streaming cluster to mitigate increased load and allow lag processing. May 21, 2026 at 1:30 PM CEST - We identified the root cause why the lag is not being processed in timely manner and took corrective action: the ingestion pipeline was stopped and restarted in the incremental mode, and additional streaming infrastructure capacity was provisioned. May 22, 2026 at 4:00 AM CEST - New segment data began flowing correctly into the analytics system. May 22, 2026 - A secondary defect was identified and fixed: the pipeline incorrectly handled a specific sequence of segment deletion events within the replay window, which could cause a small number of records to be missed. May 26, 2026 at 10:20 AM CEST - Customer-facing impact ended. Analytics was processing new data correctly and all current-period data was visible. May 27, 2026-Jun 1, 2026 - Historical data from May 4, 2026–May 21, 2026 was backfilled across all processing nodes. When verifying we noticed missing data still for some segments on May 10, 2026, May 14, 2026 and May 15, 2026. Jun 1, 2026-Jun 9, 2026 - With revised and fixed data pipeline we re-run data to backfill missing data. Jun 9, 2026 - Validated and confirmed all data were backfilled. ### Root Cause The incident was caused by a configuration change on May 4, 2026 that accidentally triggered a full re-synchronization of all historical analytics data instead of the intended incremental update. The resulting data volume significantly exceeded the capacity of our message streaming infrastructure. As the system fell behind processing the backlog, analytics statistics stopped being written, causing zeros to appear in customer-facing reports. ### Actions to Prevent Recurrence 1. **Data backfilled and verified** - Historical analytics data for the May 4, 2026–May 21, 2026 window has been restored across all processing nodes and verified against source systems. 2. **Streaming infrastructure scaled** - Capacity of the message streaming cluster has been increased to provide headroom for high-load scenarios and prevent disk pressure from recurring. 3. **Pipeline defects fixed** - A secondary bug affecting a small number of deleted segments in the replay window has been resolved. 4. **Performance improvement underway** - A fix is in progress to dramatically reduce processing time for jobs with unusually large segment volumes, reducing the risk of future processing delays. 5. **Analytics data freshness monitoring** - We are adding internal alerting so that any future delays in the analytics pipeline are detected and acted on before customers are impacted.

The scheduled maintenance has been completed.

The scheduled maintenance has been completed.

The scheduled maintenance has been completed.

The scheduled maintenance has been completed.

The scheduled maintenance has been completed.

The maintenance originally planned for today at 18:00 CEST will no longer take place. It has been postponed and will be rescheduled for a later date in the near future. We will share an update once a new time is confirmed. Thank you for your understanding.

The scheduled maintenance has been completed.

The scheduled maintenance has been completed.

The scheduled maintenance has been completed.

The scheduled maintenance has been completed.