Treasure Data Status

We have applied the fix, and are observing that traffic is flowing correctly now.

The upstream provider has restored service.

This incident has been resolved.

## Post-Mortem Summary This incident was related to a database maintenance performed on January 19, during which management of Plazma table resources was migrated from the existing RDBMS to a dedicated Aurora database. The purpose of this change was to better isolate table resource workloads and improve long-term scalability. The migration itself completed successfully, and both the original database and the newly introduced Aurora database were operating normally immediately after the maintenance. On the following day, when internal batch processes began running, the newly introduced database experienced unexpected load. Investigation determined that the internal batch system was configured to connect to the master database endpoint instead of a reader endpoint. As a result, internal batch processing competed with user-facing requests, causing increased latency when accessing table resources. To mitigate the impact and enable rapid recovery, temporary database tuning was applied. This tuning was intended solely as a short-term measure to stabilize the system and has since been fully reverted. The permanent fix consisted of correcting the configuration of the internal batch system so that it accesses the appropriate database endpoint. Following these actions, system performance recovered in both the US and Tokyo regions, and all services returned to normal operation. No data loss or data corruption occurred. To prevent similar issues in the future, we are reviewing our configuration and deployment practices, including reducing configuration differences between staging and production environments and strengthening validation of database endpoints used by internal workloads.

An unexpected behavior occurred in the REST API "GET /v3/job/result/{job_id}" which caused it to return only a partial set of results. This issue stemmed from an update to the job result download mechanism implemented around 2025/12/09 01:00 (UTC). The issue was resolved after a fix was applied around 2025/12/10 04:36(UTC). The affected API is used through the following methods: - Directly calling the REST API - Using functions in libraries (e.g., pytd) that reference job results - Toolbelt commands that refers to job results - Setting store_last_results:true in the td>:/td_run> operator in Treasure Workflows - Using the td_for_each>, td_wait> and td_wait_table>: operators in Treasure Workflows Unfortunately, we are unable to identify which API requests were affected. Please compare the number of results retrieved by your job with the download results to see if you were affected. We sincerely apologize for the inconvenience this has caused.

We confirmed the third-party service is now operational. The incident has been resolved.

Services have been restored to normal operations, and all of our systems are reporting healthy once again. If you observe any lingering issues, please connect with our support team, and we would be happy to help: https://docs.treasuredata.com/articles/#!pd/contacting-treasure-data-support

Delays should be improving for all customers, and services should be returning to normal. Given the similarity of the two issues, we'll continue tracking any concerns related to this interruption as part of the "[US Region] Issue Affecting Treasure Data Services" issue. For further updates, please track here: https://status.treasuredata.com/incidents/w7v5wtmkdllj If you experience continued issues specifically relating to our Profiles API or SSO Logins, please contact our support team: https://docs.treasuredata.com/articles/#!pd/contacting-treasure-data-support

Services have been restored to normal operations, and all of our systems are reporting healthy once again. If you observe any lingering issues, please connect with our support team, and we would be happy to help: https://docs.treasuredata.com/articles/#!pd/contacting-treasure-data-support

After a successful monitoring period, we can confirm that our fix is working and the issue is now resolved. Thank you for your patience.

The database maintenance has been completed. All systems are operating normally.

The database maintenance has been completed. All systems are operating normally.

The database maintenance has been completed. All systems are operating normally.

The database maintenance has been completed. All systems are operating normally.

The scheduled database maintenance is now complete, and all systems have been successfully validated. Services are fully operational, and no further impact is expected. If you encounter any issues, please reach out to our support team.

The database maintenance has been completed. All systems are operating normally.

The scheduled database maintenance is now complete, and all systems have been successfully validated. Services are fully operational, and no further impact is expected. If you encounter any issues, please reach out to our support team.

The scheduled database maintenance is now complete, and all systems have been successfully validated. Services are fully operational, and no further impact is expected. If you encounter any issues, please reach out to our support team.

The scheduled database maintenance is now complete, and all systems have been successfully validated. Services are fully operational, and no further impact is expected. If you encounter any issues, please reach out to our support team.

The scheduled database maintenance is now complete, and all systems have been successfully validated. Services are fully operational, and no further impact is expected. If you encounter any issues, please reach out to our support team.