Apwide status

The scheduled maintenance has been completed.

Started 2026年8月16日 UTC 18:00 · Resolved 2026年8月16日 UTC 19:00

## Summary On **2026-07-19**, our production services experienced an extended outage following a planned database security maintenance announced by OVH. While the maintenance was initially expected to last no more than **30 minutes**, an infrastructure failure at the OVH datacenter significantly prolonged the interruption. After assessing the situation and the available recovery options, we activated our Business Continuity Plan \(BCP\) by restoring the production database from the latest available backup and upgrading it to PostgreSQL 18. All services were fully restored on **2026-07-20 at 11:15 UTC**. ## Customer Impact During the incident, **Golive** and **Time Squad** were unavailable for approximately **20 hours**, preventing customers from accessing the applications and performing normal operations. As part of the recovery process, we restored the database from the latest available backup. The most recent backup had been completed **25 minutes before the outage began**, resulting in a **potential data loss window of up to 25 minutes**. A review of the database audit logs showed that, due to the weekend period, only a limited number of write operations occurred during this timeframe. These updates were primarily related to deployment and monitoring activities, significantly reducing the effective customer impact. ## Root Cause The root cause of the incident was a **hardware failure affecting a storage device in the OVH datacenter** during a planned security maintenance operation on the managed PostgreSQL infrastructure operated by Aiven on behalf of OVH. Initially, the incident was communicated as an extension of the planned maintenance, delaying visibility into the actual nature and severity of the problem. Several hours later, OVH published a dedicated incident confirming that the outage was caused by a hardware failure rather than the maintenance activity itself. ## Incident Timeline ### Context The incident occurred during a planned maintenance window announced by OVH on **2026-07-16**. OVH informed customers that they would apply a security patch to our production database over the weekend. The expected service interruption was **no longer than 30 minutes**. ### 2026-07-19 **14:39 UTC** Our monitoring systems and availability probes detected an outage affecting our production services. As this matched the scheduled maintenance window, our support engineers considered the interruption expected. **16:00 UTC** After allowing an additional one-hour buffer beyond the planned maintenance window, our monitoring continued to report that the database was unavailable. Our support engineers checked the OVH status page, which still showed the maintenance as ongoing with no additional information. The OVH control panel also reported the database status as **"Updating"**. At this stage, the extended downtime was still considered part of the maintenance. **17:15 UTC** The OVH status page remained unchanged. To gather additional information, our engineers consulted the public OVH Discord community, where other customers reported experiencing the same issue: databases remained in the **"Updating"** state while waiting for the security patch to complete. **21:20 UTC** With the announced maintenance window approaching its end, the database still unavailable, and business hours about to begin in the Australia/Oceania region, our support engineers opened a **Priority 1 \(P1\)** support ticket with OVH. **21:25 UTC** OVH acknowledged receipt of the incident and confirmed that the ticket had been assigned. **21:41 UTC** OVH confirmed an incident involving **Aiven**, the service responsible for managing the cloud database platform. They indicated that the issue was related to the planned maintenance operation and that the outage was taking significantly longer than the initially announced 30 minutes. They committed to providing further updates. [https://public-cloud.status-ovhcloud.com/incidents/f02sjn0qw27f](https://public-cloud.status-ovhcloud.com/incidents/f02sjn0qw27f) At this point, we updated the Golive and Time Squad status pages to report a service outage. ### 2026-07-20 **00:46 UTC** After more than ten hours of downtime without any meaningful update, our support engineers requested additional information from OVH to determine whether our Business Continuity Plan \(BCP\) should be activated. During this investigation, we also discovered that the OVH user interface did not allow us to create a database fork from an existing backup. **04:55 UTC** As no progress had been communicated publicly, our support engineers contacted OVH support again and also reached out to the on-call support team. **05:30 UTC** OVH support confirmed that the planned maintenance was still in progress but could not provide further details regarding the broader incident. We expressed our concerns about the lack of communication and requested additional information to support our operational decision-making. We also requested that OVH create a dedicated public incident rather than continuing to associate the outage solely with the maintenance operation. **07:20 UTC** Our engineers discovered that creating a database fork from a backup using the OVH REST API was possible, even though the option was unavailable in the web interface. The most recent backup had been created at **14:15 UTC**, while the outage started at **14:40 UTC**, resulting in a potential data loss window of approximately **25 minutes**. After reviewing the database audit logs, our engineers confirmed that only a limited number of write operations had occurred during this period, primarily related to deployment and status monitoring activities due to the weekend. Based on this assessment, management decided to activate the Business Continuity Plan. **07:48 UTC** OVH published a dedicated incident, confirming that the outage was caused by a hardware failure rather than simply an extended maintenance window. [https://public-cloud.status-ovhcloud.com/incidents/bn54j3232tlv](https://public-cloud.status-ovhcloud.com/incidents/bn54j3232tlv) **08:00 UTC** Because our PostgreSQL version did not support database forking through the OVH web interface, we took advantage of the BCP activation to upgrade the database to **PostgreSQL 18**, a version that had already been successfully validated in our development and staging environments. **08:50 UTC** The restored database became available on PostgreSQL 18. Database integrity and sanity checks completed successfully, and Time Squad was restarted. **08:55 UTC** Time Squad was fully operational. Our status page was updated, and Golive services were restarted. **09:05 UTC** Despite several restart attempts, some Golive components failed to start correctly. Our engineers began investigating the root cause. **09:30 UTC** The investigation identified the issue as a backlog of asynchronous Atlassian webhook events \(issue creation, updates, etc.\) that had accumulated during the outage. Once connectivity was restored, Atlassian attempted to replay all missed events simultaneously, creating a significant load on our infrastructure before caches had been rebuilt. **09:55 UTC** A temporary mitigation was implemented by blocking incoming webhook traffic. This allowed Golive to complete its startup sequence and resume serving normal transactional traffic. The status page was updated accordingly. **10:13 UTC** A performance improvement that had already been validated in our development and staging environments and scheduled for an upcoming release was deployed to production. Webhook traffic was then re-enabled, and the combination of rebuilt caches and the performance improvement proved effective. **11:15 UTC** After more than one hour of stable event processing and normal system behavior, the incident was declared resolved. The status page was updated to indicate that all systems were fully operational. ## Actions Taken During the incident, we initially could not activate our Business Continuity Plan because the OVH web interface did not allow us to create a database fork from a backup. This limitation was caused by the legacy PostgreSQL version running on our production environment. As part of the recovery process, we upgraded our production database to **PostgreSQL 18**, the latest version supported by OVH and already validated in our non-production environments. This upgrade removes this limitation and enables faster recovery should a similar incident occur in the future. ## Follow-up Actions * Work with OVH to improve communication during major incidents, particularly regarding impact assessment and incident classification. * Request earlier publication of dedicated incident reports instead of extending planned maintenance notifications when an unexpected failure occurs. * Continue reviewing and improving our Business Continuity Plan activation criteria to enable faster decision-making when infrastructure providers experience prolonged outages. * Follow up with OVH on the restoration of the original database service, which remains unavailable at the time of writing this post-mortem.

Started 2026年7月17日 UTC 18:30 · Resolved 2026年7月20日 UTC 11:42

The scheduled maintenance has been completed.

Started 2026年7月15日 UTC 18:30 · Resolved 2026年7月16日 UTC 07:30

The scheduled maintenance has been completed.

Started 2026年7月9日 UTC 18:30 · Resolved 2026年7月9日 UTC 22:30

This maintenance must be cancelled as the necessary prerequisites have not been met.

Started 2025年12月15日 UTC 14:45 · Resolved 2025年12月20日 UTC 17:43

The scheduled maintenance has been completed.

Started 2025年10月25日 UTC 17:00 · Resolved 2025年10月25日 UTC 18:00

Maintenance has been completed. DNS propagation is now complete, and the old load balancers have been decommissioned.

Started 2025年10月22日 UTC 11:00 · Resolved 2025年10月23日 UTC 11:29

The scheduled maintenance has been completed.

Started 2025年1月18日 UTC 18:00 · Resolved 2025年1月18日 UTC 18:46