Störungsverlauf von Apwide

We are currently investigating the issue.

We identified a DNS resolution issue on one of our production nodes. The Golive service has been moved to another node and is now available again. We are currently working with our cloud provider to determine the root cause of the issue and implement a fix.

We have identified and resolved the root cause of the issue. One of our nodes was unable to perform DNS resolution due to routing errors. The routing issues have now been fixed, and DNS resolution is working correctly. We apologize for the inconvenience caused.

We are currently experiencing a performance issue on Golive that is making the platform inaccessible. We are actively investigating the root cause. We apologize for the inconvenience.

We are continuing to investigate this issue.

We have disabled traffic from Atlassian webhooks (issue created/updated/deleted events). As a result, Go-Live automation and scheduling conflict checks are currently unavailable. We are continuing to investigate the issue.

Disabling webhook traffic has restored Golive for app navigation. We are currently working on performance fixes so that we can re-enable webhooks (Golive automation and the Scheduling Conflict Checker). We will provide an update soon.

A fix has been implemented and deployed to production. We are reopening webhook traffic to assess whether the solution meets performance requirements. This may result in some instability over the next hour.

Although the fix helped mitigate the performance issue on one of our components, it is not sufficient to handle the current webhook traffic load. We are continuing to work on performance improvements, but in the meantime, we have had to disable webhook traffic again.

A second fix has been deployed to production. Webhook traffic has been re-enabled, and the load appears to be handled correctly. Capabilities such as conflict checking and automation seem to be fully functional. We are continuing to monitor the situation.

Over the past hour, we have experienced several traffic peaks on Atlassian webhook calls, and our updated systems have handled the load without any issues. We now consider this issue to be resolved. We apologize for the inconvenience caused.

**07:23 UTC** – The support team was alerted by monitoring that some database queries were taking longer than expected. **07:34 UTC** – Our uptime probes began raising alerts indicating that Golive was no longer accessible to end users. **07:46 UTC** – The support team updated the status page to notify users of a Golive outage. **08:23 UTC** – The support team identified an abnormal traffic spike originating from Atlassian webhooks. Under normal conditions, traffic peaks reach around 1,000 webhook calls per minute; however, we experienced a sustained spike exceeding 9,000 calls per minute. This caused one of our components to open more database connections than it was designed to handle, forcing it to restart repeatedly. To protect the Golive application, webhook traffic was temporarily disabled. The component stopped restarting and access to Golive was restored. **11:20 UTC** – The engineering team implemented an initial performance improvement on the affected component. The fix was deployed to production, and webhook traffic was re-enabled to evaluate performance. **11:44 UTC** – Webhook traffic was suspended again as the initial fix proved insufficient. Although database connections were no longer an issue, the component was still receiving excessive request traffic. **14:01 UTC** – A second improvement was implemented, this time at the gateway level. The fix was deployed to production and webhook traffic was re-enabled. No further restarts were observed, and Golive appeared to be fully operational. **14:45 UTC** – After several traffic peaks reaching up to 5,000 webhook calls per minute, all components continued to operate normally, with sufficient margins in terms of database connections and execution threads. The incident was closed. With the additional performance improvements, the infrastructure now appears capable of handling a significantly higher load than originally designed. Furthermore, traffic should be better segregated to isolate asynchronous workloads \(webhooks, conflict checking, automation, etc.\) from transactional traffic \(UI application usage\). This separation should help mitigate the risk of asynchronous traffic impacting normal user navigation.

We are currently experiencing an outage affecting Golive and Time Squad since 6:00 AM UTC. Our team is actively working to resolve the issue as quickly as possible. We apologize for the inconvenience and appreciate your patience.

We’ve identified that the issue is coming from our hosting provider. We’re currently in contact with their teams and are working together to resolve the problem as soon as possible. Thank you for your continued patience and understanding.

Our hosting provider has resolved the issue. You can find details about the incident on their status page: https://public-cloud.status-ovhcloud.com/incidents/9qmxv0gbky43 All systems are currently being monitored, and both Golive and Time Squad are now fully operational. Thank you for your patience and understanding throughout this incident.

This incident has been resolved.

**Outage Recap** * **2025-11-05 at 5:56 AM UTC**, our monitoring system began raising alerts indicating that some of our services were unable to communicate with the database. * **6:20 AM UTC:** We reported an outage affecting the **Golive** and **Time Squad** services on our status page. * **6:26 AM UTC:** We suspected the issue originated from our hosting provider’s managed database service. We contacted their support team and opened a ticket. * **6:57 AM UTC:** The hosting provider confirmed the issue on their side and announced a general outage on their platform. [Incident link](https://public-cloud.status-ovhcloud.com/incidents/9qmxv0gbky43) * **7:58 AM UTC:** The hosting provider reported that the issue had been resolved. We restarted our **Golive** and **Time Squad** services. * **8:19 AM UTC:** All of our services were fully restored, and we updated our status page accordingly. * **8:31 AM UTC:** The hosting provider informed us that the root cause was a **DNS issue** on their end. We apologize for the inconvenience caused and appreciate your patience and understanding.

Golive & Time Squad were unavailable this morning for a duration of approximately 30 minutes. We are currently conducting an investigation to identify the root cause.

**After a thorough analysis, we identified that one of our components had reached the maximum number of database connections it was allowed to establish.** As traffic continued to grow, our health check system marked the component as unavailable and automatically restarted it in an attempt to restore normal operation. It took approximately 30 minutes for the component to come back online with a sufficient number of available connections to handle the load. **Remediation:** We have increased the maximum number of allowed connections for this particular component. We sincerely apologize for any inconvenience this outage may have caused. Please rest assured that we are committed to providing the best service possible.

Golive & Time Squad were unavailable this morning for a duration of 1 hour and 10 minutes. The root cause of the issue was a problem accessing the application databases. After contacting our hosting provider, they confirmed that a network maintenance was scheduled for this morning. Traffic was supposed to be rerouted, but a configuration issue caused a routing error. We apologize for the inconvenience caused.

Following our upgrade/outage of 2024-07-08, our hosting provider has to exceptionally patch our nodes. Golive/Time Squad won't be available for 30 minutes. Sorry for this inconvenience.

Here is the reference to the issue currently impacting our cluster: https://public-cloud.status-ovhcloud.com/incidents/ffhr44srdcln

Infrastructure has been patched and issue referenced by https://public-cloud.status-ovhcloud.com/incidents/ffhr44srdcln is now fixed on our cluster. We apologize for the inconvenience it caused.

We are currently experiencing an issue with our hosting provider, causing our applications to be temporarily unavailable. We apologize for any inconvenience and are actively working to resolve the issue as quickly as possible. Thank you for your patience.

Issue has been fixed on hosting provider side. Issue was due to a "CPU spikes on the ingress gateway" (load balancers exposing Golive & TIme Squad application on the web). We apologize for any inconvenience Thank you for your patience.

Apwide Golive is currently experiencing instabilities. Our teams are working to identify the causes.

Platform is stable for 5 hours but analysis to identify the root cause is still ongoing.

A potential root cause has been pinpointed, and a configuration workaround has been implemented in the production environment. We are actively monitoring its impact. Concurrently, efforts are underway to develop a permanent fix.

A fix has been implemented and we are monitoring the results.

The issue appears to be successfully addressed. It stemmed from contention within our system responsible for communication with Jira instances. To remedy this, we have refined the initiation of calls to Jira, reducing their frequency. Additionally, we have augmented the pool size of the system overseeing the contention, enhancing its capacity to handle tasks effectively.

During a short period of time (3:52am UTC to 3:56am UTC), Golive tomcat thread pool was saturated by a high number of requests and was not available. In details: 3:22am UTC: Tomcat thread pool consumption slowly increased (usually 10 threads out of 200 available but increased to 18) but was still able to handle the load. 3:39am UTC: Thread pool consumption increase accelerated. 3:52am UTC: No more available thread in the pool (200 busy threads), Tomcat stopped handling new requests. (outage) 3:56am UTC: Current busy threads completed their tasks, most of them are released, and processing was back to normal. (10 threads out of 200) We are sorry for the inconvenience it caused and we are currently working on the app to mitigate the risk of this happening again.

We are currently investigating this issue.

The issue has been identified by our hosting provider and a fix is being implemented. More information: https://public-cloud.status-ovhcloud.com/incidents/6sd9lwym7zdt

A fix is being implemented by our hosting provider.

Our hosting provider is still working on a fix and we are actively following up. There is no estimated resolution time for the time being.

A fix has been implemented by our hosting provider and the service is restored. We are monitoring the situation.

This incident has been resolved

Some calls to Rest API endpoint GET /environments/search/paginated ends with a HTTP 405. This can impact an integration made with Rest API including Jenkins Shared Library. A fix is in progress.

A fix has been pushed to production and we are currently monitoring its impact.

Fixed deployed to production has only partially solved the problem. Currently, search endpoint is not able to select correct output format in case client calling the API does not specify "accept" content type. A new fix is in progress.

New patch applied on production has fixed the issue and monitoring does not show any new occurrence.

We are currently investigating this issue.

A fixed has been applied on production and seems to have fixed the issue. Environments are under monitoring

After a configuration change, issue seems to be definitely resolved.

This incident has been resolved.

We're currently investigating possible issues impacting emails sent by Golive automation engine and watcher capability.

We've identified the potential root cause of this issue. A fix is in progress.

Fix has been pushed to production and we're currently monitoring if issue is solved.

We are continuing to monitor for any further issues.

Incident is now resolved. Issue was due to the upgrade of Jira cloud rest API client to the v3 which resulted in some email addresses not correctly retrieved by email sending job.

Following an operational/infrastructure change (update of our API gateway), Golive Rest API was in partial outage. Impact: Golive frontend was up and running, but some generated Rest API tokens were considered revoked. This resulted in some failing API calls (HTTP 500). Actions (UTC time): - 3:00am: rollback operational change - 6:32am: re-apply change with first attempt of resolution + monitoring - 8:00am: new occurrences identified on production - 9:32am: push fix for the second attempt of resolution - 10:30am: still occurrences of error found on production - 1:10pm: apply new fix on API gateway + monitoring - 4pm: no more occurrences, problem seems fixed We apologize for the inconvenience caused.

We are currently investigating this issue.

The service is back and we are monitoring it closely.

This incident has been resolved.

Golive and Time Squad cloud were not available during 5 minutes due a network issue to our hosting provider. More details here: https://network.status-ovhcloud.com/incidents/5mldyhd6v99c

We are currently investigating this issue.

We are continuing to investigate this issue.

This incident has been resolved.

We are currently investigating this issue.

The outage is due to a network misconfiguration from our hosting provider. Fix is currently in progress. Golive and Time Squad data are safe.

This incident has been resolved.

We are currently investigating this issue.

This incident has been resolved.

We are currently investigating this issue.

We are continuing to investigate this issue.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.