Xmatters Status

**What happened?**  On May 1st, 2026, some customers reported an issue to xMatters Customer Support where attempting to send a message via the web user interface or viewing an alert on the Alerts report resulted in an error being displayed. The issue only affected the Reporting functions in the EMEA region; the system continued to accept signals, generate alerts, and send notifications across all regions.  **Why did it happen?**  The issue occurred when, during routine database maintenance, the database called a mismatched version of the library, resulting in an internal database error. The version mismatch within the cluster was traced to a prior database engine upgrade where a subset of replica nodes did not restart into the upgraded version. At no point was there any risk to data integrity.  **How did we respond?**  As soon as Customer Support confirmed the issue, they engaged the Engineering teams, who were able to identify the root cause and restore version consistency across all nodes. The teams validated stability and confirmed that all services were restored.  **What are we doing to prevent it from happening again?**  The Engineering Team has added explicit post-upgrade verification checks and monitoring to ensure node alignment is confirmed and maintained. This will provide safeguards to ensure node version alignment after upgrades and prevent this issue from reoccurring.

**What happened?**  On April 1st, 2026, some customers reported an issue to xMatters Customer Support where the Alerts or Notifications reports were timing out and failing to load.  **Why did it happen?**  This issue occurred because a backend service was experiencing significant unexpected load that caused report processing to be delayed. The ongoing resource constraint resulted in timeouts.  **How did we respond?**  As soon as customers reported an issue, Customer Support launched an investigation and escalated to the Engineering teams. The teams initiated rolling restart procedures for the applicable backend services to restore functionality. Once the rolling restart was completed, service was fully restored.  **What are we doing to prevent it from happening again?**  The Engineering teams are currently working to identify any possible bottlenecks that may have caused performance issues. In the interim, they have increased and adjusted resource allocation for several services to handle potential processing delays and prevent potential recurrences.

**What happened?**  On March 10th, 2026, some customers reported an issue to xMatters Customer Support where they were encountering a 404 error page when attempting to log in to their instances via SSO. Some users may also have encountered a “We’ve run into a problem while retrieving your data.” error message.  **Why did it happen?**  This issue occurred during a routine maintenance update to the xMatters platform. Although several components of the platform were updated, specific configurations of the component related to SSO-based authentication conflicted with the update and resulted in 404 errors. This issue was limited to those few customers that had specific criteria set for their SSO configuration.  **How did we respond?**  As soon as customers reported the issue, Customer Support verified the issue and escalated immediately to Engineering. The team traced the issue to the maintenance deployment and initiated rollback procedures to restore functionality. Once the rollback was completed, the 404 errors ceased and service was confirmed restored.  **What are we doing to prevent it from happening again?**  The Engineering teams have implemented additional testing for any configuration criteria related to the SSO-based authentication component. In addition, they have begun working on an improved maintenance plan to prevent further issues that could occur during similar deployments.

**What happened?** On November 21, 2025, at 12:50 PM UTC, the xMatters internal monitoring tools detected irregular behavior in how internal traffic was being routed. Some customers in the APAC region communicating with services in North America \(specifically US-East\) may have encountered intermittent request failures or increased latency. Only traffic between these two regions was affected; all other systems and regions continued normal operations. **Why did it happen?** A temporary network disruption between Australia Southeast and US-East caused one internal routing node in Australia to lose accurate information about available backend systems in USEast. The node generated an incomplete routing configuration and temporarily stopped directing traffic to US-East. Under normal circumstances, routing updates refresh automatically when connectivity returns. In this case, the affected node did not recover cleanly and remained in a stale state until Engineering intervened. **How did we respond?** As soon as Engineering was alerted through internal monitoring, they engaged with the platform engineering team, service owners and Customer Support to launch an investigation. The teams reached out to impacted customers to validate issue symptoms and restarted routing components in both affected regions to force a configuration refresh. Once the restart completed, routing returned to normal levels while the teams continued to monitor and investigate the root cause. They were able to confirm that only one routing node and specific cross-region traffic was impacted. **What are we doing to prevent it from happening again?** While teams were mitigating this issue, they created new alerting rules to detect the routing patterns they observed during the incident and expanded internal monitoring to help identify when routing nodes fail to refresh their configuration or otherwise enter a ‘stale’ state. The teams also have planned and prepared infrastructure updates that will further reduce the risk of similar issues. These include improved configuration recovery behavior, enhanced stability for routing components, and additional logging and observability improvements for diagnosing routing anomalies. They will deploy these updates once the current code freeze window has elapsed.

**What happened?**  On October 20th, 2025, at approximately 1:21 AM Pacific, customers began reporting an issue affecting live call routing, conferences, and voice notifications. During this issue, customers in all regions would have been affected. **Why did it happen?**  This issue was caused by a global AWS outage that impacted one of our downstream providers, resulting in voice notifications, live call routing, and conferencing they were handling to fail.  **How did we respond?**  As soon as the first customer reported an issue, Customer Support engaged the Engineering teams and launched an investigation. Once they identified the root cause, the teams updated the primary provider for affected regions to an alternate provider that was not affected by the AWS outage. When the new provider was assigned, customers reported that all services were operating correctly.  **What are we doing to prevent it from happening again?**  While this issue was out of our control or that of our provider, we are working to identify potential ways to improve resilience in case of external factors such as this.

**What happened?**  On October 10th, 2025, at approximately 4:14 PM Pacific, the xMatters internal monitoring tools alerted Customer Support to service degradation related to an issue that was already being internally monitored with the Integration Platform in the North American region. While this issue was being investigated and mitigated, customers may have experienced intermittent request failures and occasional slowdowns.  **Why did it happen?**  These particular issues were caused by a security update that caused conflicts with the underlying runtime environment, specifically the memory management routine. Slow performance of the routine was triggering frequent health checks for a request processing service and causing automatic restarts.  **How did we respond?**  As soon as the internal monitoring tools alerted Engineering to a potential issue, they began performing manual rolling restarts and deployed more forgiving liveness checks to avoid increasing error rates and to minimize any potential impact to customers. They also increased resources for the impacted service to improve responsiveness of the underlying routine and deployed a configuration fix to the Http Client Cache to try and stabilize the system. While mitigating the potential impact, they also increased monitoring levels as they continued to investigate the root cause and were able to deploy an update to the service and environment configuration on October 13 that resolved the issue. They continued monitoring and confirmed that the system was stable and all services were operational.  **What are we doing to prevent it from happening again?**  Engineering has updated the backend service and deployed additional updates and monitoring to the system configuration that will improve overall stability for the environment and prevent this issue from reoccurring.

**What happened?**  On September 23rd, 2025, at approximately 8:31 PM Pacific, the xMatters internal monitoring tools alerted the xMatters Support Team to an issue with Apple iOS notifications. Users were not receiving notifications on Apple iOS devices; other device types and alert and response processing were not impacted and continued to operate without interruption.  **Why did it happen?**  The issue occurred because of a backend service patch applied by the Engineering teams that affected necessary dependencies for Apple Push notification delivery.  **How did we respond?**  As soon as the monitoring tools alerted the Support team, they engaged the Engineering teams to launch an investigation. The teams quickly identified the source of the issue as a recent backend service patch and initiated a rollback of the patch. Once the rollback was complete, the teams confirmed that all services had been restored.  **What are we doing to prevent it from happening again?**  The Engineering team is working to enhance testing of future patches and improving visibility into the health of services. This will help to proactively resolve issues or, when necessary, initiate rollbacks more quickly without impacting customers.

### What happened? On August 5th, 2025, at approximately 7:53 PM Pacific, the xMatters internal monitoring tools alerted the xMatters Support team to an issue in the North America region with notification delivery. Some xMatters customers may have experienced delays when receiving notifications or noticed failed alerts in the system. ### Why did it happen? The issue occurred because of a network problem that disrupted communication between parts of the queuing system. This caused some components to become temporarily out of sync, leading to timeouts, internal connectivity failures, and a small number of messages not being processed during the disruption. As the system recovered, some performance was degraded until normal operation could be restored. ### How did we respond? As soon as the xMatters monitoring tools reported an issue, the xMatters Support Team initiated the internal Major Incident Management process and engaged the Engineering and incident response teams. The teams quickly mitigated the issue and restored performance by redirecting traffic from our message queuing system in the affected region \(us-central\) to our message queuing system in another local region \(us-east\). After the issue was resolved, the teams directed traffic back to the restored region. ### What are we doing to prevent it from happening again? The Engineering team has determined that the best approach to prevent this issue from reoccurring is to replace the current message queuing system. Work on the replacement system is well underway and the teams will retire the current system as soon as they have finished the replacement.

We have now resolved the issue with our support center.

**What happened?**  On June 12th, 2025, at approximately 11:29 AM Pacific, the xMatters internal monitoring tools alerted to an issue in the North America region with the xMatters integration platform. While the issue was in progress, some xMatters customers may have experienced workflows not executing correctly or signals not being accepted as expected.  **Why did it happen?**  The issue occurred because multiple Google Cloud and Google Workspace products experienced external API issues for about three hours between the hours of 10:49 AM to 1:49 PM Pacific.  **How did we respond?**  As soon as the xMatters monitoring tools reported an issue with the xMatters integration platform, the xMatters Customer Support team initiated the internal Major Incident Management process and engaged the Engineering and incident response teams. The teams quickly determined that the issues were due to problems with an upstream provider and immediately opened a support case for them to investigate. They continued to work with the provider throughout the incident and confirmed with their engineers as they recovered the underlying dependencies.  **What are we doing to prevent it from happening again?**  The xMatters Support and Engineering teams requested an RCA and prevention methods from the upstream provider. In their response, they confirmed that they will be modularizing their Service Control architecture to prevent failures from affecting the entire service.

The scheduled maintenance has been completed.

The scheduled maintenance has been completed.

The scheduled maintenance has been completed.

The scheduled maintenance has been completed.

The scheduled maintenance has been completed.

The scheduled maintenance has been completed.

The scheduled maintenance has been completed.

The scheduled maintenance has been completed.

The scheduled maintenance has been completed.

The scheduled maintenance has been completed.