Cytracom outage history

We have identified an issue causing a partial service degradation for a subset of customers in the [US-SouthEast] region. ControlOne Agents may experience intermittent connectivity or "Retrying" status

The issue has been identified and a fix is being implemented.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

We are currently investigating service degradation in the US-Midwest (ORD) Region, affecting connectivity for a subset of customers.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

We are currently investigating a service degradation in the US-Midwest (ORD) Region impacting connectivity for a subset of customers.

We are continuing to investigate this issue.

The issue has been identified and a fix is being implemented.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

We are investigating an issue that is causing ControlOne's Management portal and device management API to respond to requests more slowly than usual. The management portal may appear unresponsive, and ControlOne Agents attempting to connect may experience difficulties connecting.

The issue has been identified and a fix is being implemented.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

We are currently investigating an issue affecting inbound calls for some customers. The disruption is related to a fiber network failure impacting one of our upstream service providers. Our team is actively working with the provider to monitor recovery and restore normal inbound calling as quickly as possible. Outbound calling (unless routed through this provider) and other services are not currently impacted. We will provide additional updates as more information becomes available.

We are continuing to investigate this issue.

We have identified the source of intermittent inbound call drops affecting a subset of customers. Our upstream voice carrier has confirmed a network incident causing packet loss on a portion of voice traffic. This is resulting in inbound calls intermittently failing to ring through or dropping shortly after connection. Our carrier is actively working to restore normal calling behavior, and we are closely monitoring call quality across affected accounts. We will provide updates as we have more information.

Our upstream carrier is observing expected calling behavior and will continue to monitor for stability.

This incident has been resolved.

We are currently investigating a partial service degradation in the US-SouthWest (LAX) Region. Some customers are experiencing higher than normal latency and intermittent connectivity.

A fix has been implemented and we are monitoring the results.

A third-party fiber cut affecting multiple regional carrier network paths in Southern California has been repaired, and services have been restored.

We are currently investigating a partial service degradation in the US-Southeast (ATL) Region impacting bridge connectivity for a subset of customers.

Impacted workloads have been rescheduled to alternate compute hosts and we are monitoring the results. The impacted hardware will undergo remediation actions for the affected physical components.

This incident has been resolved.

We are currently investigating a partial service degradation in the US-SouthWest (LAX) Region impacting bridge connectivity for a subset of customers.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

We have identified an issue affecting ControlOne Agent geolocation services. Agents may intermittently fail a location in the portal, or fail a device posture check that requires a location fix. As a workaround partners can remove geolocation from the policy.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

We are currently investigating an issue causing a subset of ControlOne Sites to report an incorrect ISP status in the ControlOne portal. This is a visual problem only and does not impact site connectivity.

The issue has been identified and a fix is being implemented.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

We are currently investigating a partial service degradation in the US-Southeast (ATL) Region.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

We are investigating an issue that is causing ControlOne's Management portal and device management API to respond to configuration changes more slowly than usual. The management portal may appear unresponsive until this clears.

This incident has been resolved.

We are investigating an issue impacting inbound and outbound call failures. We we identified an incorrect configuration that caused this issue. We are working on a fix and will update soon.

A fix has been implemented and we the calls are flowing as usual. We are closely monitoring for any other issues.

This incident has been resolved. We have implemented an extra step so the automated config management doesn't trigger outside a declared maintenance window. We apologize for the disruption caused by this.

We have identified an issue causing a partial service degradation for a subset of customers in the US-SouthCentral ControlOne region. ControlOne Bridges may experience intermittent connectivity ControlOne Agents may experience intermittent connectivity or "Retrying" status

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

We have identified an issue causing a partial service degradation for a 15% subset of customers in the US-Southeast ControlOne region. ControlOne Bridges may experience intermittent connectivity ControlOne Agents may experience intermittent connectivity or "Retrying" status

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

One of the servers in our fleet experiencing network connectivity issues and our engineering team is investigating. If you are in that server, you may experience intermittent call failures. We will resolve this quickly and thanks for your patience.

We restarted the network interface and the calls should flow as usual. Thanks for your patience during this issue.

This incident has been resolved.

We are currently investigating an issue causing partial inbound and outbound call failures for some customers. Affected users may experience failed call attempts, delayed call setup, or intermittent call completion issues. The issue does not appear to impact all calls or all customers. Our engineering team is actively investigating and working to restore full calling functionality. We will provide additional updates as more information becomes available.

We have taken corrective action to address the partial inbound and outbound call failures affecting some customers. A small number of call processing servers were restarted to restore normal call flow. Call traffic has recovered, and we are currently monitoring the platform to confirm continued stability. Some users may have experienced failed call attempts, delayed call setup, or intermittent call completion issues during the incident. We will continue monitoring and provide a final update once the issue is fully resolved.

The issue causing partial inbound and outbound call failures has been resolved. Corrective action was taken by restarting a small number of call processing servers, which restored normal call flow. After continued monitoring, inbound and outbound calling have remained stable. Some customers may have experienced failed call attempts, delayed call setup, or intermittent call completion issues during the incident window. We apologize for the disruption and appreciate your patience while our team worked to restore service.

We are currently investigating an issue affecting the Realtime Call Center dashboard. Call center event processing is delayed due to a backlog in the event processing service, which may cause dashboard status, agent state, and call activity updates to appear stale or delayed. Inbound and outbound calling are not impacted. Calls continue to route and complete normally. Our team is working to restore realtime event processing and will provide updates as we make progress.

**Update: Realtime Call Center Dashboard Delays** We have identified the issue affecting the Realtime Call Center dashboard. A backend event processing service became degraded due to a high volume of invalid messages, which caused call center status updates to be delayed. A fix has been implemented, and queued events are now being processed. The dashboard may continue to show delayed or stale status information while the backlog clears. Inbound and outbound calling remain unaffected. Calls continue to route and complete normally. We expect the Realtime Call Center dashboard to be current within the next hour.

Queued events are now processing concurrently, and the Realtime Call Center dashboard is recovering. Due to the backlog and out-of-sequence event processing during recovery, some dashboards may show incorrect call-waiting counts for today. We are working to correct these values where possible. Inbound and outbound calling remain unaffected. Calls continue to route and complete normally. Realtime dashboard activity is expected to normalize as event processing catches up. Daily dashboard statistics should display as usual starting tomorrow.

This issue has been resolved.

We are currently experiencing a backlog in our call event processing system, which may cause delays in event-related updates. This may impact CDR processing, BLF status updates, and voicemail arrival indicators. Inbound and outbound calling remain operational, and voicemail delivery is not impacted. Our team is actively working to clear the backlog and will continue to monitor the system.

We have identified and corrected the issue. Call Detail Records (CDRs) in Insights during the incident will be available within next 12 hours. New events are processing normally and are not expected to experience delays. Inbound and outbound calling, voicemail delivery, and new CDR processing are not impacted at this time. We will continue monitoring until the backlog is fully cleared.

All backlog Call Details Records have been processed. This issue is now resolved.

We are currently experiencing a backlog in our call event processing system, which may cause delays in event-related updates. This may impact CDR processing, BLF status updates, and voicemail arrival indicators. Inbound and outbound calling remain operational, and voicemail delivery is not impacted. Our team is actively working to clear the backlog and will continue to monitor the system.

We have identified and corrected the issue that caused a backlog in our event processing pipeline. The issue was related to a failed event consumer, which has now been restarted, and queued messages are processing successfully. Some Call Detail Records (CDRs) in Insights may continue to be delayed for the next few hours while the CDR processor catches up on previously delayed events. New events are processing normally and are not expected to experience delays. Inbound and outbound calling, voicemail delivery, and new CDR processing are not impacted at this time. We will continue monitoring until the backlog is fully cleared.

The backlog processing has been completed. This issue is now resolved.