Cytracom 장애 기록

One of the servers in our fleet experiencing network connectivity issues and our engineering team is investigating. If you are in that server, you may experience intermittent call failures. We will resolve this quickly and thanks for your patience.

We restarted the network interface and the calls should flow as usual. Thanks for your patience during this issue.

This incident has been resolved.

We are currently investigating an issue causing partial inbound and outbound call failures for some customers. Affected users may experience failed call attempts, delayed call setup, or intermittent call completion issues. The issue does not appear to impact all calls or all customers. Our engineering team is actively investigating and working to restore full calling functionality. We will provide additional updates as more information becomes available.

We have taken corrective action to address the partial inbound and outbound call failures affecting some customers. A small number of call processing servers were restarted to restore normal call flow. Call traffic has recovered, and we are currently monitoring the platform to confirm continued stability. Some users may have experienced failed call attempts, delayed call setup, or intermittent call completion issues during the incident. We will continue monitoring and provide a final update once the issue is fully resolved.

The issue causing partial inbound and outbound call failures has been resolved. Corrective action was taken by restarting a small number of call processing servers, which restored normal call flow. After continued monitoring, inbound and outbound calling have remained stable. Some customers may have experienced failed call attempts, delayed call setup, or intermittent call completion issues during the incident window. We apologize for the disruption and appreciate your patience while our team worked to restore service.

We are currently investigating an issue affecting the Realtime Call Center dashboard. Call center event processing is delayed due to a backlog in the event processing service, which may cause dashboard status, agent state, and call activity updates to appear stale or delayed. Inbound and outbound calling are not impacted. Calls continue to route and complete normally. Our team is working to restore realtime event processing and will provide updates as we make progress.

**Update: Realtime Call Center Dashboard Delays** We have identified the issue affecting the Realtime Call Center dashboard. A backend event processing service became degraded due to a high volume of invalid messages, which caused call center status updates to be delayed. A fix has been implemented, and queued events are now being processed. The dashboard may continue to show delayed or stale status information while the backlog clears. Inbound and outbound calling remain unaffected. Calls continue to route and complete normally. We expect the Realtime Call Center dashboard to be current within the next hour.

Queued events are now processing concurrently, and the Realtime Call Center dashboard is recovering. Due to the backlog and out-of-sequence event processing during recovery, some dashboards may show incorrect call-waiting counts for today. We are working to correct these values where possible. Inbound and outbound calling remain unaffected. Calls continue to route and complete normally. Realtime dashboard activity is expected to normalize as event processing catches up. Daily dashboard statistics should display as usual starting tomorrow.

This issue has been resolved.

We are currently experiencing a backlog in our call event processing system, which may cause delays in event-related updates. This may impact CDR processing, BLF status updates, and voicemail arrival indicators. Inbound and outbound calling remain operational, and voicemail delivery is not impacted. Our team is actively working to clear the backlog and will continue to monitor the system.

We have identified and corrected the issue. Call Detail Records (CDRs) in Insights during the incident will be available within next 12 hours. New events are processing normally and are not expected to experience delays. Inbound and outbound calling, voicemail delivery, and new CDR processing are not impacted at this time. We will continue monitoring until the backlog is fully cleared.

All backlog Call Details Records have been processed. This issue is now resolved.

We are currently experiencing a backlog in our call event processing system, which may cause delays in event-related updates. This may impact CDR processing, BLF status updates, and voicemail arrival indicators. Inbound and outbound calling remain operational, and voicemail delivery is not impacted. Our team is actively working to clear the backlog and will continue to monitor the system.

We have identified and corrected the issue that caused a backlog in our event processing pipeline. The issue was related to a failed event consumer, which has now been restarted, and queued messages are processing successfully. Some Call Detail Records (CDRs) in Insights may continue to be delayed for the next few hours while the CDR processor catches up on previously delayed events. New events are processing normally and are not expected to experience delays. Inbound and outbound calling, voicemail delivery, and new CDR processing are not impacted at this time. We will continue monitoring until the backlog is fully cleared.

The backlog processing has been completed. This issue is now resolved.

We are currently investigating an issue causing intermittent inbound and outbound call failures for some customers. Our engineering team is actively working to identify the root cause and restore normal call processing. We will provide an update as soon as more information is available.

We have identified and mitigated the issue that caused intermittent call setup failures for some customers. The issue was related to a degraded backend component that impacted call processing on a subset of voice infrastructure. Our engineering team applied corrective action, and calls are now processing normally. We are continuing to monitor the platform closely to ensure stability.

This incident has been resolved.

We are currently investigating intermittent inbound and outbound call failures affecting some customers. Our team is actively working to identify the issue and will provide updates as they become available.

We identified an issue affecting call processing on a subset of voice servers, which resulted in intermittent inbound and outbound call failures for some customers. Our engineering team applied corrective action, and call processing has returned to normal. We are continuing to monitor the platform closely to ensure stability.

The issue causing intermittent inbound and outbound call failures has been resolved. Our engineering team identified a capacity constraint in a backend call processing service and applied corrective action. Calls are now processing normally, and the platform remains stable.

This incident has been resolved.

We have identified that our third-party CNAM provider is not responding within our required performance thresholds. To prevent call completion delays or failures, we are temporarily bypassing Caller ID name resolution for affected calls. During this time, calls may complete without a resolved caller identity or display name. Call delivery itself remains operational. Our team is working on a remediation plan and expects to implement a permanent solution within the next 72 hours. We will continue to monitor the service and provide updates as progress is made.

The Caller Id resolution is working as expected. We are monitoring for any further issues.

The CNAM (Caller ID) service is fully functional without any performance constraints.

The UCaaS platform is experiencing intermittent inbound/outbound call failures. We are investigating the root cause.

We have identified and mitigated an issue that was causing intermittent call completion failures. The issue was related to a third-party CNAM lookup service responding slower than expected. This delay caused some calls to fail over to our secondary data center, which resulted in intermittent disruption to call completion. We have temporarily disabled this dependency, and calls are now completing as expected. We are continuing to monitor call completion and overall service stability.

We are continuing to monitor for any further issues.

This incident is marked as resolved. No further inbound call failures were observed.

In our primary data center inbound/outbound calls are failing in several voice Pods and they are being routed to our secondary data center. Callers may experience intermittent call failures until we resolve this issue.

The issue has been identified and a fix is being implemented.

We identified an issue with a third-party call-signing service provider and temporarily disabled that dependency to restore normal call flow. Calls are currently completing as expected. We are continuing to monitor call completion and will implement a remediation plan to reduce future impact from this dependency.

Call flow has returned to normal, and calls are completing as expected. We are closing this incident. Our team will continue with the planned remediation work to reduce future impact from this third-party call-signing dependency.

Between 7:30–7:45 AM CST, one of our voice servers experienced an issue that prevented it from processing inbound and outbound calls for a small subset of customers. The issue was isolated to a service module on the affected server. We restarted the server, and call processing resumed normally afterward.

We are currently investigating an issue impacting Agent connectivity in our US-SouthEast (ATL) Region.

The issue has been identified and a fix is being implemented.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

We are investigating an issue that is causing ControlOne's Management portal and device management API to respond to requests more slowly than usual. The management portal may appear unresponsive, and ControlOne Agents attempting to connect may experience difficulties connecting.

The issue has been identified and a fix is being implemented.

We are continuing to work on a fix for this issue.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

Cytracom Engineering has identified an issue impairing outbound calls for a subset of our customers. We are investigating the root cause and working towards a full resolution. As updates become available, they will be posted here.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

Some customers reported issues with inbound and outbound calls on the Desktop App. Affected users were advised to log out and log back in to resolve the issue.

We are continuing to monitor for any further issues.

This incident has been resolved.

A subset of customers reported issues with call parking not functioning correctly. The root cause has been identified, and a remediation is being developed.

A limited number of D2 devices experienced an issue with call parking due to a firmware defect. This is not a service outage and does not affect the broader UCaaS platform or other device types. The issue has been identified, and a remediation plan is in progress. Affected devices will be updated to resolve the issue. We are continuing to monitor and will provide updates as remediation is completed.

The issue affecting a limited number of D2 devices related to call parking has been resolved. The root cause was identified as a firmware defect, and affected devices have been remediated. All services are operating normally, and no broader impact to the UCaaS platform was observed. We will continue to monitor to ensure ongoing stability.

We are investigating an issue impacting the UCaaS Management Portal.

The service has been restarted. All management functions are operational, and the system is stable.

This issue has been resolved and Management Portal is working without any further issue.

The UCaaS Management Portal is currently unavailable, preventing users from accessing administrative functions and configuration interfaces. Core calling services are not impacted at this time; however, administrators are unable to manage users, devices, or settings through the portal. Engineering teams have been engaged and are actively investigating the root cause. Updates will be provided as more information becomes available.

The issue affecting the UCaaS Management Portal has been resolved. Users can now access the portal and perform administrative functions normally. The incident was caused by a spike in usage that exhausted available HTTP connections to the backend. The affected service was restarted to restore functionality. A plan is in place to increase service capacity to prevent recurrence. We are continuing to monitor the stability of the service.

The issue affecting the UCaaS Management Portal has been resolved. Users can now access the portal and perform administrative functions normally. The incident was caused by a spike in usage that exhausted available HTTP connections to the backend. The affected service was restarted, and a plan is in place to increase capacity to prevent recurrence.

We are experiencing intermittent network disruptions impacting call completion. We are currently investigating the issue.

The upstream routing issue has been resolved. We are observing normal call completion rates across affected routes.

Resolved: The upstream routing issue appears resolved. Call completion is operating within normal thresholds. Root cause is under investigation.

We are investigating a partial outbound call failure for a subset of customers.

A partial outbound calling issue has been resolved. During the incident, one of our outbound proxy servers stopped processing calls. The affected system was restarted and migrated to a different host, restoring normal service. We are continuing to monitor the platform to ensure stability and are reviewing the event to prevent recurrence.

We are continuing to monitor for any further issues.

There are no observed reports of outbound/inbound call failures. This issue is now resolved.