Clearlyip 장애 기록

1. 2026년 9월 2일 PM 3:15 UTCinvestigating

   우리는 현재 고객의 작은 비율로 아웃바운드 통화가 실패하는 문제를 조사하고 있습니다. XCast 플랫폼을 통해 직접 연결하는 고객에 영향을 주지 않습니다.

2. 2026년 9월 2일 PM 3:17 UTCidentified

   문제는 식별되고 수정이 구현됩니다.

3. 2026년 9월 2일 PM 5:45 UTCmonitoring

   수정이 구현되어 결과를 모니터링하고 있습니다.

4. 2026년 9월 3일 AM 11:54 UTCresolved

   이 사건은 해결되었습니다.

공식 인시던트 업데이트를 자동 번역했습니다.

1. 2026년 8월 25일 PM 8:51 UTCinvestigating

   우리는 Microsoft 메일 서비스에 대한 우리의 플랫폼에서 시작 메시지에 대한 메일 배달 지연 증가를보고 있습니다.

2. 2026년 8월 25일 PM 8:55 UTCmonitoring

   Queue 처리는 마이크로소프트가 우리의 플랫폼 (s)에서 우편물을 받아들일 때, 성공적으로, 천천히 계속합니다. 우리는 시간을 초과하는 합격 비율 증가를 보고 기대합니다. 공학은 활동과 납품 개선을 감시하는 것을 계속합니다.

3. 2026년 8월 26일 AM 2:24 UTCmonitoring

   Microsoft 메일 서비스에 대한 향상된 메일 전송률을 모니터링합니다.

4. 2026년 8월 26일 PM 2:07 UTCmonitoring

   우리는 여전히 Microsoft 메일 서비스에 메일을 전달하는 지연을보고 있지만, 합격률은 지속적으로 개선되고 현재 백 로그는 거의 완전히 처리되었습니다. 우리는 성과에 대 한 모니터를 계속 하 고 우리가 일관성 있는 배달율에 만족 한 번이 닫을 것입니다.

5. 2026년 8월 27일 PM 2:18 UTCresolved

   메일은 일반적으로 흐름을 계속하고 지연없이 예상 가능한 배달율.

공식 인시던트 업데이트를 자동 번역했습니다.

1. 2026년 8월 21일 PM 1:46 UTCinvestigating

   우리는 현재 우리의 업스트림 캐리어와 인바운드 통화 품질 문제 투자

2. 2026년 8월 21일 PM 3:07 UTCresolved

   상류 운반대는 대략 9:13am CST에 문제점을 해결했습니다

공식 인시던트 업데이트를 자동 번역했습니다.

1. 2026년 6월 22일 PM 4:59 UTCinvestigating

   우리는 현재 완료 문제와 오디오 문제를 일으키는 미국 동부 트렁킹 서버와 문제가 있습니다.

2. 2026년 6월 22일 PM 6:59 UTCmonitoring

   수정이 구현되어 결과를 모니터링하고 있습니다.

3. 2026년 6월 22일 PM 7:38 UTCresolved

   이 사건은 해결되었습니다.

공식 인시던트 업데이트를 자동 번역했습니다.

1. 2026년 6월 15일 PM 12:53 UTCidentified

   We are currently investigating an issue with our CA-East trunks and have taken them offline at this time.

2. 2026년 6월 15일 PM 4:32 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

3. 2026년 6월 15일 PM 5:48 UTCresolved

   This incident has been resolved.

1. 2026년 6월 9일 PM 7:22 UTCinvestigating

   IS NOC is reviewing reports of connectivity issues for some Pulsar360 hosted PBXs in the Toronto region.

2. 2026년 6월 9일 PM 8:44 UTCinvestigating

   Work continues to isolate and find the root cause for connectivity interruptions in the Toronto region.

3. 2026년 6월 9일 PM 9:04 UTCmonitoring

   IS NOC has identified and resolved the issue(s) responsible for connectivity interruption. We are seeing alarms clearing at this time and will continue to monitor.

4. 2026년 6월 10일 AM 1:16 UTCresolved

   All services remain stable and operational, free from the previously experienced issues.

1. 2026년 6월 9일 PM 2:27 UTCinvestigating

   We are currently investigating an issue which is affecting chat/sms on Clearly Cloud USA - w

2. 2026년 6월 9일 PM 2:50 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

3. 2026년 6월 9일 PM 5:24 UTCresolved

   This incident has been resolved.

1. 2026년 6월 1일 PM 1:54 UTCmonitoring

   We are seeing some isolated connectivity issues in our US-Central region facilities. They seem to be transient in nature and quick to assert/resolve. We are working with our upstream provider(s) and monitoring closely.

2. 2026년 6월 1일 PM 8:13 UTCresolved

   This incident has been resolved.

1. 2026년 5월 31일 PM 6:03 UTCinvestigating

   Our US-Central datacenter is experiencing routing issues at this time. Intermittent problems are resulting from this.

2. 2026년 5월 31일 PM 6:22 UTCidentified

   Our upstream connectivity provider(s) are experiencing degraded network performance which is affecting ClearlyIP/14IP services. They are working diligently to resolve this issue.

3. 2026년 5월 31일 PM 7:48 UTCidentified

   Work continues by our upstream provider(s) to isolate and resolve the root connectivity issues.

4. 2026년 5월 31일 PM 9:31 UTCidentified

   Work continues by our upstream provider(s) to isolate and resolve an apparent DDoS attack affecting multiple customers at the US-Central facilities.

5. 2026년 6월 1일 AM 12:40 UTCmonitoring

   We are seeing recovery but are continuing to monitor the situation while our upstream provider(s) work towards a full resolution.

6. 2026년 6월 1일 AM 1:06 UTCidentified

   We are seeing degraded connectivity once again, affecting services in US-Central. We are awaiting updates and progress reports from our upstream provider(s).

7. 2026년 6월 1일 AM 1:57 UTCmonitoring

   We are once again seeing recovery and are continuing to monitor the situation. Our upstream provider(s) have confirmed positive progress towards resolution of this ongoing DDoS event.

8. 2026년 6월 1일 AM 3:35 UTCmonitoring

   Latest updates from our upstream provider(s) indicate a full restoration of all services and connectivity. We are confirming the same and continue to see clean network operation at this time. We will continue to monitor.

9. 2026년 6월 1일 AM 11:54 UTCresolved

   Connectivity has remained stable.

10. 2026년 6월 10일 PM 8:52 UTCpostmortem

    **Summary / Impact** On the afternoon of May 31st, 2026, 14IP's network operations group began to observe connectivity issues with resources located at its US-Central \(MKE\) data center.  The team began working with their counterparts at the data center facility to understand the nature of the problem.  It was quickly determined that a distributed denial of service \(DDoS\) attack targeted upstream was affecting all tenants of the data center. Significant malicious traffic \(exceeding 450gbps\) was measured upstream during the peak of the attack which overwhelmed equipment as well as primary, secondary, and tertiary links.  This traffic was mitigated through coordinated, escalating filtering with upstream transit providers, as well as internal engineering to stabilize routing. During the incident, connectivity to systems and services primarily located in this data center were impacted significantly for periods of time between approximately 2 PM ET and 9:30 PM ET.  This included Clearly Cloud, ComXchange hosted, Xivea, Business Connect, EPlatform, and several other systems.  \(US-East trunking services continued to operate during the US-Central incident.\) **Follow-Up** Several facility-level network and system improvements have been implemented following the incident which should reduce the risk of future occurrences.  This includes the deployment of additional DDoS scrubbing capacity, improved granularity for detecting attacks, additional fiber capacity upstream, as well as core network enhancements. 14IP plans additional improvements to its systems and operations to improve platform resilience and flexibility when responding to future incidents.

1. 2026년 5월 28일 PM 5:16 UTCidentified

   An upstream Canadian carrier is currently having issues. This may affect incoming calls routed through that carrier, but is limited to ONLY Canadian DIDs through that carrier. Outbound should not be affected. At this time, we have no information from the carrier on ETA.

2. 2026년 5월 28일 PM 6:00 UTCmonitoring

   The upstream carrier is now routing properly. We are monitoring and waiting on more information related to the outage.

3. 2026년 5월 28일 PM 6:50 UTCresolved

   This incident has been resolved.

1. 2026년 4월 27일 PM 6:26 UTCinvestigating

   The team is investigating Call Panel on Clearly Cloud USA displaying a bad certificate.

2. 2026년 4월 27일 PM 6:51 UTCresolved

   This incident has been resolved.

1. 2026년 4월 3일 PM 8:32 UTCinvestigating

   We are seeing issues with provisioning and registration of some Grandstream devices. The team is actively reviewing the situation

2. 2026년 4월 5일 AM 12:01 UTCmonitoring

   We have implemented a fix. If you are still having issues with your devices, please contact us by emailing [email protected] and we can assist you further.

3. 2026년 5월 8일 PM 7:22 UTCresolved

   This incident has been resolved.

1. 2026년 3월 19일 PM 2:55 UTCidentified

   The team has identified the issue and is currently working on a fix.

2. 2026년 3월 19일 PM 3:38 UTCresolved

   This incident has been resolved.

1. 2026년 3월 13일 PM 11:52 UTCinvestigating

   We are currently investigating this issue.

2. 2026년 3월 14일 AM 12:41 UTCidentified

   The issue has been identified and a fix is being implemented.

3. 2026년 3월 14일 AM 1:19 UTCresolved

   This incident has been resolved.

1. 2026년 2월 19일 PM 2:54 UTCinvestigating

   We are currently investigating an issue with Call Panel on Clearly Cloud Canada.

2. 2026년 2월 19일 PM 4:03 UTCinvestigating

   We are continuing to investigate this issue.

3. 2026년 2월 19일 PM 4:04 UTCmonitoring

   A fix is being implemented. If you still are experiencing issues, please open a support ticket and the team will assist.

4. 2026년 2월 19일 PM 5:03 UTCresolved

   This incident has been resolved.

1. 2025년 12월 15일 PM 6:41 UTCinvestigating

   The team is currently investigating an issue with Clearly Cloud USA

2. 2025년 12월 15일 PM 6:42 UTCidentified

   The team has identified the issue and should be resolved quickly.

3. 2025년 12월 15일 PM 6:43 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

4. 2025년 12월 15일 PM 7:00 UTCresolved

   This incident has been resolved.

1. 2025년 12월 4일 PM 6:57 UTCinvestigating

   We are currently investigating this issue.

2. 2025년 12월 4일 PM 7:03 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

3. 2025년 12월 4일 PM 7:24 UTCresolved

   This incident has been resolved.

1. 2025년 12월 2일 PM 4:45 UTCinvestigating

   We are currently investigating inbound and outbound calling issues with Clearly Cloud US platform.

2. 2025년 12월 2일 PM 5:09 UTCidentified

   The issue has been identified and a fix is being implemented.

3. 2025년 12월 2일 PM 5:30 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

4. 2025년 12월 2일 PM 7:46 UTCresolved

   This incident has been resolved.

1. 2025년 11월 11일 PM 7:04 UTCresolved

   SIP2 and Proxy55 are not responding. Calls routing to failover trunks

1. 2025년 11월 3일 PM 4:49 UTCinvestigating

   We are currently investigating this issue.

2. 2025년 11월 3일 PM 4:57 UTCidentified

   The issue has been identified and a fix is being implemented.

3. 2025년 11월 3일 PM 5:29 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

4. 2025년 11월 3일 PM 6:10 UTCresolved

   This incident has been resolved.

5. 2025년 11월 4일 PM 5:13 UTCpostmortem

   At approximately 11:27AM ET on November 3, ClearlyIP's NOC team began to receive system alerts indicating excess session activity within its US-East trunking systems. The team's investigation identified unauthenticated INVITE messages being received from unknown IPs as the source of the excess traffic, as well as a customer generating unexpected levels of outbound calls. Steps were taken to mitigate the traffic's impact, including restarting affected services to clear sessions and restore normal status. By approximately 11:55AM ET, the US-East situation was stabilized and session traffic had returned to normal levels. The team has identified a number of improvements and traffic management strategies to help avoid similar incidents which will be implemented in future maintenance events.