2600hz outage history

Out of an abundance of caution due to ongoing external factors, we have temporarily disabled the Provisioner while our team puts an update in place. During this time, customer access to the Support Portal is also unavailable. All other services continue to operate normally. While the portals are unavailable, you can still reach our Support team: Email: Send your request to [email protected] and a ticket will be opened for you. Phone: For urgent issues, our support line is accepting inbound calls as usual at 415-886-7950. We apologize for the inconvenience and will post an update here as soon as access is restored. Thank you for your patience.

We are continuing to work on a fix for this issue.

Please be advised, TTS servers are also impacted. Though we believe we have mitigated this without impact to clients. If you experience any problems generating new TTS, please contact our support team.

Provisioner is operational and continues to be monitored for stability. The Support Portal will remain available via email only. Cluster Manager (CM) will remain unavailable at this time. We will provide additional updates as needed.

We are currently working on restoring Cluster Manager. As part of this work, the IP addresses associated with Cluster Manager will be changing. Customers with firewall rules or IP allowlists may need to update their configurations. We will provide the new IP addresses once they are available. We will continue to provide updates as work progresses.

Cluster Manager (CM) is now back up and fully functional. If you would like new IP addresses, please email [email protected]. Aside from the Support Portal web interface, all services are operating normally. Our team is continuing to work on restoring the web interface, and we will post an update as soon as it is available. In the meantime, Support remains reachable by email at [email protected] or by phone at 415-886-7950. Thank you for your patience.

We are seeing reports of issues when attempting to provision Yealink phones. We have identified the root cause and are working to implement a fix.

We are continuing to work on a fix for this issue.

A fix has been implemented, and the issue is now fixed. We will continue to monitor.

We are investigating potential API issues on the ZSwitch US system in the SJC zone.

We are continuing to investigate this issue.

We have identified the issue and are working on resolving the issue.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

We're currently seeing heightened alerts related to connectivity between zones, we're engaging our infrastructure team to investigate further. We'll update here as soon as we know more, in the meantime please raise a ticket with us if you're experiencing any degradation to service.

The issues we saw should now be resolved; this was limited to hosts in the SJC zone. We will send out a full RCA in due course, please let us know if you experience any further issues.

We are currently investigating Zabbix alerts/reports of outages affecting portal/calls. We will send out an update as ASAP.

This appears to a problem with Level3 currently; our infrastructure team is investigating implementing routing changes to workaround the issue.

Cloudflare appears to have stopped advertising our 8.X addresses causing this outage; we have now re-enabled them. We're working with CloudFlare to investigate the root cause but are seeing issues resolve from our side.

This incident has been resolved.

We are seeing degraded performance and intermittent issues in the ORD zone due to a Cloudflare issue: https://www.cloudflarestatus.com/ We are monitoring the situation. Users who are experiencing issues can route away from the ORD zone by cutting over to the east or west zones.

Cloudflare has reported a fix has been implemented. We are monitoring for issues.

This incident has been resolved.

We are investigating an issue with a partial outage due to loss of connectivity to one of our zones.

We are experiencing uptime issues with our San Jose datacenter and are working to resolve it. Phones that are properly configured to register to SRV/NAPTR records should fail over to the other zones.

We are seeing a fiber related issue in our San Jose datacenter and are working to resolve it.

We have resolved the issue and believe we are operational at this time. We are monitoring the incident.

This incident has been resolved.

We've observed issues with Inbound/Outbound SMS and MMS and have confirmed this appears to be consistently impacting all of ZSwitch.

This incident has been resolved.

We are currently investigating an issue with Zswitch US' database services. We have engaged our engineering and operations team to work this as highest priority. We will update this page as soon as we know more.

This issue has now been resolved and the database services are operating as expected. We are working on a full RCA and will update this statuspage with it in due course. If you have any questions in the meantime please contact support using the normal methods.

This issue has now been resolved and the database services are operating as expected. We are working on a full RCA and will update this statuspage with it in due course. If you have any questions in the meantime please contact support using the normal methods. We will continue to monitor throughout the day.

We've stabilized services and are continuing to monitor the situation.

We are seeing further degradation in the system and are investigating

We have identified a database system outage and are working with our engineers to restore the system.

We have brought services back online and are continuing to monitor for stability.

This incident has been resolved. We will continue to monitor for stability.

Please be advised that we are currently hearing of widespread ISP issues across the country. Major media outlets are reporting Verizon-related outages, but we suspect the impact is at least somewhat wider. No 2600Hz services are directly impacted at this time. However, resellers can expect to see increased reports from clients, especially Verizon clients

Please be advised that we are currently hearing of widespread ISP issues across the country. Major media outlets are reporting Verizon-related outages, but we suspect the impact is at least somewhat wider. No 2600Hz services are directly impacted at this time. However, resellers can expect to see increased reports from clients, especially Verizon clients

This incident has been resolved.

We are currently investigating reports of issues with loading Call Center in our UI and comm.land.

We have identified that we were receiving alerts from two of our apps servers having issues connecting to each other. We've restarted Kazoo Apps on both servers. We've received confirmation from some customers that this is resolved, we've received no new reports, and we've been unable to replicate since the restarts. We're monitoring to ensure this is resolved.

We are continuing to monitor for any further issues.

This incident has been resolved.

We are investigating reports of degraded call services in our EWR zone

We have mitigated the issue and our engineers are working to identify the root cause

This incident has been resolved.

We are receiving reports of call completion issues on EWR. We are investigating.

We have mitigated the issue and are monitoring the system for any further issues.

This incident has been resolved.

We are seeing increasing reports of semi-attended transfers not reaching the recipient's voice mail if the recipient does not answer. This is impacting our Hosted Platform only and we are currently investigating the issue.

We believe we have identified the issue. We have deployed a patch to the SJC zone and are currently testing before we deploy to ORD and EWR.

Testing has been successful so far and we've deployed the patch to all remaining Hosted Platform servers. We believe this to be fixed now, but will continue monitoring for any further issues.

This incident has been resolved.

We are receiving reports of call completion issues over web sockets registered devices including comm.land phones

At this time we believe this issue affected clients that were using comm.land AND immediately updated their comm.land software this morning. This issue should now be resolved, we are monitoring the situation to validate.

We've had multiple customer reports that this issue is now resolved

Presently, a concern has arisen wherein one out of the three Kamailio servers located in EWR is encountering difficulty in processing SIP registrations. It's important to note that this issue is not expected to affect our services, as phones that are appropriately configured will automatically attempt registration through alternate SBCs and/or zones. If you're currently facing challenges with phone registrations, we encourage you to reach out to our support team. This way, we can assist you in enhancing your phone's configuration to proactively avoid similar issues moving forward.

We would like to inform you that several clients have reported successful phone registrations after we implemented a TCP traffic block to and from this specific SBC. This leads us to believe that the issue is currently fully mitigated. In light of this, we are intentionally leaving this SBC in a degraded state to facilitate thorough investigation by our engineering teams as required. Our intention is to conduct a server reboot tomorrow evening. We will keep this incident active until that time to facilitate full transparency into the issue.

This incident has been resolved.

We are currently investigating this issue.

After investigation Kamailio was restarted which resolved the issue - Please let us know if you encounter any further disruption. We are working on investigating the root cause.

We are receiving increased Reports Of One Way Audio Issues in EWR datacenter

At this time we do believe this is isolated to clients on a single ISP. We are continuing to investigate

We do believe this issue is with a particular peer. We are advertising new BGP routes to attempt to work around this issue

We have multiple confirmations that our workaround has resolved the issue

We are receiving reports of call completion issues in EWR

We have paused the EWR zone and are monitoring for any further issues

This incident has been resolved.

At 13:13:56 PT we received alerts that multiple BGP sessions had dropped in our US-West datacenter. This included our backhaul between US-East and US-West, as well as one of our outbound ISPs. The backhaul traffic failed over to the secondary by 13:14:03 PT, and the session with the outbound ISP recovered by 13:25:21 PT. We believe there was no impact on inbound, or outbound traffic to/from the 2600Hz network during this incident. We have engaged our providers to get more information on what occurred, but all seems to have recovered by 13:25:21 PT. Thank you, 2600Hz Infrastructure