Störungsverlauf von Ziron

We are currently investigating packet loss to our London Volta site which may be affecting some inbound and outbound calls. Our network services provider is investigating and we will update shortly.

We are still waiting for an update from our network services provider, in the meantime we have not seen any issues since the initial update on this incident.

Our network services provider has advised that the issue has been resolved, and we have not seen any further packet loss since 09:18 UTC. We will continue to monitor this incident and will update accordingly.

We understand from our network services provider that they experienced a failure of connectivity between two of the core routers in one of their London sites. The core of the network stayed up and stable but internet traffic going in/out via this site was disrupted. They have isolated the device that appears to have malfunctioned and are investigating further. As per the previous update, no further issues have been seen since 09:18 UTC so we are now marking this incident as resolved.

We are currently investigating this issue.

We are still investigating this issue but are aware of some internet instability in the London area earlier. All outbound call traffic on our Volta platform looks stable at this time.

We are aware of a LINX (London Internet Exchange) LON1 power outage at the Equinix LD6 Slough data centre which was reported at 11:10 UTC, which was around the time we started receiving reports of issues with outbound calls from our London Volta platform. Whilst these are two separate geographical diverse data centres, we believe that the partial failure of a number of LINX peers in LD6 may have caused some route instabilities for a short period. Services have been stable for the last 60 minutes, and we will continue to monitor.

This incident has been resolved.

We are currently investigating a DNS issue that may be affecting the resolution of ziron.com and ziron.net

We are aware of a major global outage of Cloudflare which they are yet to acknowledge. As soon as we have more information we will update this incident.

Cloudflare have acknowledged an issue affecting multiple locations on their network. Our monitoring is now showing a return of services and we will await further updates.

Cloudflare have advised that they believe a router failure in their network caused bad route announcements, which in turn caused issues at a number of Cloudflare locations. They went on to say that they believe that they have addressed the root cause and are monitoring their systems for stability now.

This incident has been resolved.

We are aware of an issue on Vodafone's network in the UK which is currently affecting calls to and from Vodafone mobile subscribers. Separately but seemingly related, it also appears that this is affecting the UK non-emergency numbers (101, 105, 111, 119) which Vodafone operate on behalf of the relevant authorities. Whilst this is not a fault or issue on our network, we wanted to make customers aware and we will update as we receive more information. We are not aware of issues with any other UK operator at this time.

We are seeing an improvement in calls being connected to Vodafone subscribers and calls to non-emergency numbers are now connecting from other operators. We will continue to monitor and update accordingly.

This incident has been resolved.

We are investigating an issue around submission and completion notifications for UK local (geographic) porting requests. This only affects new porting requests and in-flight orders, and does not affect any voice traffic.

We have identified an issue with a third party API, which has been raised to the relevant supplier and they are investigating.

This incident has been resolved.

One of our transit providers is investigating connectivity issues at our London Volta site which began at 10:59 UTC. This may have caused connectivity issues and dropped calls for any customers who do not see this site via LINX or LONAP peering or direct connect.

Our transit provider have updated us to advise that they are not expecting any further disruption from this point, but will be monitoring the situation closely for a stability period. Our own monitoring has not seen any issues since 11:06 UTC but we will continue to monitor.

This incident has been resolved.

We are currently investigating this issue.

We are continuing to investigate this issue.

The issue has been identified and a fix is being implemented.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

We are currently investigating this issue.

A fix has been implemented and we are monitoring the results.

Services have been stable since 1104 UTC, with the cause of the network issues at Volta still under investigation.

We are currently investigating call quality issues with UK inbound and outbound voice. More details to follow.

We have identified an issue with a major UK operator causing call quality issues on inbound calls to UK local (441/442) numbers. Inbound traffic to UK national and toll-free numbers (443, 448) and mobile (447) are unaffected. Outbound traffic has been rerouted around this operator and we are awaiting an update from them.

We are continuing to work on a fix for this issue.

We understand that the UK operator in question has suffered a major fibre break, and call quality is being affected as a result of congestion on their IP network. This is affecting many thousands of customers across the UK. We are awaiting a further update from them and will update in due course.

The operator partner have now advised that they have two major fibre breaks on their national UK network. We are still seeing packet loss across their network but the loss has reduced as the working day draws to a close in the UK. We expect another update by 2000hrs UTC, and will update this incident as soon as we have further news.

Our operator partner have advised that they estimate engineers will arrive at the fibre break in the London area at 9pm at which point they can assess the extent of the work required to restore the fibre connectivity. Work will begin at 9pm on fibre break within the Leicester area, however the required work here is extensive due to an electrical incident in that area. We have been advised that contingency measures have been put into place and we have seen a considerable drop in packet loss. These factors, combined with the overall reduction in UK traffic as we are now outside working hours, means we are satisfied that this incident has been resolved. However, we will continue to monitor and post any service-affecting issues here.

The scheduled emergency maintenance that was due to finish at midnight has unfortunately been extended and is causing some packet loss and routing issues to the London Volta site. Unfortunately we were only notified that this maintenance would be extended at midnight as the scheduled window was coming to an end. We are liaising with our network services provider and will update shortly.

Our network services provider has experienced a failure in a core Juniper device, as soon as we have a further update we'll pass this on. Outbound voice continues to be affected, inbound voice and other services are unaffected.

Service is returning to normal, we are waiting for a further update from our network services provider.

We are again seeing packet loss and connectivity issues at our London Volta site.

Service has again been restored, we are awaiting further updates from our network services provider.

We are continuing to monitor for any further issues.

This incident has been resolved.

We are investigating some routing instability and packet loss around our London sites which is affecting outbound voice and inbound voice traffic.

It appears our network services partner suffered a core router failover at the London Volta site, which in turn caused some short routing instability. Services remain stable and we will continue to monitor.

This incident has now been resolved, however please see https://www.zironstatus.com/incidents/1rl0cfxc1d5f for related emergency maintenance tonight which may affect the London Volta site.

We have identified an issue with a third party provider that is resulting in timeouts for home_network lookups. We are engaging with their support team, and will update shortly. No other lookup services are affected at this time.

The third party provider has advised that they have suffered a failure of an application server, which has now been replaced. Lookups response times have now returned to normal.

This incident has been resolved.

We are currently investigating some intermittent DNS resolution issues on our primary London K8S cluster on which APIv2 is based. This in turn may cause issues with porting request submission via the dashboard and APIv1.

We've discovered an intermittent DNS resolution problem with one of the K8S worker nodes that was causing containers to misbehave. The offending node was drained and all affected containers have been moved to other nodes.

This incident has been resolved.

We have identified that account financial transaction data for the period 2019-09-04 00:02:08 - 2019-09-04 12:25:35 has not been processed correctly. This is in the process of being rectified, but in the meantime usage reports and prepaid balances may be incorrect.

This incident has been resolved.

We are investigating issues with the following lookup services: get_country hashed_imsi

Service is returning to normal - we will update this incident with further information shortly once we have heard back from our operator partner.

This incident has been resolved.

An operator partner have been requested by UK operator EE to cease all lookup traffic towards their network with immediate effect. At this time HLR (legacy) and home_network lookups to UK EE subscribers will fail. Please note that hashed_imsi and get_country lookups to UK EE subscribers are not affected by this issue, nor are HLR (legacy) or home_network lookups to other UK networks or other countries. We are working with other operator partners to find a suitable replacement and will update this incident in due course.

We have rerouted lookup traffic towards UK EE to an alternative operator partner and will continue to monitor traffic statistics.

This incident has been resolved.

We are aware of an issue with an upstream operator partner that has been causing a small increase in call failures to UK fixed and mobile. The operator concerned have advised that they have resolved the issue but we will continue to monitor.

This incident has been resolved.

Our global monitoring has detected some issues with Cloudflare which is causing issues accessing the Ziron dashboard and website from some regions. Cloudflare are investigating a network issue, and we will provide updates when they give them to us.

Service has been restored, and we are awaiting an update from Cloudflare.

Cloudflare have provided the following update: "Major outage impacted all Cloudflare services globally. We saw a massive spike in CPU that caused primary and secondary systems to fall over. We shut down the process that was causing the CPU spike. Service restored to normal within ~30 minutes. We’re now investigating the root cause of what happened."

We're investigating an increase in errors with hashed_imsi and get_country lookups

We are continuing to investigate this issue.

Our operator partner have advised that they are currently investigating a potential issue with their platform and will update us soon with further details.

Our operator partner have advised that the issue with their platform has now been resolved.

We are currently investigating issues with some backend porting services which are causing issues with creating new porting requests via the API or dashboard.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.