Historia awarii Teraswitch

Teraswitch is investigating reports of intermittent network connectivity issues affecting some services in our EWR2 (Newark, NJ) location.

We have identified the issue affecting network connectivity in our EWR2 (Newark, NJ) location and are implementing corrective changes.

We have located an unstable core router in EWR2 and have implemented mitigations to divert traffic away from it.

Teraswitch monitoring reports that all impacts are currently resolved. Please report any further issues to our team. At EWR2, a device sitting between our data center fabric and our internet edge routers experienced a hardware fault that triggered an ASIC-level reload. After the reload, the device came back online in a degraded state and was unable to forward traffic from the data center fabric to the internet edge routers (specifically, it could not process VXLAN-routed traffic). EWR2 operates six of these core routers. Five were unaffected; however, the impaired unit failed to signal that it was incapable of forwarding traffic. Because downstream routing protocols did not detect this bad state, traffic that landed on the affected device was blackholed. Our team manually removed the router from the pool of forwarding destinations for this traffic, restoring service. We will perform all necessary follow-up steps to fully resolve the underlying issue and prevent recurrence.

Teraswitch identified a brief impact to network connectivity at our DUB1 and DUB2 sites beginning at 23:43 UTC. Undersea cables from both London and Amsterdam were lost, resulting in the brief loss of backbone and Internet connectivity at DUB1 and DUB2. Traffic was quickly restored via Internet routes and backbone connectivity via London followed shortly thereafter. Repair of the Amsterdam cable is also underway. Teraswitch is monitoring to confirm stability at these sites and will provide an update in the event of any further issues.

The Concerto cable was fully restored - multiple redundant links are now back online.

Teraswitch has identified a loss of two undersea cables landing in our SEA1 location from Tokyo. This is likely a terrestrial issue inside Seattle. Traffic was automatically diverted to our LA to Tokyo path.

Tokyo to Seattle has been nominal operation for many hours now. We are working with our vendors to understand the issue, but at this time we consider the incident and impact resolved.

Teraswitch is investigating an issue affecting customer management of VAN1 services (power controls, virtual console, metadata retrieval, new service creation, reinstalls). VAN1 services themselves are unaffected by this issue and should be fully operational. We will provide an update as more information becomes available.

Customer management of VAN1 services should be restored at this time. Teraswitch staff are monitoring for any further impacts.

This incident has been resolved.

Teraswitch is investigating a core router failure in TYO2. Traffic is routing over the remaining redundant routers.

After an emergency software upgrade, the router is normalized. Teraswitch is monitoring the operation for further errors.

This incident has been resolved.

Teraswitch has identified a top-of-rack switch failure at our TYO1 site affecting a small subset of customers there. Services single homed to this switch are currently offline. We are working on an immediate replacement and will provide updates as more information becomes available.

The failed switch has been replaced and is being finalized for production. We will provide an update once affected services are fully restored.

Our replacement switch is in service and all affected services should be restored at this time. If you have any further issues, please contact Teraswitch Support at support@teraswitch.com.

This incident has been resolved.

Teraswitch is working to resolve a simultaneous backbone fiber cut affecting AMS1 connectivity.

We are working on a resolution to a multiple diverse fiber cut that is impacting AMS1 connectivity. The issue has been identified by the fiber vendor, and we are working on an alternative fiber solution to restore connectivity.

We have implemented a temporary fix and AMS1 connectivity is now restored. We will follow up with a permanent fix and RCA.

An RCA has been posted for this issue.

# Root Cause Analysis: AMS1 Connectivity Outage **Incident Date:** March 26, 2025 **Duration:** ~1h 21m \(22:08 UTC – 00:14 UTC \+1\) **Severity:** Critical – Full site connectivity loss **Affected Site:** AMS1 – Amsterdam, Netherlands **Status:** Resolved ## Executive Summary On March 26, 2025, Teraswitch's AMS1 facility experienced a complete loss of backbone connectivity lasting approximately 1 hour and 21 minutes. The outage was caused by a scheduled fiber vendor maintenance window that simultaneously impacted both the primary and what was believed to be a diverse redundant fiber path between AMS1 and the rest of the backbone. Investigation revealed that due to a documentation and handoff error at the fiber vendor dating back over a year, the AMS1–AMS2 fiber span had never been migrated to the intended diverse path as part of the AMS3 ring buildout. As a result, both affected spans shared physical infrastructure, eliminating the redundancy intended to protect against exactly this type of event. Connectivity was restored within the maintenance window after a rapid joint audit with the fiber vendor confirmed the provisioning discrepancy, and the AMS1–AMS2 span was moved to its correct diverse path. ## Background Teraswitch's Amsterdam backbone originally consisted of a single dark fiber span between AMS1 and AMS2. When AMS3 was later brought online, the network design called for a three-node fiber ring with fully diverse physical paths between all sites to provide redundant backbone connectivity. To support this design, the fiber vendor was engaged to: 1. Provision a new AMS1–AMS3 span 2. Provision a new AMS2–AMS3 span 3. Reroute the existing AMS1–AMS2 span onto a physically diverse path Due to an internal handoff and documentation error within the fiber vendor, step 3 was not completed. The AMS1–AMS2 span remained on its original physical route. Teraswitch was not made aware of this omission, and the span continued to operate normally for over a year. Because it carried live traffic and appeared correctly in topology, it was not identified as incorrectly provisioned during subsequent audits. ## Timeline of Events | Time \(UTC\) | Event | | --- | --- | | Prior to March 26 | Fiber vendor schedules routine maintenance affecting Amsterdam infrastructure | | ~22:08 | AMS1 backbone connectivity lost. Both AMS1–AMS2 and AMS1–AMS3 paths go down simultaneously. Teraswitch NOC begins triage. | | ~22:53 | Fiber vendor engaged and confirms maintenance is impacting both spans. Root cause identified as shared physical infrastructure due to the original provisioning error. | | ~00:14 \+1 | Fiber vendor migrates AMS1–AMS2 span to the correct diverse physical path. Connectivity restored. Monitoring confirmed stable. | ## Root Cause **Primary cause:** An internal documentation and handoff failure at the fiber vendor resulted in the AMS1–AMS2 dark fiber span never being migrated to its intended physically diverse route during the AMS3 ring buildout. Both the AMS1–AMS2 and AMS1–AMS3 spans shared common physical infrastructure, making the designed ring topology's redundancy ineffective. **Contributing factor:** Because the span was operationally active and traffic was flowing normally, the provisioning error went undetected across both Teraswitch and vendor records for over a year. When the scheduled maintenance affected the shared physical infrastructure, both paths were impacted simultaneously, leaving AMS1 with no available backbone connectivity. ## Impact * **AMS1 customers** experienced a complete loss of inbound and outbound connectivity for approximately 1 hour and 21 minutes. * No data loss or hardware damage occurred. * All other Teraswitch sites were unaffected. ## Resolution Working jointly with the fiber vendor during the incident, Teraswitch engineers and vendor technicians audited the physical path assignments for all Amsterdam spans. The discrepancy between the intended and actual routing of the AMS1–AMS2 span was identified. The vendor migrated the span to the correct physically diverse path, restoring independent redundant connectivity across the AMS1–AMS2–AMS3 ring as originally designed. ## Corrective Actions | Action | Owner | Status | | --- | --- | --- | | Confirm and document physical path diversity for all three Amsterdam spans with fiber vendor | Teraswitch / Fiber Vendor | Complete | | Obtain updated as-built fiber records from vendor reflecting correct path assignments | Fiber Vendor | Complete | | Audit all other Teraswitch sites for similar provisioning discrepancies against vendor records | Teraswitch | Complete | | Establish a fiber path verification checklist for all future vendor provisioning work prior to accepting new spans | Teraswitch | Planned | | Add physical diversity validation to change management process for any future ring or redundancy buildouts | Teraswitch | Planned | | Incorporate optical span latency validation against fiber path build sheets as an acceptance criterion for new span provisioning | Teraswitch | Planned | ## Lessons Learned * **Operational traffic is not proof of correct provisioning.** A span can carry live traffic for an extended period while still being routed incorrectly relative to its intended physical diversity design. * **Redundancy assumptions must be periodically verified against vendor as-built records**, not solely inferred from operational status. * **Fiber vendor handoffs require explicit acceptance criteria** including documented physical path confirmation before provisioning work is considered complete. * **Span latency is a low-cost signal for path verification.** In post-incident review, Teraswitch noted that the measured propagation latency on the AMS1–AMS2 span was slightly lower than expected based on the fiber path build sheet for the intended diverse route. This discrepancy, while subtle, was consistent with the span still traversing the shorter original path. Validating measured latency against estimated values from build sheets at provisioning acceptance could have surfaced this error significantly earlier. This check will be incorporated into the span acceptance process going forward. _RCA prepared by Teraswitch Network Engineering. For questions contact the NOC or network architecture team._

Teraswitch is investigating reports of network connectivity issues affecting some services at our EWR2 site. We have identified the likely cause and implemented a fix, and are monitoring to verify that the issue is resolved.

This incident has been resolved.

Teraswitch is investigating reports of select markets having issues connecting to PIT1.

We have located a network device with software issues, our team is about to reload the device. This issue affects mostly colocation customers at PIT1, but traffic from various sources may have passed through this device depending on the direction of network flow.

Reload is complete and the network operation appears to have returned to normal. We will follow up this notification with a RCA and repair plan.

This incident has been resolved.

Teraswitch is investigating a loss of the majority of VAN1 internet connectivity - possibly related to issues in Seattle.

Cogent has recovered as of 4:33am Eastern. Services should be normalized at this time.

TeraSwitch is aware and tracking a loss of two paths from Seattle to Tokyo. This has caused much of the internet traffic to divert via public Internet Transit.

This incident has been resolved.

We have reached out to the vendor asking for a update. Latency from Singapore to Europe will be elevated until the outage is resolved.

We have successfully brought online a second diverse sub-sea cable path and latency has returned to normal. Once the original severed cable comes back online, we will now have full path redundancy.

Teraswitch has identified thre core backbone links that are down within the central US. This is causing US-East traffic to wrap around the world to get to US-West. Chicago/Seattle Chicago/Salt Lake City Dallas /Los Angeles We are working with our fiber vendors for a ETR.

Chicago to Salt Lake City path is restored which has normalized operations with 1 of 3 links restored. We will close this status when 2 of 3 are restored which also restores redundancy.

This incident has been resolved.

Teraswitch is investigating reports that our console (console.tsw.io) is currently inaccessible. This issue has been identified as the result of an outage in an underlying cloud provider. We will provide an update as more information becomes available.

This issue is due to an ongoing Cloudflare outage: https://www.cloudflarestatus.com/incidents/m1xvmqf37z97. Cloudflare has implemented a fix and we are monitoring for recovery. Our API is unaffected by this issue and remains accessible.

Cloudflare has resolved the issue, and console.tsw.io is accessible once again as of 22:28 UTC. This incident has been resolved.

We are investing significant packet loss on the west coast US.

We are continuing to investigate this issue.

This incident has been resolved.

At 2PM UTC, Teraswitch reloaded software related to our route servers to fix "stuck" transport tunneling sessions. This may have caused interruptions to ULL/HFT and transport services over our network. We believe the first stuck session started approximately 6 hours before, with more dropping off over the next few hours. As of this moment, all stuck sessions appear to be resolved, and traffic is passing normally and via ULL links.

Due to multiple undersea cable cuts, SGP1 and SGP2 is no longer operating on the Teraswitch backbone. SGP1/2 networking being unreachable for a time period was seen as network routes shifted and Internet routes adjusted.

This incident has been resolved.

Teraswitch is investigating increased errors in our service portal (console.tsw.io). Teraswitch services themselves are unaffected. We will provide an update as more information becomes available.

The cause of the increased errors was identified and fixes put in place. This issue is resolved.

Terawitch Network Operations has observed a subsea cable fault between our Seattle and Tokyo POPs. Latency to Tokyo may be increased while the system is down. We have reached out to our cable provider for an update.

Fault has cleared and cable returned to service.

Teraswitch is investigating monitoring alerts/errors in our console (console.tsw.io) and API (api.tsw.io). This appears to be due to an underlying major cloud provider which is experiencing an outage. We will provide an update as more information becomes available.

This issue has been isolated to an ongoing Cloudflare network outage - see their incident page for more details: https://www.cloudflarestatus.com/incidents/8gmgl950y3h7

Cloudflare updated their incident report that their services are now operating normally - they will post a final update to the incident once their investigation has concluded. Teraswitch monitoring confirms our console / API services have stabilized with no further errors. This incident is now resolved.