Historique des pannes de Teraswitch

1. 12 août 2026 à 05:18 UTCmonitoring

   (Ce statut est séparé d'une question antérieure qui, à l'origine, aurait été liée.) Les clients de LON1, AMS1, AMS2, AMS3, DUB1, DUB2, FRA2, SGP1, SGP2, TYO1, TYO2 et TYO3 ont connu une perte d'accessibilité aux destinations Internet, ainsi qu'aux destinations internes de base de Teraswitch entre les sites touchés. D'autres sites nord-américains n'ont pas été touchés. MIA1 (Miami, FL) a été intentionnellement retiré de l'épine dorsale dans le cadre de notre réponse. MIA1 reste accessible et en service, mais fonctionne actuellement sans connectivité complète de la colonne vertébrale, le trafic à destination et en provenance d'autres sites Teraswitch est acheminé sur Internet public plutôt que sur notre colonne vertébrale privée. Les clients qui comptent sur MIA1 pour la connectivité privée ou inter-site devraient s'attendre à modifier les caractéristiques de la latence et du chemin jusqu'à ce que MIA1 soit réintégré. Cause : Teraswitch utilise une route par défaut (0.0.0.0/0) en interne pour signaler qu'un routeur de bord est capable d'avancer le trafic vers Internet. Chaque site préfère normalement la valeur par défaut de ses propres routeurs locaux, avec des attributs d'itinéraire qui distinguent une origine locale de celle distante. Une route par défaut a été propagée à MIA1 avec ses métriques et ses communautés dépouillées et un sentier AS contenant seulement notre propre ASN. À l'heure actuelle, nous comprenons que cette route n'a jamais existé dans les routeurs de MIA1 en tant que route valide. Un réflecteur de route sur notre site AMS2 a propagé cette route modifiée sur nos marchés de l'UE et de l'APAC. Les routeurs de bord de réception l'interprétaient comme originaire locale et le préféraient à leur propre défaut local valide. Ces routeurs ont ensuite annoncé la route vers le centre de données en aval, qui l'a rejeté comme invalide. En l'absence d'une présence par défaut acceptable, les tissus du site touchés ont cessé de transmettre le trafic à leurs propres routeurs, ce qui a entraîné une perte d'accessibilité. Résolution : Les ingénieurs ont identifié la voie mal formée dans les 10 minutes suivant le début et ont retiré le MIA1 de l'épine dorsale pour arrêter la propagation. Les sites touchés ont été reconvergés sur leurs routes et le service local par défaut a été restauré à 04:16:15 UTC. Prochaines étapes: Le défaut sous-jacent, que ce soit dans les routeurs MIA1 ou dans le réflecteur de route, n'a pas encore été identifié. Nous travaillons à reproduire la condition et avons engagé notre fournisseur. Entre-temps, nous mettons en oeuvre une politique visant à faire respecter les attributs minimaux requis sur les routes par défaut d'origine interne afin qu'une publicité malformée ne puisse être préférée à une publicité locale valide. MIA1 restera hors de l'épine dorsale jusqu'à ce que le défaut soit compris et que les mesures d'atténuation soient vérifiées; nous publierons une mise à jour lorsqu'il sera réintégré. Un RFO complet suivra.

2. 12 août 2026 à 18:29 UTCresolved

   Working with our network hardware vendor, we have deployed a global change across all compute sites to improve the reliability of the connection between our Internet/backbone edge routers and our data center core networks. This change addresses the blackhole condition that caused the overnight outage. With it in place, a malformed route entering the fabric will no longer prevent traffic forwarding - sites will continue to forward via their local edge routers rather than losing reachability. We considered the previous behavior a critical flaw and have prioritized this fix accordingly. All sites, including the unaffected data centers, had this configuration change completed successfully earlier today. We continue to investigate the underlying condition that caused the default route from MIA1 to be advertised/readvertised with incorrect attributes, and will publish a full RFO once that analysis is complete. Please note: a separate, unrelated incident affecting our global traffic routing systems remains open. That work involves a distinct vendor defect and route scaling improvements, and is not connected to the outage described here.

Traduit automatiquement depuis la mise à jour officielle de l'incident.

1. 29 juillet 2026 à 19:15 UTCinvestigating

   Teraswitch examine l'instabilité générale d'Internet.

2. 29 juillet 2026 à 19:35 UTCidentified

   Teraswitch travaille actuellement sur les mesures d'atténuation liées à cette question, bien que les problèmes de trafic aient été réglés. À l'heure actuelle, il semble qu'une grande partie des câbles américains à grande capacité vers l'UE aient eu une faille à peu près au même moment. Notre chemin de New York à Londres, qui avait été coupé et résolu plus tôt aujourd'hui, est retombé, et notre chemin d'Ashburn à Francfort a subi une panne à la même époque. À l'heure actuelle, le réseau s'est stabilisé et nous nous efforçons de veiller à ce que les événements supplémentaires n'aient pas autant d'impact sur le trafic.

3. 31 juillet 2026 à 14:33 UTCmonitoring

   Depuis l'impact initial, l'équipe d'ingénierie réseau de Teraswitchs a identifié l'instabilité comme une défaillance logicielle de routage déclenchée lors d'un changement de configuration de routine. Nous avons éliminé la condition de déclenchement et mis en place des mesures de protection pour prévenir la récidive. Le réseau demeure stable pendant la mise à niveau des logiciels et l'assainissement permanent. Notre chronologie actuelle des événements est que l'instabilité a été déclenchée vers 19:09:21–27 UTC le 29 Juillet 2026, et a duré ~5 minutes. D'autres mises à jour et une ACR complète seront publiées dans les prochaines mises à jour.

4. 12 août 2026 à 04:01 UTCinvestigating

   [This message has been deleted, a distinct incident has been opened instead.]

Traduit automatiquement depuis la mise à jour officielle de l'incident.

1. 28 juillet 2026 à 16:17 UTCinvestigating

   Teraswitch examine les problèmes liés au rack de calcul R102 dans TYO2.

2. 28 juillet 2026 à 16:52 UTCresolved

   Un éventuel bug firmware/hardware a été identifié, et Teraswitch CNO a choisi d'effectuer une mise à jour logicielle séquentielle sur la paire de commutateurs redondants dans R102. Une fois les recharges terminées, les opérations semblaient normalisées. Nous étudierons cette paire de commutateurs réseau pour mieux comprendre l'événement. Veuillez signaler tout problème en suspens avec TYO2 à [email protected] ou à notre équipe directement.

Traduit automatiquement depuis la mise à jour officielle de l'incident.

1. 17 juillet 2026 à 23:45 UTCinvestigating

   Teraswitch enquête sur les rapports de perte de connectivité Internet à SGP2. Nous fournirons une mise à jour sous peu.

2. 18 juillet 2026 à 00:22 UTCidentified

   Teraswitch a isolé cet incident à un événement de maintenance de l'alimentation du fournisseur avec un impact inattendu, affectant à la fois nos racks réseau et causant une perte de connectivité Internet à SGP2. Les services à la clientèle seraient alimentés sans connectivité en amont. Le personnel de l'établissement s'emploie à rétablir la connectivité Internet et à remettre le site en ligne. Nous fournirons une mise à jour dès que possible.

3. 18 juillet 2026 à 01:57 UTCresolved

   Notre partenaire de l'installation a restauré la puissance des racks réseau SGP2 de Teraswitch. Cause profonde : les deux PDU redondants de chaque rack réseau ont été alimentés par des boîtes d'alimentation reliées à la même barre de bus électrique, créant ainsi un seul point de défaillance. Les ingénieurs de l'installation ont déplacé une boîte de secours dans le bon bus, rétablissant l'alimentation électrique de l'équipement du réseau et éliminant le seul point de défaillance. Nous ne nous attendons à aucun autre impact à ce stade. Les systèmes clients n'ont pas perdu de puissance à aucun moment pendant cet événement.

Traduit automatiquement depuis la mise à jour officielle de l'incident.

1. 8 juillet 2026 à 17:48 UTCinvestigating

   Teraswitch is investigating reports of intermittent network connectivity issues affecting some services in our EWR2 (Newark, NJ) location.

2. 8 juillet 2026 à 18:01 UTCidentified

   We have identified the issue affecting network connectivity in our EWR2 (Newark, NJ) location and are implementing corrective changes.

3. 8 juillet 2026 à 18:01 UTCidentified

   We have located an unstable core router in EWR2 and have implemented mitigations to divert traffic away from it.

4. 8 juillet 2026 à 18:20 UTCresolved

   Teraswitch monitoring reports that all impacts are currently resolved. Please report any further issues to our team. At EWR2, a device sitting between our data center fabric and our internet edge routers experienced a hardware fault that triggered an ASIC-level reload. After the reload, the device came back online in a degraded state and was unable to forward traffic from the data center fabric to the internet edge routers (specifically, it could not process VXLAN-routed traffic). EWR2 operates six of these core routers. Five were unaffected; however, the impaired unit failed to signal that it was incapable of forwarding traffic. Because downstream routing protocols did not detect this bad state, traffic that landed on the affected device was blackholed. Our team manually removed the router from the pool of forwarding destinations for this traffic, restoring service. We will perform all necessary follow-up steps to fully resolve the underlying issue and prevent recurrence.

1. 10 juin 2026 à 00:24 UTCmonitoring

   Teraswitch identified a brief impact to network connectivity at our DUB1 and DUB2 sites beginning at 23:43 UTC. Undersea cables from both London and Amsterdam were lost, resulting in the brief loss of backbone and Internet connectivity at DUB1 and DUB2. Traffic was quickly restored via Internet routes and backbone connectivity via London followed shortly thereafter. Repair of the Amsterdam cable is also underway. Teraswitch is monitoring to confirm stability at these sites and will provide an update in the event of any further issues.

2. 23 juin 2026 à 18:43 UTCresolved

   The Concerto cable was fully restored - multiple redundant links are now back online.

1. 4 juin 2026 à 15:27 UTCidentified

   Teraswitch has identified a loss of two undersea cables landing in our SEA1 location from Tokyo. This is likely a terrestrial issue inside Seattle. Traffic was automatically diverted to our LA to Tokyo path.

2. 4 juin 2026 à 21:11 UTCresolved

   Tokyo to Seattle has been nominal operation for many hours now. We are working with our vendors to understand the issue, but at this time we consider the incident and impact resolved.

1. 12 mai 2026 à 21:18 UTCinvestigating

   Teraswitch is investigating an issue affecting customer management of VAN1 services (power controls, virtual console, metadata retrieval, new service creation, reinstalls). VAN1 services themselves are unaffected by this issue and should be fully operational. We will provide an update as more information becomes available.

2. 12 mai 2026 à 21:50 UTCmonitoring

   Customer management of VAN1 services should be restored at this time. Teraswitch staff are monitoring for any further impacts.

3. 22 mai 2026 à 18:22 UTCresolved

   This incident has been resolved.

1. 7 mai 2026 à 02:03 UTCinvestigating

   Teraswitch is investigating a core router failure in TYO2. Traffic is routing over the remaining redundant routers.

2. 7 mai 2026 à 02:29 UTCmonitoring

   After an emergency software upgrade, the router is normalized. Teraswitch is monitoring the operation for further errors.

3. 22 mai 2026 à 18:24 UTCresolved

   This incident has been resolved.

1. 30 avril 2026 à 23:33 UTCidentified

   Teraswitch has identified a top-of-rack switch failure at our TYO1 site affecting a small subset of customers there. Services single homed to this switch are currently offline. We are working on an immediate replacement and will provide updates as more information becomes available.

2. 1 mai 2026 à 04:01 UTCidentified

   The failed switch has been replaced and is being finalized for production. We will provide an update once affected services are fully restored.

3. 1 mai 2026 à 05:44 UTCmonitoring

   Our replacement switch is in service and all affected services should be restored at this time. If you have any further issues, please contact Teraswitch Support at [email protected].

4. 1 mai 2026 à 17:30 UTCresolved

   This incident has been resolved.

1. 26 mars 2026 à 22:08 UTCidentified

   Teraswitch is working to resolve a simultaneous backbone fiber cut affecting AMS1 connectivity.

2. 26 mars 2026 à 22:53 UTCidentified

   We are working on a resolution to a multiple diverse fiber cut that is impacting AMS1 connectivity. The issue has been identified by the fiber vendor, and we are working on an alternative fiber solution to restore connectivity.

3. 27 mars 2026 à 00:14 UTCmonitoring

   We have implemented a temporary fix and AMS1 connectivity is now restored. We will follow up with a permanent fix and RCA.

4. 2 avril 2026 à 22:38 UTCresolved

   An RCA has been posted for this issue.

5. 2 avril 2026 à 22:38 UTCpostmortem

   # Root Cause Analysis: AMS1 Connectivity Outage **Incident Date:** March 26, 2025 **Duration:** ~1h 21m \(22:08 UTC – 00:14 UTC \+1\) **Severity:** Critical – Full site connectivity loss **Affected Site:** AMS1 – Amsterdam, Netherlands **Status:** Resolved ## Executive Summary On March 26, 2025, Teraswitch's AMS1 facility experienced a complete loss of backbone connectivity lasting approximately 1 hour and 21 minutes. The outage was caused by a scheduled fiber vendor maintenance window that simultaneously impacted both the primary and what was believed to be a diverse redundant fiber path between AMS1 and the rest of the backbone. Investigation revealed that due to a documentation and handoff error at the fiber vendor dating back over a year, the AMS1–AMS2 fiber span had never been migrated to the intended diverse path as part of the AMS3 ring buildout. As a result, both affected spans shared physical infrastructure, eliminating the redundancy intended to protect against exactly this type of event. Connectivity was restored within the maintenance window after a rapid joint audit with the fiber vendor confirmed the provisioning discrepancy, and the AMS1–AMS2 span was moved to its correct diverse path. ## Background Teraswitch's Amsterdam backbone originally consisted of a single dark fiber span between AMS1 and AMS2. When AMS3 was later brought online, the network design called for a three-node fiber ring with fully diverse physical paths between all sites to provide redundant backbone connectivity. To support this design, the fiber vendor was engaged to: 1. Provision a new AMS1–AMS3 span 2. Provision a new AMS2–AMS3 span 3. Reroute the existing AMS1–AMS2 span onto a physically diverse path Due to an internal handoff and documentation error within the fiber vendor, step 3 was not completed. The AMS1–AMS2 span remained on its original physical route. Teraswitch was not made aware of this omission, and the span continued to operate normally for over a year. Because it carried live traffic and appeared correctly in topology, it was not identified as incorrectly provisioned during subsequent audits. ## Timeline of Events | Time \(UTC\) | Event | | --- | --- | | Prior to March 26 | Fiber vendor schedules routine maintenance affecting Amsterdam infrastructure | | ~22:08 | AMS1 backbone connectivity lost. Both AMS1–AMS2 and AMS1–AMS3 paths go down simultaneously. Teraswitch NOC begins triage. | | ~22:53 | Fiber vendor engaged and confirms maintenance is impacting both spans. Root cause identified as shared physical infrastructure due to the original provisioning error. | | ~00:14 \+1 | Fiber vendor migrates AMS1–AMS2 span to the correct diverse physical path. Connectivity restored. Monitoring confirmed stable. | ## Root Cause **Primary cause:** An internal documentation and handoff failure at the fiber vendor resulted in the AMS1–AMS2 dark fiber span never being migrated to its intended physically diverse route during the AMS3 ring buildout. Both the AMS1–AMS2 and AMS1–AMS3 spans shared common physical infrastructure, making the designed ring topology's redundancy ineffective. **Contributing factor:** Because the span was operationally active and traffic was flowing normally, the provisioning error went undetected across both Teraswitch and vendor records for over a year. When the scheduled maintenance affected the shared physical infrastructure, both paths were impacted simultaneously, leaving AMS1 with no available backbone connectivity. ## Impact * **AMS1 customers** experienced a complete loss of inbound and outbound connectivity for approximately 1 hour and 21 minutes. * No data loss or hardware damage occurred. * All other Teraswitch sites were unaffected. ## Resolution Working jointly with the fiber vendor during the incident, Teraswitch engineers and vendor technicians audited the physical path assignments for all Amsterdam spans. The discrepancy between the intended and actual routing of the AMS1–AMS2 span was identified. The vendor migrated the span to the correct physically diverse path, restoring independent redundant connectivity across the AMS1–AMS2–AMS3 ring as originally designed. ## Corrective Actions | Action | Owner | Status | | --- | --- | --- | | Confirm and document physical path diversity for all three Amsterdam spans with fiber vendor | Teraswitch / Fiber Vendor | Complete | | Obtain updated as-built fiber records from vendor reflecting correct path assignments | Fiber Vendor | Complete | | Audit all other Teraswitch sites for similar provisioning discrepancies against vendor records | Teraswitch | Complete | | Establish a fiber path verification checklist for all future vendor provisioning work prior to accepting new spans | Teraswitch | Planned | | Add physical diversity validation to change management process for any future ring or redundancy buildouts | Teraswitch | Planned | | Incorporate optical span latency validation against fiber path build sheets as an acceptance criterion for new span provisioning | Teraswitch | Planned | ## Lessons Learned * **Operational traffic is not proof of correct provisioning.** A span can carry live traffic for an extended period while still being routed incorrectly relative to its intended physical diversity design. * **Redundancy assumptions must be periodically verified against vendor as-built records**, not solely inferred from operational status. * **Fiber vendor handoffs require explicit acceptance criteria** including documented physical path confirmation before provisioning work is considered complete. * **Span latency is a low-cost signal for path verification.** In post-incident review, Teraswitch noted that the measured propagation latency on the AMS1–AMS2 span was slightly lower than expected based on the fiber path build sheet for the intended diverse route. This discrepancy, while subtle, was consistent with the span still traversing the shorter original path. Validating measured latency against estimated values from build sheets at provisioning acceptance could have surfaced this error significantly earlier. This check will be incorporated into the span acceptance process going forward. _RCA prepared by Teraswitch Network Engineering. For questions contact the NOC or network architecture team._

1. 19 mars 2026 à 21:43 UTCmonitoring

   Teraswitch is investigating reports of network connectivity issues affecting some services at our EWR2 site. We have identified the likely cause and implemented a fix, and are monitoring to verify that the issue is resolved.

2. 25 mars 2026 à 22:00 UTCresolved

   This incident has been resolved.

1. 17 mars 2026 à 20:05 UTCinvestigating

   Teraswitch is investigating reports of select markets having issues connecting to PIT1.

2. 17 mars 2026 à 20:36 UTCidentified

   We have located a network device with software issues, our team is about to reload the device. This issue affects mostly colocation customers at PIT1, but traffic from various sources may have passed through this device depending on the direction of network flow.

3. 17 mars 2026 à 20:43 UTCmonitoring

   Reload is complete and the network operation appears to have returned to normal. We will follow up this notification with a RCA and repair plan.

4. 25 mars 2026 à 22:03 UTCresolved

   This incident has been resolved.

1. 20 février 2026 à 07:47 UTCinvestigating

   Teraswitch is investigating a loss of the majority of VAN1 internet connectivity - possibly related to issues in Seattle.

2. 20 février 2026 à 11:32 UTCresolved

   Cogent has recovered as of 4:33am Eastern. Services should be normalized at this time.

1. 20 février 2026 à 07:01 UTCidentified

   TeraSwitch is aware and tracking a loss of two paths from Seattle to Tokyo. This has caused much of the internet traffic to divert via public Internet Transit.

2. 24 février 2026 à 15:44 UTCresolved

   This incident has been resolved.

1. 19 février 2026 à 02:08 UTCidentified

   We have reached out to the vendor asking for a update. Latency from Singapore to Europe will be elevated until the outage is resolved.

2. 9 mars 2026 à 02:08 UTCresolved

   We have successfully brought online a second diverse sub-sea cable path and latency has returned to normal. Once the original severed cable comes back online, we will now have full path redundancy.

1. 4 février 2026 à 14:26 UTCidentified

   Teraswitch has identified thre core backbone links that are down within the central US. This is causing US-East traffic to wrap around the world to get to US-West. Chicago/Seattle Chicago/Salt Lake City Dallas /Los Angeles We are working with our fiber vendors for a ETR.

2. 4 février 2026 à 16:04 UTCmonitoring

   Chicago to Salt Lake City path is restored which has normalized operations with 1 of 3 links restored. We will close this status when 2 of 3 are restored which also restores redundancy.

3. 9 février 2026 à 16:46 UTCresolved

   This incident has been resolved.

1. 3 février 2026 à 19:54 UTCidentified

   Teraswitch is investigating reports that our console (console.tsw.io) is currently inaccessible. This issue has been identified as the result of an outage in an underlying cloud provider. We will provide an update as more information becomes available.

2. 3 février 2026 à 21:16 UTCidentified

   This issue is due to an ongoing Cloudflare outage: https://www.cloudflarestatus.com/incidents/m1xvmqf37z97. Cloudflare has implemented a fix and we are monitoring for recovery. Our API is unaffected by this issue and remains accessible.

3. 3 février 2026 à 22:37 UTCresolved

   Cloudflare has resolved the issue, and console.tsw.io is accessible once again as of 22:28 UTC. This incident has been resolved.

1. 30 janvier 2026 à 09:53 UTCinvestigating

   We are investing significant packet loss on the west coast US.

2. 30 janvier 2026 à 10:06 UTCinvestigating

   We are continuing to investigate this issue.

3. 30 janvier 2026 à 10:30 UTCresolved

   This incident has been resolved.

1. 28 janvier 2026 à 14:17 UTCresolved

   At 2PM UTC, Teraswitch reloaded software related to our route servers to fix "stuck" transport tunneling sessions. This may have caused interruptions to ULL/HFT and transport services over our network. We believe the first stuck session started approximately 6 hours before, with more dropping off over the next few hours. As of this moment, all stuck sessions appear to be resolved, and traffic is passing normally and via ULL links.