Historique des pannes de Google Maps Platform

1. 1 septembre 2026 à 16:03 UTCresolved

   ** Résumé** Plusieurs produits dans nous-central1-b connaissent une dégradation du service réseau. **Description** Nous vivons un problème de réseautage à partir de mardi, 2026-09-01 07:41 US/Pacific. La question touche une partie de la zone nous-central1-b. Les zones restantes de la région ne sont pas affectées. Notre équipe d'ingénierie réseau a mis en place des mesures d'atténuation et nous voyons des services commencer à se rétablir. Nous fournirons plus d'information d'ici mardi, 2026-09-01 09:30 États-Unis/Pacifique avec les détails actuels. Nous excusons tous ceux qui sont touchés par la perturbation. **Symptômes du client** Les clients subissent une perte de paquets et des erreurs élevées. * BigQuery * Flux de données en nuage * Nuage SQL * Moteur App * Exécution en nuage * AlloyDB * Cloud Bigtable * Nuageur * Moteur de calcul * VPN en nuage * Connexion Cloud * Nuage NAT * Cloud Filestore * Connectivité hybride Cloud * Nuage privé virtuel * Moteur Google Kubernetes * Looker (noyau Google Cloud) **Travaux complémentaires** * Les clients avec des relevés de couche de service peuvent voir des réalisations réussies car seulement une partie de la zone est affectée. * Les clients qui sont gravement touchés et qui ont des services disponibles dans d'autres zones peuvent échouer vers d'autres zones.

2. 1 septembre 2026 à 16:19 UTCresolved

   ** Résumé** Plusieurs produits dans nous-central1-b connaissent une dégradation du service réseau. **Description** La question touche une partie de la zone nous-central1-b. Les zones restantes de la région ne sont pas affectées. À la suite des efforts d'atténuation au niveau du réseau, l'infrastructure de réseau sous-jacente s'est complètement rétablie. Les équipes d'ingénierie de produits de Cloud valident actuellement le statut de récupération des différents services. Nous fournirons plus d'information d'ici mardi, 2026-09-01 10h00 États-Unis/Pacifique avec des détails à jour. **Symptômes du client** Les clients subissent une perte de paquets et des erreurs élevées. * BigQuery * Flux de données en nuage * Nuage SQL * Moteur App * Exécution en nuage * AlloyDB * Cloud Bigtable * Nuageur * Moteur de calcul * VPN en nuage * Connexion Cloud * Nuage NAT * Cloud Filestore * Connectivité hybride Cloud * Nuage privé virtuel * Moteur Google Kubernetes * Looker (noyau Google Cloud) * Apigee * Google SecOps SOAR **Travaux complémentaires** * Les clients avec des relevés de couche de service peuvent voir des réalisations réussies car seulement une partie de la zone est affectée. * Les clients qui sont gravement touchés et qui ont des services disponibles dans d'autres zones peuvent échouer vers d'autres zones.

3. 1 septembre 2026 à 17:09 UTCresolved

   ** Résumé** Plusieurs produits dans nous-central1-b connaissent une dégradation du service réseau. **Description** La question touche une partie de la zone nous-central1-b. Les zones restantes de la région ne sont pas affectées. La plupart des produits ont été récupérés et l'équipe d'ingénierie valide encore quelques impacts restants. Nous fournirons plus d'information d'ici mardi, 2026-09-01 11:00 US/Pacific avec les détails actuels. **Symptômes du client** Les clients subissent une perte de paquets et des erreurs élevées. * Exécution en nuage * Moteur App Après récupération des produits * BigQuery * Flux de données en nuage * Cloud Bigtable * Moteur de calcul * Connectivité hybride Cloud * Cloud Filestore * Nuage privé virtuel * Nuageur * Moteur Google Kubernetes * Nuage SQL * VPN AlloyDBCloud * Apigee * Connexion Cloud * Nuage NAT * Looker (noyau Google Cloud) * Google SecOps SOAR * Magasin d'incendie en nuage **Travaux complémentaires** * Les clients avec des relevés de couche de service peuvent voir des réalisations réussies car seulement une partie de la zone est affectée. * Les clients qui sont gravement touchés et qui ont des services disponibles dans d'autres zones peuvent échouer vers d'autres zones.

4. 1 septembre 2026 à 18:01 UTCresolved

   ** Résumé** Plusieurs produits dans nous-central1-b connaissent une dégradation du service réseau. **Description** La question touche une partie de la zone nous-central1-b. Les zones restantes de la région ne sont pas affectées. La plupart des produits ont été récupérés et l'équipe d'ingénierie valide encore quelques impacts restants. La maintenance courante du réseau a déclenché des problèmes dans nous-central1-b. Nous avons récupéré la capacité du réseau qui a été touchée et rétabli tous les services zonaux et régionaux. L'entretien dans la région a été interrompu pendant que des audits proactifs étaient effectués. Nous fournirons plus d'information d'ici mardi, 2026-09-01 12:30 États-Unis/Pacifique avec les détails actuels. **Symptômes du client** Les clients subissent une perte de paquets et des erreurs élevées. * Exécution en nuage * Moteur App Après récupération des produits * BigQuery * Flux de données en nuage * Cloud Bigtable * Moteur de calcul * Connectivité hybride Cloud * Cloud Filestore * Nuage privé virtuel * Nuageur * Moteur Google Kubernetes * Nuage SQL * VPN AlloyDBCloud * Apigee * Connexion Cloud * Nuage NAT * Looker (noyau Google Cloud) * Google SecOps SOAR * Magasin d'incendie en nuage **Travaux complémentaires** * Les clients avec des relevés de couche de service peuvent voir des réalisations réussies car seulement une partie de la zone est affectée.

5. 1 septembre 2026 à 18:36 UTCresolved

   ** Résumé** Plusieurs produits dans nous-central1-b connaissent une dégradation du service réseau. **Description** La question touche une partie de la zone nous-central1-b. Les zones restantes de la région ne sont pas affectées. La plupart des produits ont été récupérés. Les produits impactés Cloud Run et App Engine voient également la récupération et les erreurs ont diminué les niveaux de pré-incident. Le génie continue de surveiller et de valider. L'entretien régulier du chemin de tissu du réseau a déclenché des problèmes inattendus dans deux grappes dans nous-central1-b. Nous avons récupéré la capacité du réseau qui a été touchée et rétabli tous les services zonaux et régionaux. L'entretien dans la région a été interrompu pendant que des audits proactifs étaient effectués. Nous fournirons plus d'information d'ici mardi, 2026-09-01 12:30 États-Unis/Pacifique avec les détails actuels. **Symptômes du client** Les clients subissent une perte de paquets et des erreurs élevées. * Exécution en nuage * Moteur App Après récupération des produits * BigQuery * Flux de données en nuage * Cloud Bigtable * Moteur de calcul * Connectivité hybride Cloud * Cloud Filestore * Nuage privé virtuel * Nuageur * Moteur Google Kubernetes * Nuage SQL * VPN AlloyDBCloud * Apigee * Connexion Cloud * Nuage NAT * Looker (noyau Google Cloud) * Google SecOps SOAR * Magasin d'incendie en nuage **Travaux complémentaires** * Les clients avec des relevés de couche de service peuvent voir des réalisations réussies car seulement une partie de la zone est affectée.

6. 1 septembre 2026 à 19:12 UTCresolved

   ** Résumé** Plusieurs produits dans nous-central1-b dégradation du service réseau expérimenté. **Description** La question touchait une partie de la zone nous-central1-b. Les autres zones de la région n'ont pas été touchées. Tous les produits touchés ont récupéré. D'après l'analyse préliminaire, l'entretien régulier du réseau a déclenché des problèmes inattendus dans un cluster dans notre centre1-b. Nous avons récupéré la capacité du réseau qui a été touchée et rétabli tous les services zonaux et régionaux. L'entretien dans la région a été interrompu pendant que des audits proactifs étaient effectués. Nous vous remercions de votre patience pendant que nous avons travaillé à résoudre le problème. **Symptômes du client** Les clients ont connu une perte de paquets élevée et des erreurs. **Travaux complémentaires** La question a été atténuée.

7. 3 septembre 2026 à 17:47 UTCresolved

   ## Rapport préliminaire d'incident Nous nous excusons sincèrement pour la perturbation de cet incident causé à votre entreprise. Nous savons combien vous comptez sur Google Cloud, et nous regrettons l'impact sur votre environnement. Nous nous efforçons d'aborder la cause fondamentale et d'empêcher que cela se produise à l'avenir. Veuillez noter que ces renseignements sont fondés sur nos meilleures connaissances au moment de l'affichage et peuvent être modifiés au fur et à mesure que notre enquête se poursuit. Un rapport final d'incident comportant des mesures préventives sera affiché une fois notre enquête terminée. Si vous avez eu un impact en dehors de ce qui est listé ci-dessous, veuillez contacter Google Cloud Support en utilisant https://cloud.google.com/support ## Date/heure de l'émission (tous les jours aux États-Unis/Pacifique) **Début de l'incident**: 1er septembre 2026 07:41 **Fin de l'incident**: 1er septembre 2026 11:52 **Durée**: 4 heures, 11 minutes Résumé Le mardi 1er septembre 2026, certains clients de notre centre1-b ont connu une dégradation du service réseau et un isolement par exemple pour une durée de 4 heures et 11 minutes. Nous excusons sincèrement nos clients dont les entreprises ont été touchées par cette perturbation. Ce n'est pas le niveau de qualité et de fiabilité que nous nous efforçons de vous offrir, et nous prenons des mesures immédiates pour améliorer les performances et la disponibilité de la plateforme. Cause racine préliminaire Le déclencheur technique immédiat pour cet événement a été la déconnexion physique par inadvertance des câbles fibre optique du réseau lors d'une procédure de maintenance matérielle courante. Un ingénieur effectuait une mise à niveau de capacité programmée sur des routeurs de datacenter qui supportent une fraction de capacité dans la zone nous-central1-b. L'architecture réseau de chaque datacenter est conçue avec redondance sur plusieurs appareils de routage. Le système est conçu pour résister à toutes les défaillances d'un seul appareil ou d'un chemin de fibres, et la plupart des doubles ou triples défauts n'affectent pas le trafic des clients. Pour cela, les appareils et les voies fibreuses sont physiquement séparés dans chaque datacenter, avec diverses sources d'énergie. Une erreur de procédure signifiait que l'action de maintenance physique débranchait séquentiellement 100% des chemins de fibres sur tous les appareils en 13 minutes. La nature de l'erreur, combinée à la vitesse de l'action, a empêché les avertissements d'action incorrecte d'atteindre l'ingénieur avant la déconnexion complète. La capacité de calcul de la zone touchée a donc été isolée du réseau. Les clients étaient incapables d'atteindre leurs machines virtuelles, et ces machines virtuelles ne pouvaient pas établir de connexions en dehors de leur zone. Réparation Le problème a été détecté immédiatement par des systèmes automatisés de surveillance des pertes du réseau ainsi que par des sondes proactives, qui ont rapidement mobilisé les équipes d'ingénierie du réseau et d'intervention en cas d'incident. Afin d'atténuer l'impact immédiat sur la clientèle, les équipes d'ingénierie ont déplacé activement le trafic de l'infrastructure touchée et ont automatiquement déplacé le trafic de charge de travail compatible vers une capacité saine ailleurs dans la région. Parallèlement, les techniciens d'exploitation du matériel sur le site ont identifié les liaisons optiques déconnectées et physiquement remis en place les fibres. Une fois que les liaisons physiques ont été entièrement rétablies, les débits de trafic ont été normalisés et le trafic a été réorienté pour rendre la capacité de service. Description de l'impact Mardi 1er septembre, de 07:41 à 11:52 États-Unis/Pacifique, une partie de la zone us-central1-b a connu une grave dégradation du réseau et l'isolement des ressources. * **Produits multiples touchés :** La perturbation du réseau d'une partie de nous-central1-b a touché plusieurs produits GCP pour certains clients de cette zone. * **Taux d'erreur :** Les taux de baisse du trafic pour les ressources hébergées dans la zone touchée ont atteint 100 % pendant le pic de l'incident, ce qui a entraîné une perte de paquets et de machines virtuelles inaccessibles. **Services et caractéristiques touchés** * ** Moteur de calcul Google :** L'impossibilité d'accéder aux machines virtuelles à l'extérieur et l'incapacité pour les VM d'atteindre des ressources distantes. * **Google Kubernetes Moteur:** Clusters et nœuds régionaux/zonaux inaccessibles dans la zone touchée, ce qui a retardé les périodes de rétablissement des grappes. * **Clou Exécuter / Google App Engine:** Les pics de latence temporaires et les files d'attente s'avortent à mesure que les charges de travail du moteur sont automatiquement évacuées et transférées vers une capacité saine. Un sous-ensemble de charges de travail dans la zone spécifiquement touchée a subi une dégradation jusqu'à 11:52 États-Unis/Pacifique. * **Cloud SQL, AlloyDB pour PostgreSQL, Cloud Slanner, Google Cloud Bigtable, Cloud Filestore:** L'accès aux données et la connectivité ont été coupés pour des cas localisés strictement à l'infrastructure touchée. * **Virtual Private Cloud (VPC), Cloud NAT, Apigee, Google BigQuery, Google Cloud Dataflow, Looker, Google SecOps SOAR, Connectivité hybride, Cloud Interconnect:** Perte de paquets réseau élevée, baisses de connectivité et délais d'API dans la zone touchée.

Traduit automatiquement depuis la mise à jour officielle de l'incident.

1. 20 août 2026 à 16:44 UTCresolved

   ** Résumé** Nous enquêtons sur un problème où les clients peuvent faire l'objet de délais, de dégradations de service, d'erreurs et de latences élevées sur plusieurs produits dans la région de nous-Ouest1. **Description** Nous avons un problème avec plusieurs produits, à partir de jeudi, 2026-08-20 08:40 PDT. Notre équipe d'ingénieurs continue d'étudier la question. Nous fournirons une mise à jour d'ici jeudi, 2026-08-20 10:30 PDT avec les détails. Nous excusons tous ceux qui sont touchés par la perturbation. **Diagnostic / Symptômes du client** Les clients de nous-west1 peuvent éprouver des délais, des dégradations de service, des erreurs et des latences élevées pour plusieurs produits. **Travaux complémentaires** Aucune pour l'instant.

2. 20 août 2026 à 17:13 UTCresolved

   ** Résumé** Nous enquêtons sur un problème où les clients peuvent faire l'objet de délais, de dégradations de service, d'erreurs et de latences élevées sur plusieurs produits dans la région de nous-Ouest1. **Description** Des mesures d'atténuation sont mises en oeuvre par nos équipes d'ingénierie, des tendances de rétablissement ont été observées entre les couches d'infrastructure. Des efforts actifs sont en cours pour ramener les services cloud touchés à leur pleine exploitation. Nous fournirons une mise à jour d'ici jeudi, 2026-08-20 10:45 PDT avec les détails. **Diagnostic / Symptômes du client** Les clients de nous-west1 peuvent éprouver des délais, des dégradations de service, des erreurs et des latences élevées pour plusieurs produits. **Travaux complémentaires** Nous recommandons aux clients de faire faillite dans d'autres régions lorsque cela est possible.

3. 20 août 2026 à 17:32 UTCresolved

   ** Résumé** Nous enquêtons sur un problème où les clients peuvent faire l'objet de délais, de dégradations de service, d'erreurs et de latences élevées sur plusieurs produits dans la région de nous-Ouest1. **Description** Des mesures d'atténuation sont mises en oeuvre par nos équipes d'ingénierie, des tendances de rétablissement ont été observées entre les couches d'infrastructure. Des efforts actifs sont en cours pour ramener les services cloud touchés à leur pleine exploitation. Nous fournirons une mise à jour d'ici jeudi, 2026-08-20 11:00 PDT avec les détails. **Diagnostic / Symptômes du client** Les clients de nous-west1 peuvent éprouver des délais, des dégradations de service, des erreurs et des latences élevées pour plusieurs produits. **Travaux complémentaires** Nous recommandons aux clients de faire faillite dans d'autres régions lorsque cela est possible.

4. 20 août 2026 à 18:03 UTCresolved

   ** Résumé** La question des délais, des dégradations, des erreurs et des latences pour plusieurs produits us-west1 a été atténuée et nous travaillons à récupérer tous les produits. **Description** Les mesures d'atténuation ont été menées à bien par nos équipes d'ingénierie et nous assistons à la récupération de plusieurs produits. Nous continuons à travailler pour récupérer les produits restants. Nous fournirons une mise à jour d'ici jeudi, 2026-08-20 11:45 PDT avec les détails. **Diagnostic / Symptômes du client** Les clients de nous-west1 peuvent éprouver des délais, des dégradations de service, des erreurs et des latences élevées pour plusieurs produits. **Travaux complémentaires** Pas de solution de rechange pour le moment.

5. 20 août 2026 à 18:53 UTCresolved

   ** Résumé** La question des délais, des dégradations, des erreurs et des latences pour plusieurs produits us-west1 a été atténuée et nous travaillons à récupérer tous les produits. **Description** Les mesures d'atténuation ont été menées à bien par nos équipes d'ingénierie et nous assistons à la récupération de plusieurs produits. Nous continuons à travailler pour récupérer les produits restants. Nous fournirons une mise à jour d'ici jeudi, 2026-08-20 12:30 PDT avec les détails. **Diagnostic / Symptômes du client** Les clients de nous-west1 peuvent éprouver des délais, des dégradations de service, des erreurs et des latences élevées pour plusieurs produits. **Travaux complémentaires** Pas de solution de rechange pour le moment.

6. 20 août 2026 à 19:21 UTCresolved

   ** Résumé** La question des délais, des dégradations, des erreurs et des latences pour plusieurs produits us-west1 a été atténuée et nous travaillons à récupérer tous les produits. **Description** Les mesures d'atténuation ont été menées à bien par nos équipes d'ingénierie et nous assistons à la récupération de plusieurs produits. Nous continuons à travailler pour récupérer les produits restants. Nous fournirons une mise à jour d'ici jeudi, 2026-08-20 13:30 PDT avec les détails. **Diagnostic / Symptômes du client** Les clients de nous-west1 peuvent éprouver des délais, des dégradations de service, des erreurs et des latences élevées pour plusieurs produits. **Travaux complémentaires** Pas de solution de rechange pour le moment.

7. 20 août 2026 à 19:37 UTCresolved

   ** Résumé** La question des délais, des dégradations, des erreurs et des latences pour plusieurs produits us-west1 a été atténuée. **Description** Nous avons atténué l'impact de plusieurs produits dans notre région de l'ouest1 à partir de jeudi, 2026-08-20 10:22 PDT. Nos équipes d'ingénierie ont rétabli la capacité d'un entretien optique planifié qui a causé une congestion inattendue à Dalles, dans la région de l'Oregon métro /us-west1. Nos systèmes se sont stabilisés et les services ont récupéré une fois la capacité rétablie. Nos équipes continuent de surveiller tout impact résiduel. Nous publierons une analyse de cet incident une fois notre enquête interne terminée. Nous vous remercions de votre patience pendant que nous avons travaillé à résoudre le problème. **Diagnostic / Symptômes du client** Les clients de nous-west1 ont peut-être connu des temps morts, des dégradations de service, des erreurs et des latences élevées pour plusieurs produits. **Travaux complémentaires** Cette question est maintenant atténuée.

8. 25 août 2026 à 06:20 UTCresolved

   # Rapport préliminaire d'incident Nous nous excusons sincèrement pour la perturbation de cet incident causé à votre entreprise. Reconnaissant votre confiance dans Google Cloud, nous exprimons nos sincères regrets pour tout impact opérationnel vécu. Nos équipes d'ingénieurs s'attaquent activement à la cause fondamentale sous-jacente pour prévenir les récidives futures. Veuillez noter que les renseignements fournis dans le présent document reflètent notre compréhension actuelle au moment de la publication et demeurent sujets à révision à mesure que l'enquête progresse. Un rapport d'incident détaillé sur les mesures préventives sera publié à la fin de notre enquête. Si vous avez eu un impact en dehors de ce qui est listé ci-dessous, veuillez contacter Google Cloud Support en utilisant https://cloud.google.com/support. ## Date/heure de l'émission (tous les jours aux États-Unis/Pacifique) * Début de l'incident : 20 août 2026 08:00 * Fin de l'incident : 20 août 2026 10:22 * Durée: 2 heures, 22 minutes Résumé Le jeudi 20 août 2026, plusieurs services Google Cloud ont rencontré des latences élevées, des défaillances de provisionnement, des taux d'erreur accrus et des dégradations de service d'une durée de 2 heures et 22 minutes. Cause racine préliminaire La perturbation est survenue lors de la maintenance programmée de la fibre optique, ce qui a compromis de façon inattendue la capacité du réseau entre les centres de données de la région de l'ouest1. Les mécanismes automatisés de réacheminement n'ont pas réussi à redistribuer correctement le trafic à d'autres capacités, ce qui a entraîné une congestion du réseau, les volumes dépassant la bande passante disponible dans la zone touchée. Cette dégradation sous-jacente du réseau a par la suite eu des répercussions sur les composantes de service de plus haut niveau par l'effritement de la demande, l'augmentation de la latence et l'encaissement des relevés. Par conséquent, les opérations d'avion de contrôle et les requêtes d'avion de données n'ont pas réussi à exécuter avec succès sur plusieurs services Google Cloud dépendants, produisant la latence observée et des taux d'erreur élevés. Réparation La surveillance interne a alerté les ingénieurs de Google à l'incident, confirmant que plusieurs services à travers nous-ouest1 ont été touchés. Afin d'atténuer l'impact opérationnel immédiat, les ingénieurs ont mis en place des protocoles de drainage du trafic et de réacheminement de la charge de travail du service en dehors de l'infrastructure du réseau dégradé. Après la restauration de la capacité du réseau intercampus et la résolution du problème central, les ingénieurs ont systématiquement rétabli le trafic vers la région de l'US-West1 et confirmé la normalisation du service. Des solutions d'ingénierie à long terme sont actuellement mises au point et assignées pour s'attaquer à la cause fondamentale et renforcer la résilience du système en ce qui concerne les procédures d'entretien des fibres. Description de l'impact Le jeudi 20 août, entre 08:00 et 10:22 US/Pacific, les clients de Google Cloud dans la région us-west1 ont rencontré des latences élevées, des défaillances de provisionnement et des taux d'erreur accrus. * Taux d'impact / d'erreur proportionnels : L'étendue précise de l'impact, y compris la proportion de projets et d'applications touchés, et les paramètres d'erreur détaillés seront publiés dans l'analyse complète des causes profondes, car la capacité réduite causée par le throttling des demandes, les pics de latence et l'augmentation du volume de rétry. * Étendue Exclusion : Les services et les charges de travail des régions autres que nous-Ouest1 sont demeurés pleinement opérationnels et n'ont pas été touchés. **Services et caractéristiques touchés** Les services suivants ont connu des retards et/ou des taux d'erreur élevés dans leurs plans de données et plans de contrôle respectifs: * AlloyDB * Apache Kafka * Apigee Edge Nuage public * Apigee X * Registre des artéfacts * Service de transfert de données BigQuery & BigQuery * Construction de nuages * Fusion de données en nuage * Flux de données en nuage * Cloud Filestore * Service de gestion des clés en nuage (KMS) * Surveillance des nuages * Exécution en nuage * Nuage SQL * Centre de contact Plateforme AI * Métastore Dataproc * Moteur Google App * Google Cloud Bigtable * Google Cloud Pub/Sub * Stockage Google Cloud (GCS) * Moteur de calcul Google (GCE) * Moteur Google Kubernetes (GKE) * Gestion de l'identité et de l'accès (IAM) * Débit d'air géré (compositeur de nuages) * Service géré pour Apache Spark (Dataproc) Disque persistant

9. 27 août 2026 à 21:45 UTCresolved

   # Rapport d'incident Résumé Le jeudi 20 août 2026, de 08:00 à 10:22 US/Pacific (15:00 à 17:22 UTC), plusieurs services Google Cloud dans la région nous-ouest1 ont connu une latence élevée, des défaillances de provisionnement, des taux d'erreur accrus et des dégradations généralisées du service pour une durée totale de 2 heures et 22 minutes. L'incident a touché une large gamme de services de base, y compris le disque persistant, le moteur Google Kubernetes, le moteur Google Compute, Cloud Run, Google Cloud Bigtable, le stockage Cloud et la gestion de l'identité et de l'accès, à la fois pour les opérations d'avion de contrôle et les demandes d'avion de données. Nous nous excusons sincèrement de la perturbation que cet incident a causée à vos activités et à votre charge de travail critique. Nous reconnaissons le rôle vital que Google Cloud joue dans le soutien de votre organisation, et nous regrettons profondément l'impact sur vos activités et vos charges de travail critiques. Les équipes d'ingénierie et d'infrastructure mettent activement en œuvre des mesures visant à s'attaquer aux causes profondes et à renforcer la résilience des réseaux afin de prévenir les récidives à l'avenir. Plus précisément, les équipes surveillent les progrès de récupération, perfectionnent les contrôles de sécurité pour l'entretien optique prévu et optimisent les mécanismes régionaux d'acheminement du trafic afin de prévenir les contraintes de capacité imprévues. La cause profonde La perturbation est survenue lors de la maintenance programmée de la fibre optique, ce qui a compromis de façon inattendue la capacité du réseau entre les centres de données de la région de l'ouest1. Les mécanismes automatisés de réacheminement n'ont pas réussi à redistribuer correctement le trafic à d'autres capacités, ce qui a entraîné une congestion du réseau, les volumes dépassant la bande passante disponible dans la zone touchée. Cette dégradation du réseau sous-jacente a par la suite eu des répercussions sur les composants de service de plus haut niveau en raison de la perte grave de paquets, de la demande de grottling et de la latence accrue entre les dépendances intercampus. Les services d'infrastructure de base, y compris le consensus de Spanner Paxos et le Serveur de métadonnées unifiées (UMS), ont connu d'importants pics de latence, qui se sont accumulés dans des délais et des taux d'erreur élevés pour les produits dépendants en aval tels que Cloud Storage, Cloud IAM, Disk persistant et Google Kubernetes Engine. Par conséquent, les opérations d'avion de contrôle et les demandes d'avion de données n'ont pas réussi à exécuter plusieurs services Google Cloud dans l'ouest1 pendant toute la durée de l'incident. La réhabilitation et la prévention Les systèmes de surveillance interne ont d'abord détecté des anomalies de service généralisées et a alerté les ingénieurs de Google, qui ont rapidement confirmé que plusieurs services de base opérant dans la région de nous-Ouest1 étaient gravement touchés. En réponse aux risques opérationnels immédiats, les équipes d'ingénierie ont rapidement exécuté des protocoles d'évacuation de la circulation d'urgence afin de réacheminer la charge de travail active de l'infrastructure du réseau intercampus compromise et de réduire au minimum la perturbation de la clientèle. Une fois que les équipes ont rétabli la capacité du réseau de fibres intercampus, les ingénieurs ont réintroduit systématiquement le trafic de production vers nous-ouest1 par des étapes de validation contrôlées, confirmant finalement la normalisation complète du service sur toutes les plateformes touchées. Les travaux d'assainissement et d'amélioration architecturale à plus long terme sont en cours de finalisation et sont confiés à des équipes propriétaires. Les principaux domaines d'intervention comprennent la réforme des protocoles d'entretien programmés, l'établissement de contrôles de sécurité plus stricts et des exigences de redondance des circuits avant l'entretien courant, et l'affinage des mécanismes automatisés de déroutement du trafic régional pour gérer automatiquement la dégradation soudaine de la capacité sans entraîner de grave congestion du réseau. Description détaillée de l'impact Le jeudi 20 août, entre 08:00 et 10:22 US/Pacific, les clients de Google Cloud dans la région us-west1 ont rencontré des latences élevées, des défaillances de provisionnement et des taux d'erreur accrus. * ** Taux d'incidence / d'erreur :** La réduction de la capacité a causé des étranglements de la demande, des pics de latence et une augmentation du volume de rétry. * **Exclusion de la portée :** Les services et la charge de travail dans les régions autres que nous-Ouest1 sont restés pleinement opérationnels et n'ont pas été touchés. **Services et caractéristiques touchés** Les services suivants ont connu des retards et/ou des taux d'erreur élevés dans leurs plans de données et plans de contrôle respectifs: * AlloyDB * Apache Kafka * Apigee Edge Nuage public * Apigee X * Registre des artéfacts * Service de transfert de données BigQuery & BigQuery * Construction de nuages * Fusion de données en nuage * Flux de données en nuage * Cloud Filestore * Service de gestion des clés en nuage (KMS) * Surveillance des nuages * Exécution en nuage * Nuage SQL * Centre de contact Plateforme AI * Métastore Dataproc * Moteur Google App * Google Cloud Bigtable * Google Cloud Pub/Sub * Stockage Google Cloud (GCS) * Moteur de calcul Google (GCE) * Moteur Google Kubernetes (GKE) * Gestion de l'identité et de l'accès (IAM) * Débit d'air géré (compositeur de nuages) * Service géré pour Apache Spark (Dataproc) * Disque persistant L'incident régional en nous-Ouest1 a suivi une récupération structurée en trois phases dictée par des couches de dépendance à la plate-forme : Alors que la connectivité de la plate-forme centrale et le service de demande en direct ont été restaurés par 10:22 États-Unis/Pacifique, certains services ont pris plus de temps pour récupérer complètement en raison du traitement asynchrone de l'arriéré et du rapprochement d'état de contrôle-plan localisé pour un très petit ensemble de clients. Pour les services liés aux événements et aux pipelines, les taux d'erreur entrants sont tombés à 0 immédiatement, mais certaines opérations ont connu une latence élevée pendant que les travailleurs traitaient les arriérés accumulés pendant la panne, éliminant ensuite les suppressions de Cloud Pub/Sub, Cloud Build & Deploy, Cloud Dataflow et Cloud Storage cycle de vie. Parallèlement, alors que le trafic de lecture/écriture primaire était sain dans toute la région, des workflows spécifiques du cycle de vie à long terme ont nécessité plus de temps pour nettoyer les serrures et réconcilier les machines d'état distribuées. Calculer l'approvisionnement en VM du moteur dans la zone us-west1-c et Cloud Filestore plan de contrôle des verrous d'allocation d'instances et les vérifications de validation des ressources ont pris plus de temps pour normaliser les backends de stockage régionaux.

Traduit automatiquement depuis la mise à jour officielle de l'incident.

1. 16 juillet 2026 à 03:30 UTCresolved

   **Summary** Google Cloud VMware Engine (GCVE), Google Cloud NetApp Volumes, and Bare Metal Solution (BMS) services are experiencing temperature alerts in europe-west4-a. **Description** **GCVE:** Our engineering teams are currently working with data center facility teams to address the rising temperatures. A shutdown has already occurred for some private clouds, and we expect that others will shut down as well. **We recommend shutting down your VMware workload VMs for precaution so that recovery is safer. A shutdown of all systems started at 19:00 PDT \[4:00 Local Time (GMT+2)\]** We will notify you as soon as your Private Cloud has been powered back up and it is safe to restart the VMs. **BMS:** Some Bare Metal Solution servers are experiencing high temperatures and we are in the process of shutting them down. Customers may be unable to access their systems until we resolve the issue. **Google Cloud NetApp Volumes** Our engineering team has proactively shut down ONTAP clusters to protect the data stored on them. Customers may experience unavailability of Cloud NetApp Volumes until this issue is resolved. There is no estimated time for resolution. We will provide an update by Wednesday, 2026-07-15 21:30 PDT with current details. We apologize to all who are affected by the disruption. **Customer Symptoms** **GCVE:** Customers may have already observed a loss of connectivity to their private cloud. As a protective measure for hardware integrity and a recovery, we are shutting down all Private Clouds in this zone. **BMS:** Customers may have already lost connectivity to their BMS machines. As a protective measure for hardware integrity and a recovery, we are shutting down servers in this zone. **Google Cloud NetApp Volumes** Customers may have already lost connectivity to ONTAP clusters, or will lose connectivity to them shortly, as we proactively shut them down for their protection. Only Standard, Premium, and Extreme service levels are affected. **Workaround** There are no workarounds available for this issue at this time. Customers with multi-regional deployments are advised to route traffic to an alternate site.

2. 16 juillet 2026 à 04:17 UTCresolved

   **Summary** Google Cloud VMware Engine (GCVE), Google Cloud NetApp Volumes, and Bare Metal Solution (BMS) services are experiencing temperature alerts in europe-west4-a **Description** The datacenter serving europe-west4-a for GCVE, BMS, and NetApp has experienced a power failure, which subsequently caused a cooling failure. Google has proactively turned down workloads in order to protect customer data from any risks posed by running infrastructure in a high temperature environment. The datacenter operator team is working diligently to bring cooling and power back online so we can safely restart affected workloads. **GCVE:** Our engineering teams have shut down private clouds, and we expect that others will shut down as well. We will notify customers as soon as their private cloud has been powered back up and it is safe to restart the VMs. **BMS:** Some Bare Metal Solution servers are experiencing high temperatures and we are in the process of shutting them down. Customers may be unable to access their systems until we resolve the issue. **Google Cloud NetApp Volumes** Our engineering team has proactively shut down ONTAP clusters to protect the data stored on them. Customers may experience unavailability of Cloud NetApp Volumes until this issue is resolved. There is no estimated time for resolution. We will provide an update by Wednesday, 2026-07-15 22:30 PDT with current details. We apologize to all who are affected by the disruption. **Customer Symptoms** **GCVE:** Customers would have already observed a loss of connectivity to their private cloud. As a protective measure for hardware integrity and a recovery, we have shut down all private clouds in this zone. **BMS:** Customers may have already lost connectivity to their BMS machines. As a protective measure for hardware integrity and a recovery, we are shutting down some servers in this zone. **Google Cloud NetApp Volumes** Customers would have already lost connectivity to ONTAP clusters, as we proactively shut them down for their protection. Only Standard, Premium, and Extreme service levels are affected. **Workaround** There are no workarounds available for this issue at this time. Customers with multi-regional deployments are advised to route traffic to an alternate site.

3. 16 juillet 2026 à 04:41 UTCresolved

   **Summary** Google Cloud VMware Engine (GCVE), Google Cloud NetApp Volumes, and Bare Metal Solution (BMS) services are experiencing a service outage in europe-west4-a due to a cooling failure **Description** The datacenter serving europe-west4-a for GCVE, BMS, and NetApp has experienced a power failure, which subsequently caused a cooling failure. Google has proactively turned down workloads in order to protect customer data from any risks posed by running infrastructure in a high temperature environment. The datacenter operator team is working diligently to bring cooling and power back online so we can safely restart affected workloads. Some cooling has been restored, and we are working to restore affected workloads as available capacity allows. **GCVE:** Cooling has been restored at the site. As a precautionary measure, our engineering teams had shut down private clouds to protect these clouds from any damage. We are beginning service restoration and will notify customers as soon as their private cloud has been powered back up and it is safe to restart the VMs. **BMS:** Some Bare Metal Solution servers experienced high temperatures and we proactively shut them down. We are starting to bring the servers back online, and will provide an update once they are ready. Customers may be unable to access these systems until service is restored. **Google Cloud NetApp Volumes:** Our engineering team has proactively shut down ONTAP clusters to protect the data stored on them. Customers may experience unavailability of Cloud NetApp Volumes until this issue is resolved. There is no estimated time for resolution. We will provide an update by Wednesday, 2026-07-15 23:30 PDT with current details. We apologize to all who are affected by the disruption. **Customer Symptoms** **GCVE:** Customers would have already observed a loss of connectivity to their private cloud. As a protective measure for hardware integrity and recovery, we have shut down all private clouds in this zone. **BMS:** Customers may have lost connectivity to their BMS machines. As a protective measure for hardware integrity and recovery, we shut down some servers in this zone, and are actively restoring them. Customers may be unable to access these systems until restoration is complete. **Google Cloud NetApp Volumes:** Customers would have already lost connectivity to ONTAP clusters, as we proactively shut them down to protect these clusters from any damage. Only Standard, Premium, and Extreme service levels are affected. **Workaround** There are no workarounds available for this issue at this time. However, customers with multi-regional deployments are advised to route traffic to an alternate site.

4. 16 juillet 2026 à 06:29 UTCresolved

   **Summary** Google Cloud VMware Engine (GCVE), Google Cloud NetApp Volumes, and Bare Metal Solution (BMS) services are experiencing a service outage in europe-west4-a due to a cooling failure. **Description** The datacenter serving europe-west4-a for GCVE, BMS, and NetApp has experienced a power failure, which subsequently caused a cooling failure. Google proactively turned down workloads in order to protect customer data from any risks posed by running infrastructure in a high temperature environment. Cooling and power have been restored at the site. The datacenter operator team is now working to safely restart affected workloads. **GCVE:** As a precautionary measure, our engineering teams had shut down private clouds to protect these clouds from any damage. Service restoration is still in progress. We will notify customers as soon as their private cloud has been powered back up and it is safe to restart the VMs. **BMS:** We are continuing to restore servers and monitor their health. Customers may be unable to access these systems until service restoration is complete. **Google Cloud NetApp Volumes:** Our engineering team had proactively shut down ONTAP clusters to protect the data stored on them. Customers may experience unavailability of Cloud NetApp Volumes until this issue is resolved. We will provide an update by Thursday, 2026-07-16 00:30 PDT with current details. **Customer Symptoms** **GCVE:** Customers would have already observed a loss of connectivity to their private cloud. As a protective measure for hardware integrity and recovery, we have shut down all private clouds in this zone. **BMS:** Customers may have lost connectivity to their servers. As a protective measure for hardware integrity and recovery, we shut down some servers in this zone. We are continuing to restore servers and monitor their health. Customers may be unable to access these systems until service restoration is complete. **Google Cloud NetApp Volumes:** Customers would have already lost connectivity to ONTAP clusters, as we proactively shut them down to protect these clusters from any damage. Only Standard, Premium, and Extreme service levels are affected. **Workaround** There are no workarounds available for this issue at this time. However, customers with multi-regional deployments are advised to route traffic to an alternate site.

5. 16 juillet 2026 à 07:39 UTCresolved

   **Summary** Google Cloud VMware Engine (GCVE), Google Cloud NetApp Volumes, and Bare Metal Solution (BMS) services are experiencing a service outage in europe-west4-a due to a cooling failure. **Description** The datacenter serving europe-west4-a for GCVE, BMS, and NetApp has experienced a power failure, which subsequently caused a cooling failure. Google proactively turned down workloads in order to protect customer data from any risks posed by running infrastructure in a high temperature environment. Cooling and power have been restored at the site. The datacenter operator team is now working to safely restart affected workloads. **GCVE:** As a precautionary measure, our engineering teams had shut down private clouds to protect these clouds from any damage. Network fabric has been restored, and Private Cloud restoration remains underway. We will notify affected customers once their Private Clouds are restored to allow for VM operational verification. **BMS:** We are continuing to restore servers and monitor their health. Customers may be unable to access these systems until service restoration is complete. **Google Cloud NetApp Volumes:** Our engineering team had proactively shut down ONTAP clusters to protect the data stored on them. Recovery is currently in progress. Customers may experience unavailability of Cloud NetApp Volumes until this issue is resolved. We will provide an update by Thursday, 2026-07-16 01:30 PDT with current details. **Customer Symptoms** **GCVE:** Customers would have already observed a loss of connectivity to their private cloud. As a protective measure for hardware integrity and recovery, we have shut down all private clouds in this zone. **BMS:** Customers may have lost connectivity to their servers. As a protective measure for hardware integrity and recovery, we shut down some servers in this zone. We are continuing to restore servers and monitor their health. Customers may be unable to access these systems until service restoration is complete. **Google Cloud NetApp Volumes:** Customers would have already lost connectivity to ONTAP clusters, as we proactively shut them down to protect these clusters from any damage. Only Standard, Premium, and Extreme service levels are affected. **Workaround** There are no workarounds available for this issue at this time. However, customers with multi-regional deployments are advised to route traffic to an alternate site.

6. 16 juillet 2026 à 08:32 UTCresolved

   **Summary** Google Cloud VMware Engine (GCVE), Google Cloud NetApp Volumes, and Bare Metal Solution (BMS) services experienced a service outage in europe-west4-a due to a cooling failure. **Description** The datacenter serving europe-west4-a for GCVE, BMS, and NetApp has experienced a power failure, which subsequently caused a cooling failure. Google proactively turned down workloads in order to protect customer data from any risks posed by running infrastructure in a high temperature environment. Cooling and power have been restored at the site. The datacenter operator team is now validating site stability. **GCVE:** Network fabric has been restored, and Private Cloud restoration remains underway. We are notifying the affected customers as their Private Clouds are restored to allow for VM operational verification. **BMS:** Server restoration is largely complete, with most systems now online and reporting a healthy status. We are actively working through the remaining validation efforts to achieve full recovery. **Google Cloud NetApp Volumes:** Service restoration is complete, and all affected servers are now operational. We continue to monitor these for full service recovery. If customers are still experiencing impact from this issue, please contact Support, and we will work with you to resolve any residual impact. We will provide an update by Thursday, 2026-07-16 02:30 PDT with current details. **Customer Symptoms** **GCVE:** Customers would have observed a loss of connectivity to their private cloud. As a protective measure for hardware integrity and recovery, we had shut down all Private Clouds in this zone. We are in the process of bringing all restored Private Clouds online. **BMS:** Customers may have lost connectivity to their servers. As a protective measure for hardware integrity and recovery, we had shut down some servers in this zone. We are continuing to restore servers and monitor their health. **Google Cloud NetApp Volumes:** Customers would have lost connectivity to ONTAP clusters, as they were proactively shut down to protect these clusters from any damage. Only Standard, Premium, and Extreme service levels were affected. Customers should now be able to access their volumes again. **Workaround** There are no workarounds available for this issue at this time. However, customers with multi-regional deployments are advised to route traffic to an alternate site.

7. 16 juillet 2026 à 09:21 UTCresolved

   **Summary** Google Cloud VMware Engine (GCVE), Google Cloud NetApp Volumes, and Bare Metal Solution (BMS) services experienced a service outage in europe-west4-a due to a cooling failure. **Description** The datacenter serving europe-west4-a for GCVE, BMS, and NetApp has experienced a power failure, which subsequently caused a cooling failure. Google proactively turned down workloads in order to protect customer data from any risks posed by running infrastructure in a high temperature environment. Cooling and power have been restored at the site. The datacenter operator team is now validating site stability. **GCVE:** Network fabric has been restored, and Private Cloud restoration remains underway. We are notifying the affected customers as their Private Clouds are restored to allow for VM operational verification. **BMS:** Server restoration is largely complete, with most systems now online and reporting a healthy status. We are actively working through the remaining validation efforts to achieve full recovery. **Google Cloud NetApp Volumes:** Service restoration is complete, and all affected servers are now operational. We continue to monitor these for full service recovery. If customers are still experiencing impact from this issue, please contact Support, and we will work with you to resolve any residual impact. We will provide an update by Thursday, 2026-07-16 03:30 PDT with current details. **Customer Symptoms** **GCVE:** Customers would have observed a loss of connectivity to their private cloud. As a protective measure for hardware integrity and recovery, we had shut down all Private Clouds in this zone. We are in the process of bringing all restored Private Clouds online. **BMS:** Customers may have lost connectivity to their servers. As a protective measure for hardware integrity and recovery, we had shut down some servers in this zone. We are continuing to restore servers and monitor their health. **Google Cloud NetApp Volumes:** Customers would have lost connectivity to ONTAP clusters, as they were proactively shut down to protect these clusters from any damage. Only Standard, Premium, and Extreme service levels were affected. Customers should now be able to access their volumes again. **Workaround** None at this time.

8. 16 juillet 2026 à 10:14 UTCresolved

   **Summary** Google Cloud VMware Engine (GCVE), Google Cloud NetApp Volumes, and Bare Metal Solution (BMS) services experienced a service outage in europe-west4-a due to a cooling failure. **Description** The datacenter serving europe-west4-a for GCVE, BMS, and NetApp has experienced a power failure, which subsequently caused a cooling failure. Google proactively turned down workloads in order to protect customer data from any risks posed by running infrastructure in a high temperature environment. Cooling and power have been restored at the site. The issue with GCVE and Google Cloud NetApp Volumes has been resolved for all affected users. We continue to recover the impact for BMS at the moment. **GCVE [Mitigated]:** Service restoration is complete across all the Private Clouds. Customers have been informed about this restoration and requested to verify that their VMs are fully operational. **Google Cloud NetApp Volumes [Mitigated]:** Service restoration is complete, and all affected servers are now operational. We continue to monitor these for full service recovery. **BMS:** Server restoration is largely complete, with most systems now online and reporting a healthy status. We are actively working through the remaining validation efforts to achieve full recovery. For Google Cloud NetApp Volumes and GCVE, if customers are still experiencing impact from this issue, please contact Support, and we will work with you to resolve any residual impact. We will provide the BMS restoration status update by Thursday, 2026-07-16 04:30 PDT with current details. **Customer Symptoms** The issue is now resolved for GCVE and Google Cloud NetApp Volumes. We continue to recover the impact for BMS at the moment. **GCVE [Mitigated]:** Customers would have observed a loss of connectivity to their private cloud. **Google Cloud NetApp Volumes [Mitigated]:** Customers would have lost connectivity to ONTAP clusters, as they were proactively shut down to protect these clusters from any damage. Only Standard, Premium, and Extreme service levels were affected. Customers should now be able to access their volumes again. **BMS:** Customers may have lost connectivity to their servers. As a protective measure for hardware integrity and recovery, we had shut down some servers in this zone. We are continuing to restore servers and monitor their health. **Workaround** None at this time.

9. 16 juillet 2026 à 11:39 UTCresolved

   **Summary** Google Cloud VMware Engine (GCVE), Google Cloud NetApp Volumes, and Bare Metal Solution (BMS) services experienced a service outage in europe-west4-a due to a cooling failure. **Description** The datacenter serving europe-west4-a for GCVE, BMS, and NetApp has experienced a power failure, which subsequently caused a cooling failure. Google proactively turned down workloads in order to protect customer data from any risks posed by running infrastructure in a high temperature environment. Cooling and power have been restored at the site. The issue with GCVE and Google Cloud NetApp Volumes has been resolved for all affected users. We continue to recover the impact for BMS at the moment. **GCVE [Mitigated]:** Service restoration is complete across all the Private Clouds. Customers have been informed about this restoration and requested to verify that their VMs are fully operational. **Google Cloud NetApp Volumes [Mitigated]:** Service restoration is complete, and all affected servers are now operational. We continue to monitor these for full service recovery. **BMS:** Server restoration is largely complete, with most systems now online and reporting a healthy status. We are actively working through the remaining validation efforts to achieve full recovery. For Google Cloud NetApp Volumes and GCVE, if customers are still experiencing impact from this issue, please contact Support, and we will work with you to resolve any residual impact. We will provide the BMS restoration status update by Thursday, 2026-07-16 05:30 PDT with current details. **Customer Symptoms** The issue is now resolved for GCVE and Google Cloud NetApp Volumes. We continue to recover the impact for BMS at the moment. **GCVE [Mitigated]:** Customers would have observed a loss of connectivity to their private cloud. **Google Cloud NetApp Volumes [Mitigated]:** Customers would have lost connectivity to ONTAP clusters, as they were proactively shut down to protect these clusters from any damage. Only Standard, Premium, and Extreme service levels were affected. Customers should now be able to access their volumes again. **BMS:** Customers may have lost connectivity to their servers. As a protective measure for hardware integrity and recovery, we had shut down some servers in this zone. We are continuing to restore servers and monitor their health. **Workaround** None at this time.

10. 16 juillet 2026 à 12:25 UTCresolved

    **Summary** Google Cloud VMware Engine (GCVE), Google Cloud NetApp Volumes, and Bare Metal Solution (BMS) services experienced a service outage in europe-west4-a due to a cooling failure. **Description** The datacenter serving europe-west4-a for GCVE, BMS, and NetApp has experienced a power failure, which subsequently caused a cooling failure. Google proactively turned down workloads in order to protect customer data from any risks posed by running infrastructure in a high temperature environment. Cooling and power have been restored at the site. The issue with GCVE and Google Cloud NetApp Volumes has been resolved for all affected users as of Thursday, 2026-07-16 02:31 PDT and 01:00 PDT respectively. The issue with BMS is largely resolved and is believed to be affecting a very small number of customers. Our Engineering will continue to recover this residual impact directly with the impacted customers. If you have questions or are still impacted, please open a case with the Support Team and we will work with you until this issue is resolved. **GCVE:** Service restoration is complete across all the Private Clouds. Customers have been informed about this restoration and requested to verify that their VMs are fully operational. **Google Cloud NetApp Volumes:** Service restoration is complete, and all affected servers are now operational. **BMS:** Service restoration is largely complete. We continue working with a very small number of remaining affected customers to achieve full recovery. We thank you for your patience while we continue to fully resolve the issue. **Customer Symptoms** The issue is now resolved for GCVE and Google Cloud NetApp Volumes. The issue with BMS is largely resolved and our engineering will continue to recover this residual impact directly with the impacted customers. **GCVE:** Customers would have observed a loss of connectivity to their private cloud. **Google Cloud NetApp Volumes:** Customers would have lost connectivity to ONTAP clusters, as they were proactively shut down to protect these clusters from any damage. Only Standard, Premium, and Extreme service levels were affected. **BMS:** Customers may have lost connectivity to their servers. As a protective measure for hardware integrity and recovery, we had shut down some servers in this zone. **Workaround** None at this time.

11. 18 juillet 2026 à 00:22 UTCresolved

    # Preliminary Incident Report We sincerely apologize for the disruption this incident caused to your business. We know how much you rely on Google Cloud, and we regret the impact on your productivity. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. A final Incident Report with preventative actions will be posted once our investigation is complete. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support. ## Date/Time of the Issue (All time US/Pacific) **Google Cloud VMware Engine Impact:** - Start: 15 July 2026 17:09 - End: 16 July 2026 2:33 - Duration: 9 Hours, 24 Minutes **Google Cloud NetApp Volumes Impact:** - Start: 15 July 2026 16:39 - End: 16 July 2026 01:10 - Duration: 8 Hours 31 Minutes **Bare Metal Solution Impact:** - Start: 15 July 2026 18:37 - End: 16 July 2026 07:34 - Duration: 12 Hours 57 Minutes ## Summary On Wednesday, 15 July 2026, Google Cloud VMware Engine, Bare Metal Solution, and Google Cloud NetApp Volumes experienced service interruptions for a total duration of 14 hours, 55 minutes. We are taking immediate steps to ensure this doesn’t happen again. ## Preliminary Root Cause A regional data center hosting services in europe-west4-a experienced a loss of utility power and subsequent cooling capacity. The sequence of events leading to customer impact was as follows: - An electrical fault occurred on the utility grid upstream of the data center, disrupting the electrical distribution gear and cooling equipment. - The loss of cooling infrastructure resulted in ambient temperatures rising rapidly within the affected data halls. Host servers, storage clusters, and network switches were shut down to prevent equipment damages due to the extreme heat. - The high temperatures in the data hall resulted in disruption to customer workloads and control plane operations across the affected services. Google engineers have begun a full root cause analysis and we will provide additional information once it is available. ## Remediation Google engineering teams were alerted to the issue via automated temperature and hardware unreachability telemetry starting at 16:39 US/Pacific. Engineers collaborated with the third-party facility provider to safely restore primary utility power and cooling systems, returning ambient room temperatures to safe operational levels. With the environment stabilized, field support engineers and remote teams systematically booted and verified the server hosts, storage nodes, and network fabric in a controlled sequence. Automated recovery playbooks were executed to restore underlying network switches and routing infrastructure, allowing storage appliances and compute clusters to be brought back online and verified for health. The underlying infrastructure has been fully recovered, and normal operations have been successfully restored. ## Description of Impact On 15 July 2026 from 16:39 to 16 July 2026 07:34 US/Pacific, customers experienced service disruptions across several products in europe-west4-a zone: - **Google Cloud VMware Engine:** Customers lost connectivity to their private clouds and workloads as foundational hosts and network switches were powered off for thermal protection. - **Bare Metal Solution:** Customers experienced connectivity loss to their database servers and storage appliances as the underlying hardware was powered down. - **Google Cloud NetApp Volumes:** Customers in the STANDARD, PREMIUM, and EXTREME service levels were unable to access their storage volumes. Control plane operations, including the creation of new storage pools, volumes, and backups, experienced failures in the affected region.

12. 25 juillet 2026 à 13:16 UTCresolved

    ## \# Incident Report ## \#\# Summary On Wednesday, 15 July 2026, Google Cloud VMware Engine (GCVE), Bare Metal Solution (BMS), and Google Cloud NetApp Volumes (GCNV) experienced service interruptions for a total duration of 14 hours, 55 minutes. The root cause for this outage is a 3ms voltage drop in the power feed from the utility provider and subsequent utility breaker protective action. This incident was mitigated by the data center provider rectifying the failed systems followed by the Google engineering team restoring the services. ## \#\# Root Cause A regional data center hosting services in europe-west4-a experienced an upstream voltage transient affecting both utility power feeds A and B. During this event, the utility breakers on both A and B feeds tripped and initiated the transfer to the back-up power source DRUPS (Diesel Rotary Uninterruptible Power Supply). The transition of side B to DRUPS system was successful without any power interruption. The DRUPS back-up power system for side A failed to take over the facility load due to electrical component failures. The DRUPS failure to take over load initiated automatic transfer of the affected 3 rows from feed A to redundant power feed B. Rows 1 and 2 transferred to power feed B successfully. Row 3 failed to transfer to power feed B and experienced complete power loss due to an overload protection breaker trip. The root cause of the failed transfer on the affected row was identified as load deployment discrepancy which is under further investigation by Google engineering team. This resulted in loss of both redundant power feeds to the single row. During the voltage transient event, the server data hall experienced an increased temperature due to a cooling system failure. The chiller controller dropped offline during the voltage transient event, failing to signal the chilled water distribution pumps to restart and ultimately causing the chiller system A to shut down. The redundant source was not available due to known ongoing construction work at the facility. This resulted in the data hall temperatures reaching 44°C and subsequent shutdown of the affected data hall machines. Google engineering teams initiated machine shutdown procedures for the remainder of reachable devices as part of the cooling emergency shutdown process. A timeline of events during the incident is provided below. ### DataCenter Events | Timestamp(PST) | Event Description | | :---- | :---- | | 07-15-2026 16:24 | An electrical fault occurred on the utility grid upstream of the data center, disrupting the electrical distribution. | | 07-15-2026 16:24 | DRUPS back-up power system for side A failed to take over the data hall load. | | 07-15-2026 16:24 | The chiller controller dropped offline during the voltage transient event causing distribution pumps to stop and unable to restart. | | 07-15-2026 16:37 | Rows 1 and 2 transferred to power feed B successfully. Row 3 failed to transfer to power feed B and lost power. | | 07-15-2026 18:29 | Data hall temperatures reached 44C and crossed the safe machine operating threshold. | | 07-15-2026 19:55 | Machine shutdown procedures implemented. | | 07-15-2026 21:05 | Cooling system fully recovered and temperatures in the data hall returned to normal operating range. | | 07-15-2026 21:24 | Notification about cooling recovery sent by provider | | 07-15-2026 21:46 | Notification about power recovery sent by provider | ### GCVE Timelines | Timestamp(PST) | Event Description | | :---- | :---- | | 07-15-2026 16:30 | GCVE service power monitoring alert received | | 07-15-2026 16:41 | First symptom detected, servers reporting redundancy power feed failure. | | 07-15-2026 17:05 | Switch temperature \>60°C alerts triggered. | | 07-15-2026 17:09 | Start of user impact, first prober alert failure for north south traffic for customers. | | 07-15-2026 19:00 | Server / Network devices shutdown initiated. | | 07-15-2026 19:55 | All reachable server/network devices were shut down. | | 07-15-2026 21:24 | Cooling system fully recovered and temperatures in the datahall were returning to normal operating range. | | 07-15-2026 21:56 | Network restoration started with reachable hydra devices. | | 07-15-2026 22:40 | Onsite technician arrived, to recover the console servers. | | 07-15-2026 23:10 | Console server reboot/recovery complete. | | 07-16-2026 00:33 | Network recovery for all Placement Groups complete. | | 07-16-2026 02:36 | Incident mitigated, after recovering all the customer PCs are fully healthy. | This outage impacted 24 private clouds of 20 distinct customers in europe-west4-a region. ### Bare Metal Solutions (BMS) Timelines | Timestamp(PST) | Event Description | | :---- | :---- | | 07-15-2026 16:41 | First symptom detected, servers and storage reporting redundancy power feed failure | | 07-15-2026 17:52 | Automated monitoring detected rising temperatures | | 07-15-2026 18:37 | Four Netapp SAN storage nodes failed and were shut down due to overheating, resulting in storage availability issue | | 07-15-2026 18:37 | Customer impact started, first server shutdown detected | | 07-15-2026 18:59 | Reserve and buffer servers started to get powered off | | 07-15-2026 20:43 | Drop in the temperature has been detected | | 07-15-2026 20:52 | Four Netapp SAN storage nodes recovered, mitigating storage availability issue | | 07-15-2026 21:05 | The cooling system fully recovered and temperatures in the datahall were returning to normal operating range. | | 07-15-2026 23:23 | First customer server reboot started | | 07-16-2026 06:32 | Last customer server reboot started | | 07-16-2026 08:42 | Incident mitigated. All impacted customer servers were rebooted and confirmed in healthy state. | This outage impacted 9 distinct BMS customers in europe-west4 ### NetApp Timelines | Timestamp(PST) | Event Description | | :---- | :---- | | 07-15-2026 16:32 | Netapp facility remote monitoring alert on high temp and network switches switches failure received | | 07-15-2026 17:32 | All 6 clusters failed and were shut down automatically due to high temperature | | 07-15-2026 21:05 | Cooling capacity restored | | 07-15-2026 22:08 | Cluster recovery process initiated for all 6 clusters. 1st cluster recovered | | 07-15-2026 23:55 | All 6 Netapp Clusters and all customers recovered and validated as healthy & operational | ## \#\# Remediation and Prevention Google engineers were alerted to the infrastructure power loss on Wednesday, 15 July 2026 at 16:30 US/Pacific and subsequent increase in data hall temperatures at 17:44 US/Pacific via our automated monitoring and telemetry, and immediately began investigating. During the voltage transient event, the chillers maintained power but shut down because the distribution pumps failed to restart, resulting in no water flow. The pumps were manually switched from auto to hand mode to restore circulation and cooling. The facility team deployed an interim portable UPS to support the chiller controllers and prevent localized controller power loss. Upon confirmation of utility power restoration, the facility returned to utility power via automatic switching with the exception of two Remote Power Panels (RPPs), which required manual verification before restoration of power to affected server row. To provide full resiliency the facility team aligned a swing DRUPS unit to restore the redundant configuration on the A feed and is working on repairing the faulty DRUPS. To remediate the power loss at Row 3, once the electrical fault was confirmed to be fully isolated, teams reset the tripped breakers, successfully restoring both redundant power to the impacted server racks. Google is committed to preventing a repeat of this issue in the future and is completing the following actions: * Detailed investigation by the engineering team of the sequence of events that prevented transfer of load to DRUPS system and identified system improvements **\[ETA Aug 2026\].** * Determine and resolve the cause of overload conditions that caused power loss to Row 3 **\[ETA Aug 2026\].** * Perform investigation of chiller pump control system redundancy setup to ensure cooling system resiliency **\[ETA Sep 2026\].** * Implement improvements to facility system monitoring alerts for power events and configure early alert thresholds for cooling excursion detection. **\[ETA Aug 2026\].** * The GCVE engineering team is working to improve the automated triggering of shutdown during quickly evolving thermal runoff situations **\[ETA Oct 2026\]**. * The BMS engineering team is working with partner teams to update the incident classification and SLO definitions to improve personnel availability during major datacenter events **\[ETA Sep 2026\]**. * The GCNV engineering team is collaborate with the data center team to review incident handling SLA and runbooks, identify potential future occurrences and create playbooks to address/mitigate them, establish monthly joint emergency drill, and review if the Netapp cluster recovery process can be expedited **\[ETA Aug 2026\].** ## \#\# Detailed Description of Impact On 15 July 2026 from 16:39 to 16 July 2026 07:34 US/Pacific, customers experienced service disruptions across several products in europe-west4-a zone: * **Google Cloud VMware Engine**: Customers lost connectivity to their private clouds and workloads as foundational hosts and network switches were powered off for thermal protection. * **Bare Metal Solution**: Customers experienced connectivity loss to their database servers and storage appliances as the underlying hardware was powered down. * **Google Cloud NetApp Volumes**: Customers in the STANDARD, PREMIUM, and EXTREME service levels were unable to access their storage volumes. Control plane operations, including the creation of new storage pools, volumes, and backups, experienced failures in the affected region.

1. 14 juillet 2026 à 20:24 UTCresolved

   **Summary** Google Cloud VMware Engine (GCVE) Stretched Cluster customers are experiencing zonal outages impacting network connectivity across multiple regions. **Description** We are experiencing an issue with Google Cloud VMware Engine (GCVE) beginning Tuesday, 2026-07-14 at 10:00 PDT. We are continuing to investigate and mitigate the issue. We believe this disruption is isolated to network connectivity issues for stretch clusters, while Storage and Compute services appear to be unaffected. GCVE VMs are running as expected but customers may experience connectivity issues to the VMs. We will provide another update by Tuesday, 2026-07-14 14:00 PDT with current details. **Customer Symptoms** Some GCVE customers may experience inter-site communication failures to their GCVE environments within the affected zones. **Workaround** None at this time.

2. 14 juillet 2026 à 21:31 UTCresolved

   **Summary** Google Cloud VMware Engine (GCVE) Stretched Cluster customers are experiencing zonal outages impacting network connectivity across multiple regions. **Description** We are experiencing an inter-site communication issue with Google Cloud VMware Engine (GCVE) stretched cluster customers beginning Tuesday, 2026-07-14 at 10:00 PDT. We have identified the customers that are affected by this issue and we are working with them on mitigation. Our preliminary investigation indicates this is stemming from an underlying network connectivity issue affecting the infrastructure that links the zones within a stretch cluster. This disruption is causing synchronization issues between the affected zones. We believe Storage and Compute services remain unaffected, and VMs are running as expected, though connectivity to them may be degraded. We will provide another update by Tuesday, 2026-07-14 15:00 PDT with current details. **Customer Symptoms** Some GCVE customers using Stretched Cluster may experience inter-site communication failures to their GCVE environments within the affected zones. **Workaround** While we work on restoring full connectivity, we recommend the following workarounds to restore access to your workloads: - VM Migration (recommended): Where possible, migrate your affected VMs to the healthy and unaffected side of the stretch cluster. We strongly recommend consulting with Google Support before proceeding.

3. 14 juillet 2026 à 22:16 UTCresolved

   **Summary** Google Cloud VMware Engine (GCVE) Stretched Cluster customers are experiencing zonal outages impacting network connectivity across multiple regions. **Description** We are experiencing an inter-site communication issue with Google Cloud VMware Engine (GCVE) stretched cluster customers beginning Tuesday, 2026-07-14 at 10:00 PDT. We have identified the customers that are affected by this issue and we are working with them on mitigation. Our investigation has identified underlying inter-zone communication failures and Border Gateway Protocol (BGP) session flapping between cluster zones. Specifically, network connectivity has been lost between the affected zones and the witness appliance. Because the witness appliance is currently unreachable, the cluster zones are unable to safely synchronize state. As a result, VMs on the affected sites are becoming isolated and may be left without writable data. **Failover / VM Migration:** Migrating affected VMs to the healthy, secondary zone of your stretch cluster remains the primary mitigation strategy. Because of the complexities surrounding failover risks and secondary zone health, we highly encourage you to open a ticket with Google Cloud Support if you are severely impacted. We do not currently have an ETA for resolution. We will provide another update by Tuesday, 2026-07-14 16:00 PDT with current details. **Customer Symptoms** Some GCVE customers using Stretched Cluster may experience inter-site communication failures to their GCVE environments within the affected zones. **Workaround** While we work on restoring full connectivity, we recommend the following workarounds to restore access to your workloads: - VM Migration (recommended): Where possible, migrate your affected VMs to the healthy and unaffected side of the stretch cluster. We strongly recommend consulting with Google Support before proceeding.

4. 14 juillet 2026 à 23:05 UTCresolved

   **Summary** Google Cloud VMware Engine (GCVE) Stretched Cluster customers are experiencing zonal outages impacting network connectivity across multiple regions. **Description** We are experiencing an inter-site communication issue with Google Cloud VMware Engine (GCVE) stretched cluster customers beginning Tuesday, 2026-07-14 at 10:00 PDT. We have identified the customers that are affected by this issue and we are working with them on mitigation. Our investigation has identified a recent configuration update that is the likely cause of the inter-zone network disruption. Teams are working on remediation. **Failover / VM Migration:** Migrating affected VMs to the healthy, secondary zone of your stretch cluster remains the primary mitigation strategy. Because of the complexities surrounding failover risks and secondary zone health, we highly encourage you to open a ticket with Google Cloud Support if you are severely impacted. We do not currently have an ETA for resolution. We will provide another update by Tuesday, 2026-07-14 17:00 PDT with current details. **Customer Symptoms** Some GCVE customers using Stretched Cluster may experience inter-site communication failures to their GCVE environments within the affected zones. **Workaround** While we work on restoring full connectivity, we recommend the following workarounds to restore access to your workloads: - VM Migration (recommended): Where possible, migrate your affected VMs to the healthy and unaffected side of the stretch cluster. We strongly recommend consulting with Google Support before proceeding.

5. 15 juillet 2026 à 00:13 UTCresolved

   **Summary** Google Cloud VMware Engine (GCVE) Stretched Cluster customers are experiencing zonal outages impacting network connectivity across multiple regions. **Description** We are experiencing an inter-site communication issue with Google Cloud VMware Engine (GCVE) stretched cluster customers beginning Tuesday, 2026-07-14 at 10:00 PDT. We have identified the customers that are affected by this issue and we are working with them on mitigation. A remediation rollout is currently in progress to address the underlying network issue. **Failover / VM Migration:** Migrating affected VMs to the healthy, secondary zone of your stretch cluster remains the primary mitigation strategy. Because of the complexities surrounding failover risks and secondary zone health, we highly encourage you to open a ticket with Google Cloud Support if you are severely impacted. We do not currently have an ETA for resolution. We will provide another update by Tuesday, 2026-07-14 18:00 PDT with current details. **Customer Symptoms** Some GCVE customers using Stretched Cluster may experience inter-site communication failures to their GCVE environments within the affected zones. Upon further investigation we identified that the northamerica-northeast2 region was not impacted. **Workaround** While we work on restoring full connectivity, we recommend the following workarounds to restore access to your workloads: - VM Migration (recommended): Where possible, migrate your affected VMs to the healthy and unaffected side of the stretch cluster. We strongly recommend consulting with Google Support before proceeding.

6. 15 juillet 2026 à 01:15 UTCresolved

   **Summary** Google Cloud VMware Engine (GCVE) Stretched Cluster customers are experiencing zonal outages impacting network connectivity across multiple regions. **Description** We are experiencing an inter-site communication issue with Google Cloud VMware Engine (GCVE) stretched cluster customers beginning Tuesday, 2026-07-14 at 10:00 PDT. We have identified the customers that are affected by this issue and we are working with them on mitigation. A remediation rollout is currently in progress to address the underlying network issue. **Failover / VM Migration:** Migrating affected VMs to the healthy, secondary zone of your stretch cluster remains the primary mitigation strategy. Because of the complexities surrounding failover risks and secondary zone health, we highly encourage you to open a ticket with Google Cloud Support if you are severely impacted. We do not currently have an ETA for resolution. We will provide another update by Tuesday, 2026-07-14 19:00 PDT with current details. **Customer Symptoms** Some GCVE customers using Stretched Cluster may experience inter-site communication failures to their GCVE environments within the affected zones. **Workaround** While we work on restoring full connectivity, we recommend the following workarounds to restore access to your workloads: - VM Migration (recommended): Where possible, migrate your affected VMs to the healthy and unaffected side of the stretch cluster. We strongly recommend consulting with Google Support before proceeding.

7. 15 juillet 2026 à 01:55 UTCresolved

   **Summary** Google Cloud VMware Engine (GCVE) Stretched Cluster customers are experiencing zonal outages impacting network connectivity across multiple regions. **Description** We are experiencing an inter-site communication issue with Google Cloud VMware Engine (GCVE) stretched cluster customers beginning Tuesday, 2026-07-14 at 10:00 PDT. We have identified the customers that are affected by this issue and we are working with them on mitigation. A remediation rollout is currently in progress to address the underlying network issue. **Failover / VM Migration:** Migrating affected VMs to the healthy, secondary zone of your stretch cluster remains the primary mitigation strategy. Because of the complexities surrounding failover risks and secondary zone health, we highly encourage you to open a ticket with Google Cloud Support if you are severely impacted. We do not currently have an ETA for resolution. We will provide another update by Tuesday, 2026-07-14 20:30 PDT with current details. **Customer Symptoms** Some GCVE customers using Stretched Cluster may experience inter-site communication failures to their GCVE environments within the affected zones. **Workaround** While we work on restoring full connectivity, we recommend the following workarounds to restore access to your workloads: - VM Migration (recommended): Where possible, migrate your affected VMs to the healthy and unaffected side of the stretch cluster. We strongly recommend consulting with Google Support before proceeding.

8. 15 juillet 2026 à 03:27 UTCresolved

   **Summary** Google Cloud VMware Engine (GCVE) Stretched Cluster customers are experiencing zonal outages impacting network connectivity across multiple regions. **Description** We are experiencing an inter-site communication issue with Google Cloud VMware Engine (GCVE) stretched cluster customers beginning Tuesday, 2026-07-14 at 10:00 PDT. We have identified the customers that are affected by this issue and we are working with them on mitigation. A remediation rollout is currently in progress to address the underlying network issue. **Failover / VM Migration:** Migrating affected VMs to the healthy, secondary zone of your stretch cluster remains the primary mitigation strategy. Because of the complexities surrounding failover risks and secondary zone health, we highly encourage you to open a ticket with Google Cloud Support if you are severely impacted. We do not currently have an ETA for resolution. We will provide another update by Tuesday, 2026-07-14 22:00 PDT with current details. **Customer Symptoms** Some GCVE customers using Stretched Cluster may experience inter-site communication failures to their GCVE environments within the affected zones. **Workaround** While we work on restoring full connectivity, we recommend the following to restore access to your workloads: * VM Migration (recommended): Where possible, migrate your affected VMs to the healthy and unaffected side of the stretch cluster. We strongly recommend consulting with Google Cloud Support before proceeding.

9. 15 juillet 2026 à 05:34 UTCresolved

   **Summary** Google Cloud VMware Engine (GCVE) Stretched Cluster customers may have experienced zonal outages impacting network connectivity across multiple regions. **Description** We experienced an inter-site communication issue with Google Cloud VMware Engine (GCVE) Stretched Cluster customers beginning Tuesday, 2026-07-14 at 10:00 PDT. We identified the affected customers, and have worked with them to fully mitigate the issue by Tuesday, 2026-07-14 at 21:46 PDT. Preliminary analysis indicates that a network configuration update was the cause of the inter-zone network disruption. Our engineering team mitigated the issue by rolling back the faulty configuration to its last-known good value. If customers are still experiencing impact from this issue, please contact Support, and we will work with you to resolve any residual impact. We thank you for your patience while we worked to resolve this issue. **Customer Symptoms** Some GCVE customers using Stretched Cluster may have experienced inter-site communication failures to their GCVE environments within the affected zones. **Workaround** The issue is now mitigated.

10. 20 juillet 2026 à 18:31 UTCresolved

    ## Preliminary Incident Report We sincerely apologize for the disruption this incident caused to your business. We know how much you rely on Google Cloud, and we regret the impact on your productivity. We are working to address the root cause and prevent this from occurring in the future. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. A final Incident Report with preventative actions will be posted once our investigation is complete. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using [**https://cloud.google.com/support**](https://cloud.google.com/support). ## Date/Time of the Issue (All time US/Pacific) Incident Start: 14 July 2026 10:00 Incident End: 14 July 2026 20:40 Duration: 10 hours, 40 minutes ## Summary On Tuesday, 14 July 2026, Google Cloud VMware Engine (GCVE) Stretched Cluster customers in the australia-southeast2 and europe-west3 zones experienced inter-site communication failures. The disruption was traced to a network configuration update that introduced a conflict, causing inter-site communication failures, which triggered VMware Stretch Cluster failover events. Google engineers successfully mitigated the disruption by rolling back the configuration change. ## Preliminary Root Cause A configuration change deployed within the Google Cloud network affected traffic between the VMware Engine stretched cluster zones. This resulted in a loss of connectivity between zones in the stretched cluster deployment. Google engineers are doing a full root cause analysis and will provide additional information once it is available. ## Remediation To stabilize the environment, engineers identified the working network paths and deployed configuration changes to reroute traffic and restore connectivity. GCVE stretch cluster inter-zonal connectivity was completely restored for all supported locations on Tuesday, 14 July 2026 at 20:40 US/Pacific. ## Description of Impact On Tuesday, 14 July 2026 from 10:00 to 20:40 US/Pacific, customers utilizing GCVE stretch clusters in the affected zones (australia-southeast2, and europe-west3) experienced inter-site communication failures. For some customers, depending on their architecture, this disruption led to VMWare HA event causing VM restarts/movement across zones as designed, Host disconnects, VSAN alarms and Intermittent access to VMware Management (vCenter/NSX Manager) components. ---

11. 24 juillet 2026 à 07:10 UTCresolved

    ## Incident Report ## Summary On Tuesday, 14 July 2026 10:00 PT, Google Cloud VMware Engine (GCVE) Stretched Cluster customers in the australia-southeast2 and europe-west3 zones experienced inter-site communication failures. The disruption was traced to a network configuration update that introduced a conflict, causing inter-site communication failures, which triggered VMware Stretched Cluster failover events. Google engineers successfully mitigated the impact by rolling back the issue-causing configuration change and resetting the network state. We sincerely apologize for the disruption this incident caused to your business. We know how much you rely on Google Cloud, and we regret the impact on your productivity. We are working to address the root cause and prevent this from occurring in the future. ## Root Cause A network configuration update intended to prepare the cloud network infrastructure for new capabilities was deployed to the foundational network control plane. While the configuration payload itself was structurally valid, it exposed an implementation gap within the control plane's routing logic. This logical gap caused the underlying network hosts to misconfigure program routing tables. As a result, traffic destined for the private IP address space used by GCVE Stretched Clusters was dropped. Because Stretched Clusters rely on this private address space to establish routing sessions for inter-zonal connectivity, the traffic drops severed communications between the active zones of the clusters, triggering VMware High Availability (HA) failovers. Standard routing health-checking protocols, such as Border Gateway Protocol (BGP) and Bidirectional Forwarding Detection (BFD), remained fully functional because their control plane sessions run on separate, unaffected address spaces. Because the control plane remained healthy, standard failover mechanisms failed to detect that the selective private IP range used for inter-zonal data tunneling was being dropped. Automated safeguards did not block the deployment because the configuration passed initial payload validations. The impact only manifested once the update began routing data traffic through the specific affected IP range. ## Remediation and Prevention To stabilize the environment, engineers identified the working network paths and deployed configuration changes to reroute traffic and restore connectivity. GCVE Stretched Cluster inter-zonal connectivity was completely restored for all supported locations on Tuesday, 14 July 2026 at 20:40 US/Pacific. Google is committed preventing a repeat of this issue in the future and is completing the following actions: * **Expanded Testing:** We are adding more detailed GCVE network setups to our existing testing environments. This allows us to automatically test future network updates against these configurations before they go live. * **Service-Level Data Path Failover:** We are implementing additional service-level data path failover mechanisms that actively probe the specific data-tunneling traffic space. This will ensure a path failover is triggered if the data plane itself is degraded even when the BGP control plane remains functional. * **Detailed Alerting**: We are adding faster, more specific alerts for connection issues between zones. This builds on our current platform monitoring to catch minor disruptions early and speed up our response. * **Improved Cluster Resilience:** We are fine-tuning the cluster's high-availability and storage settings. This makes virtual machines more resilient to short network drops, preventing them from restarting unnecessarily if the main site is still healthy. * **Workload Resilience Alignment (Shared Responsibility):** We are proactively reaching out to customers utilizing non-vSAN replicated virtual machine configurations within Stretched Clusters. Because these workloads are pinned to a single zone without active cross-site replication, they cannot survive inter-site network disruptions. We are ready to assist customers in auditing their storage policies, adjusting Stretched Cluster configurations, and planning the secondary zone capacity required to enable robust high-availability failovers. ## Detailed Description of Impact On Tuesday, 14 July 2026 from 10:00 to 20:40 US/Pacific, customers utilizing GCVE Stretched Clusters in the affected zones (australia-southeast2, and europe-west3) experienced inter-site communication failures. For some customers, depending on their architecture, this disruption led to a VMWare HA event causing VM restarts/movement across zones as designed, host disconnects, VSAN alarms and intermittent access to VMware Management (vCenter/NSX Manager) components.

1. 10 juin 2026 à 01:13 UTCresolved

   **Summary** Network traffic to Google Cloud originating from Delhi, Chennai, Mumbai and surrounding areas is experiencing intermittent periods of elevated latency and possible packet loss. **Description** A fire at a third-party data center facility required an emergency power shutdown of networking equipment, isolating a non-compute local Point of Presence (POP) in Delhi and reducing available network capacity in the metro area. We rerouted significant traffic from the impacted facility in Delhi to address reduced local serving capabilities. As a result, a subset of Hybrid Connectivity and Virtual Private Cloud (VPC) customers may be impacted by the routing changes made to address reduced local, latency-optimized serving capabilities in Delhi. Affected customers may experience intermittent latency spikes due to demand exceeding capacity across Indian metros and regional ISPs. We are investigating additional traffic mitigations and Internet Edge peering augmentation to alleviate the latency issues affecting our customers. We will provide our next update by Wednesday, 2026-06-10 17:00 PDT. **Symptoms** Customers may experience slightly elevated latency and non-optimal network routing into Google Cloud until the affected facility is fully restored. **Workaround** There is no workaround at this time.

2. 10 juin 2026 à 22:25 UTCresolved

   **Summary** Network traffic to Google Cloud originating from Delhi, Chennai, Mumbai and surrounding areas is experiencing intermittent periods of elevated latency and possible packet loss. **Description** A fire at a third-party data center facility required an emergency power shutdown of networking equipment, isolating a non-compute local Point of Presence (POP) in Delhi and reducing available network capacity in the metro area. We rerouted significant traffic from the impacted facility in Delhi to address reduced local serving capabilities. As a result, a subset of Hybrid Connectivity and Virtual Private Cloud (VPC) customers may be impacted by the routing changes made to address reduced local, latency-optimized serving capabilities in Delhi. Affected customers may experience intermittent latency spikes due to demand exceeding capacity across Indian metros and regional ISPs. We are investigating additional traffic mitigations and Internet Edge peering augmentation to alleviate the latency issues affecting our customers. We are continuing to work with the ISP partners in the region to mitigate any additional impact from unplanned failures. We will provide our next update by Thursday, 2026-06-11 17:00 PDT. **Symptoms** Customers may experience slightly elevated latency and non-optimal network routing into Google Cloud until the affected facility is fully restored. **Workaround** There is no workaround at this time.

3. 11 juin 2026 à 23:25 UTCresolved

   **Summary** Network traffic to Google Cloud originating from Delhi, Chennai, Mumbai and surrounding areas is experiencing intermittent periods of elevated latency and possible packet loss. **Description** A fire at a third-party data center facility required an emergency power shutdown of networking equipment, isolating a non-compute local Point of Presence (POP) in Delhi and reducing available network capacity in the metro area. We rerouted significant traffic from the impacted facility in Delhi to address reduced local serving capabilities. As a result, a subset of Hybrid Connectivity and Virtual Private Cloud (VPC) customers may be impacted by the routing changes made to address reduced local, latency-optimized serving capabilities in Delhi. Affected customers may experience intermittent latency spikes due to demand exceeding capacity across Indian metros and regional ISPs. Initial traffic mitigations have yielded positive results for some Cloud customers. In parallel, we are pursuing additional Internet Edge peering capacity to reduce existing fragility in the metro. We are also optimizing capacity across network backbones to increase additional headroom for VPC customers in the affected region. Additionally, we are planning to augment our local Delhi POP and migrating select peering partners to further increase regional capacity. We will continue to closely monitor latency deviations and packet drops. We will provide our next update by Friday, 2026-06-12 at 17:00 PDT. **Symptoms** Customers may experience slightly elevated latency and non-optimal network routing into Google Cloud until the affected facility is fully restored. **Workaround** There is no workaround at this time.

4. 12 juin 2026 à 22:50 UTCresolved

   **Summary** Network traffic to Google Cloud originating from Delhi, Chennai, Mumbai and surrounding areas is experiencing intermittent periods of elevated latency and possible packet loss. **Description** A fire at a third-party data center facility required an emergency power shutdown of networking equipment, isolating a non-compute local Point of Presence (POP) in Delhi and reducing available network capacity in the metro area. We rerouted significant traffic from the impacted facility in Delhi to address reduced local serving capabilities. As a result, a subset of Hybrid Connectivity and Virtual Private Cloud (VPC) customers may be impacted by the routing changes made to address reduced local, latency-optimized serving capabilities in Delhi. Affected customers may experience intermittent latency spikes due to demand exceeding capacity across Indian metros and regional ISPs. Initial traffic mitigations have yielded positive results for some Cloud customers. In parallel, we are pursuing additional Internet Edge peering capacity to reduce latency in the local Delhi metropolitan area. Further, we are augmenting out-of-region Internet Edge regional peering capacity in Chennai to provide additional load-balancing and redundancy to large ISPs in India (expected to be done by Wednesday, 2026-06-17 PDT). We have optimized capacity across network backbones to increase available headroom. Additionally, we are further augmenting our Delhi backbone capacity (expected to be complete by Monday, 2026-06-15 PDT). We will continue to closely monitor latency deviations and packet drops. We will provide our next update by Monday, 2026-06-15 17:00 PDT. **Symptoms** Customers may experience slightly elevated latency and non-optimal network routing into Google Cloud until the affected facility is fully restored. **Workaround** There is no workaround at this time.

5. 15 juin 2026 à 23:11 UTCresolved

   **Summary** Network traffic to Google Cloud originating from Delhi, Chennai, Mumbai and surrounding areas is experiencing intermittent periods of elevated latency and possible packet loss. **Description** A fire at a third-party data center facility required an emergency power shutdown of networking equipment, isolating a non-compute local Point of Presence (POP) in Delhi and reducing available network capacity in the metro area. We rerouted significant traffic from the impacted facility in Delhi to address reduced local serving capabilities. As a result, a subset of Hybrid Connectivity and Virtual Private Cloud (VPC) customers may be impacted by the routing changes made to address reduced local, latency-optimized serving capabilities in Delhi. Affected customers may experience intermittent latency spikes due to demand exceeding capacity across Indian metros and regional ISPs. Media CDN customers may have experienced higher latencies than normal. Initial traffic mitigations have yielded positive results for some Cloud customers. We have restored a portion of Internet Edge peering capacity to reduce latency in the local Delhi metropolitan area. Further, we are augmenting out-of-region Internet Edge regional peering capacity in Chennai to provide additional load-balancing and redundancy to large ISPs in India (expected to be done by Wednesday, 2026-06-17 PDT). We have optimized capacity across network backbones to increase available headroom. Additionally, we have augmented our Delhi backbone capacity over the weekend. We will continue to closely monitor latency deviations and packet drops. We will provide our next update by Wednesday, 2026-06-17 at 17:00 PDT. **Symptoms** Customers may experience slightly elevated latency and non-optimal network routing into Google Cloud until the affected facility is fully restored. **Workaround** There is no workaround at this time.

6. 17 juin 2026 à 22:36 UTCresolved

   **Summary** Network traffic to Google Cloud originating from Delhi, Chennai, Mumbai and surrounding areas is experiencing intermittent periods of elevated latency and possible packet loss. **Description** Traffic rerouting from the impacted Delhi facility has caused a subset of Hybrid Connectivity and Virtual Private Cloud (VPC) customers to experience intermittent latency spikes as demand exceeds regional capacity. Media CDN customers may also notice increased latencies. Initial traffic mitigations have yielded positive results for some Cloud customers. We have restored a portion of Internet Edge peering capacity to reduce latency in the local Delhi metropolitan area. Further, we are nearly finished with the augmentation of out-of-region Internet Edge regional peering capacity in Chennai to provide additional load-balancing and redundancy to large ISPs in India. We have optimized capacity across network backbones to increase available headroom. Additionally, we have augmented our Delhi backbone capacity. We have also restored additional capacity between Delhi-Chennai and Delhi-Mumbai. We will continue to closely monitor latency deviations and packet drops. We will provide our next update by Monday, 2026-06-22 17:00 PDT. **Symptoms** Customers may experience slightly elevated latency and non-optimal network routing into Google Cloud until the affected facility is fully restored. **Workaround** There is no workaround at this time.

7. 22 juin 2026 à 23:51 UTCresolved

   **Summary** Network traffic to Google Cloud originating from Delhi, Chennai, Mumbai and surrounding areas is experiencing intermittent periods of elevated latency and possible packet loss. **Description** Traffic rerouting from the impacted Delhi facility has caused a subset of Hybrid Connectivity and Virtual Private Cloud (VPC) customers to experience intermittent latency spikes as demand exceeds regional capacity. Media CDN customers may also notice increased latencies. Initial traffic mitigations have yielded positive results for some Cloud customers. We have restored a large portion of Internet Edge peering capacity to reduce latency in the local Delhi metropolitan area. Since the last update, we have completed the augmentation of out-of-region Internet Edge regional peering capacity in Chennai to provide additional load-balancing and redundancy to large ISPs in India. Following safety clearance, teams obtained access to the damaged site and will further restore additional incremental user-facing backbone capacity on Tuesday, 2026-06-23. We have optimized capacity across network backbones to increase available headroom, and augmented our Delhi user-facing backbone capacity. We will continue to closely monitor latency deviations and packet drops. We will provide our next update by Tuesday, 2026-06-23 at 17:00 PDT. **Symptoms** Customers may experience slightly elevated latency and non-optimal network routing into Google Cloud until the affected facility is fully restored. **Workaround** There is no workaround at this time.

8. 23 juin 2026 à 22:52 UTCresolved

   **Summary** Network traffic to Google Cloud originating from Delhi, Chennai, Mumbai and surrounding areas is experiencing intermittent periods of elevated latency and possible packet loss. **Description** Traffic rerouting from the impacted Delhi facility has caused a subset of Hybrid Connectivity and Virtual Private Cloud (VPC) customers to experience intermittent latency spikes as demand exceeds regional capacity. Media CDN customers may also notice increased latencies. Initial traffic mitigations have yielded positive results for some Cloud customers. We have restored a large portion of Internet Edge peering capacity to reduce latency in the local Delhi metropolitan area. Since the last update, we have completed the augmentation of out-of-region Internet Edge regional peering capacity in Chennai to provide additional load-balancing and redundancy to large ISPs in India. Following safety clearance, our team has accessed the damaged site and is continuing to restore additional capacity throughout this week. We have optimized capacity across network backbones to increase available headroom and augmented our Delhi user-facing backbone capacity. We will continue to closely monitor latency deviations and packet drops. We will provide our next update by Monday, 2026-06-29 at 17:00 PDT. **Symptoms** Customers may experience slightly elevated latency and non-optimal network routing into Google Cloud until the affected facility is fully restored. **Workaround** There is no workaround at this time.

9. 29 juin 2026 à 23:06 UTCresolved

   **Summary** Network traffic to Google Cloud originating from Delhi, Chennai, Mumbai and surrounding areas experienced intermittent periods of elevated latency and possible packet loss. **Description** Traffic rerouting from the impacted Delhi facility caused a subset of Hybrid Connectivity, Virtual Private Cloud (VPC) and Media CDN customers to experience intermittent latency spikes as demand exceeded regional capacity. We completed the augmentation of out-of-region Internet Edge regional peering capacity in Chennai to provide additional load-balancing and redundancy to large ISPs in India. Service to a large portion of Internet Edge peering capacity has been restored to reduce latency in the local Delhi metropolitan area. We are now recovered and returned to normal service as of Friday, 2026-06-26 PDT. Following safety clearance, our team restored all lost capacity. We have restored capacity between Delhi-Chennai and Delhi-Mumbai and will continue to closely monitor latency deviations and packet drops. We thank you for your patience during the resolution of this issue. **Symptoms** The impacted customers may have experienced slightly elevated latency and non-optimal network routing into Google Cloud. **Workaround** None

1. 27 février 2026 à 16:12 UTCresolved

   **Description** \ Between Friday, 2026-02-27, 04:36 and 06:45 PST, customers experienced increased error rates when accessing the Vertex Gemini API Global endpoint. The issue impacted API requests to multiple Gemini models. The incident also caused downstream impact to Dialogflow CX, Agent Assist, Google Cloud Support AI agent, and Customer Experience Agent Studio, which rely on Gemini APIs. Preliminary analysis indicates the issue was triggered by a recent configuration change. Service was fully restored after the configuration change was rolled back. We thank you for your patience while we worked on resolving the issue. **Symptom** \ Customers experienced increased error rates when sending API requests to impacted multiple Gemini models through the global endpoint.

2. 4 mars 2026 à 23:23 UTCresolved

   # Preliminary Incident Report We apologize for the inconvenience this service disruption may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. A final Incident Report with preventative actions will be posted once our investigation is complete. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support. ## Date/Time of the Issue (All time US/Pacific) Incident Start: 27 February 2026 04:37 Incident End: 27 February 2026 06:35 Duration: 1 hour, 58 minutes ## Summary On Friday, 27 February 2026 at 04:37 US/Pacific, customers using Vertex AI Gemini API models (including Gemini 2.0, 2.5, and 3.0 previews) experienced increased error rates. Impacted services included Google Cloud Support, Agent Assist, the Vertex Gemini API and Dialogflow CX in US regions and the global endpoint for a duration of 1 hour and 58 minutes. This is not the level of quality and reliability we strive to offer you, and we are taking immediate steps to improve the platform’s performance and availability. ## Preliminary Root Cause This incident was caused by a configuration change to a safety filtering service that supports all Gemini models. This configuration change enabled a code path that interacted poorly with specific requests, leading to service disruption for the safety filtering service. This in turn led to customers seeing overload (429 and 503) errors for their queries. Google engineers have begun a full root cause analysis and will provide additional information once it is available. ## Remediation Google engineers were alerted to the service disruption via automated alert on Friday, 27 February 2026 04:54 US/Pacific and immediately started an investigation. Engineers identified the faulty configuration change for a safety filtering service and initiated a rollback to restore the previous stable configuration. Additionally, engineers added more capacity to the service. Full service restoration was confirmed by 06:35 US/Pacific as the rollback propagated and servers became healthy. ## Description of Impact On Friday, 27 February 2026 between 04:37 and 06:35 US/Pacific, customers accessing Vertex Gemini APIs may have experienced the following: - Affected Models: All Gemini versions were affected, including gemini-2.5-flash, gemini-2.5-flash-lite, gemini-2.5-pro, gemini-3.0-flash-preview, gemini-3.0-pro-preview, gemini-2.0-flash, gemini-2.0-flash-lite. - Error Experience: - PayGo Customers: Experienced primarily 429 Resource Exhausted errors. - Provisioned Throughput (PT) Customers: Received 503 Service Unavailable errors. - For PT customers, most errors stopped at 06:00. For PayGo customers, most errors stopped at 06:20. - Geographic Scope: Global endpoint, us-central1, us-east4, and other US regions were impacted.

3. 9 mars 2026 à 05:25 UTCresolved

   # Incident Report ## Summary On Friday, 27 February 2026 at 04:37 US/Pacific, customers using Vertex AI Gemini API models experienced increased error rates. Impacted services included Google Cloud Support, Agent Assist, Vertex Gemini API and Dialogflow CX in US regions and the global endpoint. The issue persisted for a duration of 1 hour and 58 minutes. This is not the level of quality and reliability we strive to offer you, and we have taken immediate steps to improve the platform’s performance and availability. ## Root Cause This incident was caused by a configuration change to a safety filtering service that supports all Gemini models. For some specific requests, this created code paths that eventually led to service disruptions and capacity loss for the safety filtering service. Consequently, customers encountered overload (429 and 503) errors for their queries, with some users reporting elevated error rates for specific models in US regions. ## Remediation and Prevention Google engineers were alerted to the issue via our automated monitoring system on Friday, 27 February 2026 04:54 US/Pacific and immediately started an investigation. Engineers identified the faulty configuration change and initiated a rollback to restore the previous stable configuration. Engineers also added more capacity to the service to stabilize it. Full service restoration was confirmed by 06:35 US/Pacific as the rollback propagated and servers became healthy. \ \ Google is committed to preventing a repeat of this issue and is taking the following actions: * Reinforcing rollout processes to include mandatory validation checkpoints. * Improving alerting systems to monitor critical dependencies more closely. ## ## Detailed Description of Impact On Friday, 27 February 2026 between 04:37 and 06:35 US/Pacific, customers accessing Vertex Gemini APIs may have experienced the following: * **Affected Models:** All Vertex AI Gemini API models were affected, including gemini-2.5-flash, gemini-2.5-flash-lite, gemini-2.5-pro, gemini-3.0-flash-preview, gemini-3.0-pro-preview, gemini-2.0-flash, gemini-2.0-flash-lite. * **Error Experience:** * **PayGo Customers:** Experienced primarily 429 Resource Exhausted errors. * **Provisioned Throughput (PT) Customers:** Received 503 Service Unavailable errors. * For PT customers, most errors stopped at **06:00**. For PayGo customers, most errors stopped at **06:20**. * **Geographic Scope:** Global endpoint, us-central1, us-east4, and other US regions were impacted.

1. 18 juillet 2025 à 15:54 UTCresolved

   We're currently experiencing elevated latency and error rates for several Cloud services in the us-east1 region, beginning at 7:06 AM PDT today, July 18, 2025. Our initial investigation points to a hardware infrastructure failure as the likely cause. We apologize for any disruption this may be causing. We'll provide an update with more details by 9:15 AM PDT today.

2. 18 juillet 2025 à 16:29 UTCresolved

   Our engineers have confirmed that us-east1-b is partially affected. All other zones in us-east1 are currently operating normally. Our engineers have recovered the failed hardware and are currently recovering the network control plane in the affected zones. We'll provide another update by 10:00 AM US/Pacific, July 18, 2025.

3. 18 juillet 2025 à 16:58 UTCresolved

   Our engineers have successfully recovered the network control plane in the affected us-east1 zones. We're seeing multiple services reporting full recovery, and product engineers are now validating the remaining services. We'll provide another update with more details by 10:30 AM US/Pacific, July 18, 2025.

4. 18 juillet 2025 à 17:32 UTCresolved

   Our engineers have successfully recovered the network control plane in the affected us-east1 zones. We're seeing multiple services reporting full recovery, and product engineers continue to validate the remaining services. We'll provide another update with more details by 11:00 AM US/Pacific, July 18, 2025.

5. 18 juillet 2025 à 18:03 UTCresolved

   The issue has been resolved for all affected products as of 2025-07-18 09:47 US/Pacific. From preliminary analysis, during a routine maintenance of our network in us-east1-b, we experienced elevated packet loss, causing service disruption in the zone. We will publish a full Incident Report with root cause once we have completed our internal investigations. We thank you for your patience while we worked on resolving the issue.

6. 18 juillet 2025 à 22:08 UTCresolved

   # Mini Incident Report We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support or to Google Workspace Support using help article https://support.google.com/a/answer/1047213. (All Times US/Pacific) **GCP Impact start and end time:** 18 July 2025 08:10 - 09:47 **Duration:** 1 hour 37 minutes **GWS Impact start and end time:** 18 July 2025 08:10 - 08:40 **Duration:** 30 minutes **Regions/Zones:** us-east1 **Description:** On Friday, 18 July 2025 08:10 US/Pacific multiple GCP and GWS products experienced elevated latencies and error rates in the us-east1 region for a duration of up to 1 hour and 37 minutes. Based on the preliminary analysis, the root cause of the issue is a procedural error during a planned hardware maintenance in one of our data centers in the us-east1 region. Our engineering team mitigated the issue by draining traffic away from the clusters and then restoring the affected hardware. Google will be completing a full incident report in the following days that will provide a full root cause and preventive actions. **Customer Impact:** The affected GCP and GWS products experienced elevated latencies and errors rates in the us-east1 region. **Affected Products:** **GCP :** AlloyDB for PostgreSQL, Apigee, Artifact Registry, Cloud Armor, Cloud Billing, Cloud Build, Cloud External Key Manager, Cloud Filestore, Cloud HSM, Cloud Key Management Service, Cloud Load Balancing, Cloud Monitoring, Cloud Run, Cloud Spanner, Cloud Storage for Firebase, Cloud Workflows, Database Migration Service, Dialogflow CX, Dialogflow ES, Google BigQuery, Google Cloud Dataflow, Google Cloud Dataproc, Google Cloud Storage, Google Cloud Support, Google Cloud Tasks, Google Compute Engine, Hybrid Connectivity, Media CDN, Network Telemetry, Private Service Connect, Secret Manager, Service Directory, Vertex AI Online Prediction, Virtual Private Cloud (VPC) **Workspace :** Gmail, Google Meet, Google Drive, Google Chat, Google Calendar, Google Groups, Google Doc/Editors, Google Voice **Google SecOps:** Google SecOps SOAR & Google SecOps

7. 22 juillet 2025 à 13:42 UTCresolved

   ## \# Incident Report ## \#\# Summary On Friday, 18 July 2025 07:50 US/Pacific, several Google Cloud Platform (GCP) and Google Workspace (GWS) products experienced elevated latencies and error rates in the us-east1 region for a duration of up to 1 hour and 57 minutes. **GCP Impact Duration:** 18 July 2025 07:50 \- 09:47 US/Pacific : 1 hour 57 minutes **GWS Impact Duration:** 18 July 2025 07:50 \- 08:40 US/Pacific : 50 minutes We sincerely apologize for this incident, which does not reflect the level of quality and reliability we strive to offer. We are taking immediate steps to improve the platform’s performance and availability. ## ## \#\# Root Cause The service interruption was triggered by a procedural error during a planned hardware replacement in our datacenter. An incorrect physical disconnection was made to the active network switch serving our control plane, rather than the redundant unit scheduled for removal. The redundant unit had been properly de-configured as part of the procedure, and the combination of these two events led to partitioning of the network control plane. Our network is designed to withstand this type of control plane failure by failing open, continuing operation. However, an operational topology change while the network control plane was in a failed open state caused our network fabric's topology information to become stale. This led to packet loss and service disruption until services were moved away from the fabric and control plane connectivity was restored. ## \#\# Remediation and Prevention Google engineers were alerted to the outage by our monitoring system on 18 July 2025 07:06 US/Pacific and immediately started an investigation. The following timeline details the remediation and restoration efforts: * **07:39 US/Pacific**: The underlying root cause (device disconnect) was identified and onsite technicians were engaged to reconnect the control plane device and restore control plane connectivity. At that moment, network failure open mechanisms worked as expected and no impact was observed. * **07:50 US/Pacific**: A topology change led to traffic being routed suboptimally, due to the network being in a fail open state. This caused congestion on the subset of links, packet loss, and latency to customer traffic. Engineers made a decision to move traffic away from the affected fabric, which mitigated the impact for the majority of the services. * **08:40 US/Pacific**: Engineers mitigated Workspace impact by shifting traffic away from the affected region. * **09:47 US/Pacific**: Onsite technicians reconnected the device, control plane connectivity was fully restored and all services were back to stable state. Google is committed to preventing a repeat of the issue in the future, and is completing the following actions: * Pause non-critical workflows until safety controls are implemented (complete). * Strengthen safety controls for hardware upgrade workflows by end of Q3 2025\. * Design and implement a mechanism to prevent control plane partitioning in case of dual failure of upstream routers by end of Q4 2025\. ## \#\# Detailed Description of Impact \#\#\# GCP Impact: Multiple products in us-east1 were affected by the loss of network connectivity, with the most significant impacts seen in us-east1-b. Other regions were not affected. The outage caused a range of issues for customers with zonal resources in the region, including packet loss across VPC networks, increased error rates and latency, service unavailable (503) errors, and slow or stuck operations up to loss of networking connectivity. While regional products were briefly impacted, they recovered quickly by failing over to unaffected zones. A small number (0.1%) of Persistent Disks in us-east1-b were unavailable for the duration of the outage: these disks became available once the outage was mitigated, with no customer data loss. \#\#\# GWS Impact: A small subset of Workspace users, primarily around the Southeast US, experienced varying degrees of unavailability and increased delays across multiple products, including Gmail, Google Meet, Google Drive, Google Chat, Google Calendar, Google Groups, Google Doc/Editors, and Google Voice.