Google Workspace Status

# Incident Report ## Summary On Wednesday, June 10, 2026, at 03:30 US/Pacific, an incident occurred affecting the availability of Gemini services across all surfaces, including web, mobile (iOS and Android), and integrations within Google Chrome. The incident resulted in an elevated error rate for approximately 6 hours and 55 minutes, with full service restoration confirmed after a total duration of 14 hours and 49 minutes. We sincerely apologize for the disruption this incident caused to your business. We know how much you rely on Google Cloud, and we regret the impact on your productivity. We are working to address the root cause and prevent this from occurring in the future. ## Root Cause The disruption was caused by extreme read contention within the foundational database service that manages tool deployment metadata. The primary cause was an index design issue within the database where a column used for tracking deployment expirations contained a high volume of rows with similar values. This resulted in "hotspotting," where traffic was concentrated on a very small number of database shards, overwhelming them. Additionally, the service utilized an in-memory cache with a very short Time To Live (TTL) of only one minute, requiring frequent database refreshes. The failure sequence was as follows: 1. **Technical Trigger:** The trigger was an upward spike in the Queries Per Second (QPS) from the frontend. The underlying system was already running close to maximum usage and the incoming QPS crossed a tipping threshold beyond which the underlying spanner started getting overloaded. 2. **Systemic Effect:** This short window, combined with an indexing anomaly where specific metadata fields containing empty values were clustered on a single database partition, led to a "hot shard" condition. 3. **Contention and Latency:** As request volume increased, the internal tool management service experienced a more than 10x surge in database calls because the cache hit rate dropped significantly. 4. **Failure State:** The underlying storage layer reached its capacity for handling incoming queries, and database failure rates skyrocketed to 60%. This led to cache hit rate dropping down to 50%. Automated validation failed to prevent this because the indexing strategy did not sufficiently account for the concentration of data rows when specific identifiers were absent. ## Remediation and Prevention The issue was detected by internal real-time monitoring alerts indicating high processing error rates. Engineering teams performed several targeted actions to restore the service: * **Throttling:** Applied Rate Access Control Lists (RateACLs) to the most affected database shards to allow internal caches to recover. * **Index Redistribution:** Gradually redistributed common values in the database index to random values in a specific target range, effectively spreading the load across multiple shards. * **Cache Policy Update:** Deployed a configuration change to increase the in-memory cache Time To Live (TTL) from 1 minute to 20 minutes, significantly reducing database load. * **Deadline Adjustment:** Shortened the Remote Procedure Call (RPC) deadlines for database queries to prevent request pile-ups during periods of high latency. ## Prevention To prevent a recurrence, Google engineering will be implementing the following actions: * **Index Design Standards:** Restructure of database index to prevent "hotspotting". * **Adaptive Cache Tuning:** Implement more robust cache policies that prevent load amplification during periods of backend stress. * **Enhanced Hotspot Monitoring:** Deploy improved monitoring and alerting specifically designed to detect uneven database load distribution before it impacts user traffic. * **Service Resiliency:** Implement advanced load protection mechanisms, such as smart task resizing and query coalescing for preventing spanner overloads. ## Detailed Description of Impact The impact window occurred between 03:30 US/Pacific and 10:25 US/Pacific on June 10, 2026\. **Gemini App (Web and Mobile):** * High Failure Rates: Users experienced a 50% error rate when attempting to send prompts, receiving "Something went wrong" messages . * Service Unavailability: The issue affected both consumer accounts and enterprise Google Workspace customers . * Enterprise Google Workspace Users encountered disruptions when attempting to attach assets from Google Drive within the web application (gemini.google.com). Affected users observed the "Add from Drive" functionality to be visually disabled and unresponsive following interaction with the "+" prompt. **Integrations:** * **Google Chrome Integration:** The Gemini Side Panel and integrated conversational features encountered persistent timeouts.

# Incident Report ## Summary Starting at 20:15 US/Pacific on Saturday, 30 May 2026, various Google Workspace services encountered service-level disruptions lasting approximately 75 minutes. The scope of impact spanned several products, specifically affecting Gmail, Google Docs, Drive, Calendar, Chat, Meet, and Google Voice. Throughout the duration of this event, users faced obstacles while attempting to initialize applications, in addition to experiencing sporadic instances of elevated latency and request failures. Google offers a sincere apology for the operational interference this issue introduced to your organization. Recognizing the critical nature of Google Cloud to your daily operations, we deeply regret any resulting loss in productivity. Engineering efforts are currently focused on remediating the underlying cause to ensure future stability. ## Root Cause The disruption originated from failures in a critical cluster infrastructure component responsible for Google's service discovery and configuration management systems. This system facilitates resource synchronization, manages shared data access, and coordinates background tasks for numerous Google services. A high volume of rapid configuration changes triggered the regional failure in Workspace Services, preventing Workspace servers from reliably processing traffic. To mitigate the impact, Google Workspace traffic was redirected to alternative regions. ## Remediation and Prevention Google engineers were alerted to the issue via automated monitoring on Saturday, 30 May 2026 at 20:28 US/Pacific and immediately began an investigation. The investigation identified elevated error rates and high latency affecting multiple Google Workspace backend services. Upon investigation our engineering team confirmed errors were primarily originating from some regional servers due to coordination service failures in the region. To mitigate the impact, Google engineers redirected traffic away from the affected servers. Google is committed preventing a repeat of this issue in the future and is completing the following actions: - Prevent recurrence of this incident by rolling out the fix for the latent encryption library issue, which improves the handling of internal log compaction compactions during active file writes, to prevent recurrence of the cluster instability. - Investigating additional resilience in our database layer. ## Detailed Description of Impact On 30 May 2026 from 20:15 to 21:30 US/Pacific, affected customers encountered difficulties while loading applications or experienced failures when performing tasks within the following products: - Gmail - Google Calendar - Google Chat - Google Docs - Google Drive - Google Meet - Google Voice

**Summary** Gmail users experienced delayed delivery of messages sent from the bank.in domains. **Description** Our internal telemetry shows that the issue with Gmail has been resolved for all affected users as of Thursday, 2026-05-21 at 08:47 US/Pacific. Our engineering team is actively monitoring the system to ensure continued stability. **Diagnosis / Customer Symptoms** Customers should expect new messages to be delivered normally, older emails could take some time as the senders continue to retry. **Workaround** This issue is now mitigated.

**Title** NotebookLM customers may have experienced an issue where all premium account tiers are being incorrectly downgraded to free. **Description** The issue with NotebookLM has been resolved for all affected users as of Tuesday, 2026-05-19 21:04 PDT. From preliminary analysis, the issue was triggered by a recent software update resulting in accounts on premium plans being incorrectly recognized as free accounts. The issue was mitigated by restoring the system to a previous stable state, which successfully reinstated the appropriate service tiers for all affected accounts. We thank you for your patience while we worked on resolving the issue. **Symptoms** Customers may have noticed lower resource limits (e.g., hitting limits for Studio artifacts, source count per notebook, etc) and the absence of premium features like advanced sharing and analytics. **Workaround** None at this time.

**Summary** Some Cameyo by Google customers could not start their sessions after rolling out the new channel version. **Description** We were experiencing an issue with Cameyo by Google, beginning on Tuesday, 12 May 2026 09:01 US/Pacific. The issue was mitigated on Tuesday, 12 May 2026 15:16 US/Pacific by restarting the RAP system service, and performing full server reboots. From preliminary investigation, this was caused by a faulty code for logging improvement on the Player VMs. Log files were unexpectedly duplicated until the disk was full leading to the session failures. Some customers were recovered automatically after the channel binaries were rolled back, while the others were required to restart their VMs manually. Currently, the issue is fully mitigated and change is rolled back. We apologise to those who were impacted by this issue. **Customer Symptoms** The customers were experiencing an issue that couldn’t connect to the Player VMs normally. The disk utilization on the affected VMs reaches 99% or even 100%. **Workaround** Not needed because all affected servers have been mitigated.

**Summary** [Resolved] Gmail Android users utilizing Microsoft Exchange Online accounts experienced authentication and synchronization failures. **Description** The issue with Gmail has been resolved in the new Gmail android release (version 2026.05.11) as of Thursday, 2026-05-21 15:00 PDT. Our engineering team identified the issue and released a new version of the Gmail Android application to the Google Play Store, which contains a fix to unblock affected users. If you are currently encountering this issue, please upgrade to the latest version of the Gmail Android app. Users operating on the new app version will be able to successfully log in and sync their accounts. We thank you for your patience while we worked on resolving the issue. **Symptoms** Impacted users might have experienced authentication and sync failures while using Microsoft Exchange Online accounts through Gmail App on Android devices. **Workaround** Update to the latest version of Gmail Android (2026.05.11) in Google Play Store.

**Summary** Gemini conversations may have experienced elevated error rates. **Description** The issue with Gemini has been resolved for all affected users as of Friday, 2026-05-08 11:53 PDT. The issue was triggered due to a combination of insufficient database resources, and application bugs. The database capacity issues were resolved, and further code changes are being made to mitigate the application issue. We thank you for your patience while we worked on resolving the issue. **Symptoms** Affected customers may have experienced "Something went wrong(13)" error message in their Gemini conversations. **Workaround** None at this time.

**Title:** An issue with Classroom is resolved. **Description:** A fix has been deployed, and services have stabilized for all users. The issue with Google Classroom has been resolved for all affected users as of Wednesday, 2026-04-29 13:45 PDT. The issue was due to an internal software migration. Our engineers have mitigated the issue by rolling back the migration for impacted users. We thank you for your patience while we worked on resolving the issue. **Symptoms:** Affected users may have experienced issues accessing the classroom. **Workaround:** The issue is resolved.

**Title:** Mitigated issues with slow loading of AppSheet apps. **Description:** The issue with AppSheet has been resolved for all affected users as of Tuesday, 2026-04-28 08:50 PDT. We thank you for your patience while we worked on resolving the issue. **Symptoms:** Affected customers would have experienced slow loading times and internal server errors for AppSheet apps.

**Summary** Google Workspace users using Google IdP were unable to sign into their accounts using security keys and hardware based passkeys. **Description** The issue with Admin Console has been resolved for all affected users as of Friday, 2026-04-10 07:50 PDT. Suspected CL have been rolled back and we are seeing in our graphs errors rate have gone down. Customers also confirmed that issues have been mitigated. We thank you for your patience while we worked on resolving the issue. **Customer Symptoms** Google Workspace users using Google IdP were unable to sign into their accounts using security keys and hardware based passkeys. **Workaround** Below steps should be performed by Admin as the workaround: * Admin should revoke the key. * Admin to create the verification code to sign-in. * The user should sign-in using Backup verification code and then re-enrolls the security key.