Histórico de interrupções do Sauce Labs

1. 2 Gwengolo 2026 da 09:39 UTCinvestigating

   Estamos vendo taxas de erro elevadas para testes de dispositivos reais em data center EU-Central-1. Estamos a investigar.

2. 2 Gwengolo 2026 da 11:30 UTCidentified

   A causa raiz das taxas de erro elevadas para os testes de dispositivos reais no centro de dados EU-Central-1 foi identificada. Foi aplicada uma reparação e estamos a acompanhar activamente a situação.

3. 2 Gwengolo 2026 da 11:53 UTCresolved

   Após tomar medidas corretivas, as taxas de erro de teste do dispositivo real retornaram ao normal no centro de dados EU-Central-1. Todos os serviços estão totalmente operacionais.

Traduzido automaticamente da atualização oficial do incidente.

1. 25 Eost 2026 da 12:33 UTCinvestigating

   Atualmente estamos vendo elevadas taxas de erro para os testes macOS e iOS no centro de dados US-West-1. Estamos a investigar.

2. 25 Eost 2026 da 13:19 UTCresolved

   Após tomar medidas corretivas, todos os testes macOS e iOS estão agora funcionando como esperado no Centro de Dados US-West-1. Todos os serviços estão totalmente operacionais.

3. 3 Gwengolo 2026 da 20:23 UTCpostmortem

   Datas: Terça-feira 25 de Agosto de 2026, 11:38 – 13:10 UTC O que aconteceu? Os clientes que executam testes macOS e iOS em nossa região EUA-Oeste experimentaram serviço degradado. Cerca de 50% da capacidade virtual do Mac na região parou de aceitar novos testes, então testes em fila ou não foram iniciados. A capacidade restante veio sob pressão adicional à medida que o trabalho se deslocou para ele, o que estendeu os tempos de início para testes de desktop e simulador. Porque aconteceu Um certificado de segurança interna usado pelos nossos hosts Mac para alcançar um serviço de nuvem de suporte atingiu sua data de validade. Depois de caducado, os hosts não podiam mais estabelecer uma conexão confiável com esse serviço e pararam de fornecer novas máquinas de teste. O certificado foi emitido manualmente e não teve renovação automática nem alerta de expiração. Como o consertamos Nós emitimos e implementamos um certificado de substituição, que restaurou a conectividade e devolveu a capacidade do Mac a níveis normais. O que estamos a fazer para evitar que volte a acontecer Estamos transferindo esses certificados para a renovação automatizada e adicionando alertas para que eles sejam substituídos bem antes da expiração, juntamente com a revisão das ferramentas circundantes para tornar o manuseio do certificado mais seguro.

Traduzido automaticamente da atualização oficial do incidente.

1. 7 Eost 2026 da 12:08 UTCinvestigating

   Estamos vendo taxas de erro elevadas para testes de desktop virtual no centro de dados US-West-1. Estamos a investigar.

2. 7 Eost 2026 da 13:06 UTCresolved

   After taking remedial action, Virtual Desktop tests are starting successfully in the US-West-1 Data center. All services are fully operational. We are closely monitoring the situation

3. 14 Eost 2026 da 16:43 UTCpostmortem

   ### **Dates:** Friday, August 7th 2026, 11:00 UTC - 13:04 UTC. ### **What happened:** Windows and Intel Mac jobs in `us-west1` failed to start due to virtual machine \(VM\) allocation starvation. ### **Why it happened:** A service crash loop left VMs in an allocated but unclaimed state, while a cleanup bug prevented the system from releasing the orphaned capacity to boot new VMs. ### **How we fixed it:** Restored service stability and cleared stale allocations to resume VM provisioning and clear queued jobs. ### **What we are doing to prevent it from happening again:** Fixing allocator cleanup logic, strengthening deployment health checks, and improving capacity accounting for stale allocations.

Traduzido automaticamente da atualização oficial do incidente.

1. 16 Gouere 2026 da 20:33 UTCresolved

   Entre 16 de julho 13:03 UTC e 16 de julho 17:33 UTC, nós experimentamos uma interrupção de serviço onde arquivos de símbolos utilizando protocolos de upload multi-parte não conseguiram processar. O incidente foi resolvido e todos os serviços estão operacionais.

2. 14 Eost 2026 da 16:48 UTCpostmortem

   ### **Dates:** Thursday July 16th 2026, 13:03 – 17:33 UTC ### **What happened:** Symbol archives uploaded in multiple parts failed to process. All regions were affected. ### **Why it happened:** Our symbol processing service sent an upload-verification field that our cloud storage provider's API does not accept for multi-part uploads, so those uploads were rejected. A fix for this had already been developed, but it had not yet been included in a released build and the service was not configured to use it. As in the first incident, rejected uploads were retried and accumulated on local disk. ### **How we fixed it:** We deployed a build containing the fix and corrected the service configuration on the affected workers. A large backlog of uploads then processed, which briefly re-filled the disks before draining completely. ### **What we are doing to prevent it from happening again:** We are releasing the fix formally through our build pipeline and persisting the corrected configuration in our configuration management, so it can't be lost. The disk-utilization alerting added after the first incident also covers the disk-exhaustion pattern common to both.

Traduzido automaticamente da atualização oficial do incidente.

1. 16 Gouere 2026 da 20:30 UTCresolved

   Entre 14 de julho 23:45 UTC e 15 de julho 23:05 UTC, tivemos um problema onde uploads de arquivos de símbolos para projetos Backtrace falharam com erros HTTP 400. O incidente foi resolvido e todos os serviços estão operacionais.

2. 14 Eost 2026 da 16:47 UTCpostmortem

   ### **Dates:** Tuesday July 14th 2026, 23:45 UTC – Wednesday July 15th 2026, 23:05 UTC ### **What happened:** Symbol archive uploads to Error Reporting \(Backtrace\) projects failed with HTTP 400 errors. All regions were affected. ### **Why it happened:** The credentials our symbol processing service used to write to cloud storage were no longer valid, so uploads could not be stored. Failed uploads were retried repeatedly and accumulated on local disk until the service ran out of space, at which point it also began rejecting new uploads. ### **How we fixed it:** We reissued the storage credentials, increased disk capacity on the affected workers, and restarted the service. The queued uploads then processed successfully, and we confirmed recovery with affected customers. ### **What we are doing to prevent it from happening again:** We've added disk-utilization monitoring and alerting to this service so we detect the condition ourselves before it affects uploads, rather than relying on customer reports.

Traduzido automaticamente da atualização oficial do incidente.

1. 14 Gouere 2026 da 17:19 UTCinvestigating

   Estamos experimentando um problema no centro de dados EU Central 1, onde os testes de aplicativos do iOS ARM Simulator falham com erros de infraestrutura. Estamos a investigar.

2. 14 Gouere 2026 da 18:05 UTCinvestigating

   Os downloads de aplicativos estão demorando mais do que o esperado quando esses erros de infraestrutura são vistos. Continuamos a investigar.

3. 14 Gouere 2026 da 18:56 UTCmonitoring

   Implementámos uma solução para esta questão. Os testes deverão agora realizar-se como esperado no centro de dados da EU Central 1. Estamos a monitorizar.

4. 14 Gouere 2026 da 19:11 UTCmonitoring

   Continuamos monitorando quaisquer outros problemas.

5. 14 Gouere 2026 da 20:23 UTCresolved

   Após tomar medidas corretivas e monitorar a situação, estamos vendo testes de aplicativos ARM iOS executarem como esperado no Centro de Dados da UE. Todos os serviços estão totalmente operacionais.

6. 29 Gouere 2026 da 17:40 UTCpostmortem

   Datas: Terça-feira 14 de julho de 2026, 09:00 - 18:28 UTC O que aconteceu? Os testes do iOS ARM em execução no nosso centro de dados da UE falharam. Porque aconteceu Ocorreu uma falha com o nosso principal fornecedor de rede no nosso centro de dados da UE. Como o consertamos Falhámos com um fornecedor secundário de rede do nosso centro de dados da UE. O que estamos a fazer para evitar que volte a acontecer Nós melhoramos nosso monitoramento e alerta para problemas de captura com provedores de rede de terceiros.

Traduzido automaticamente da atualização oficial do incidente.

1. 1 Gouere 2026 da 21:37 UTCresolved

   Em 1o de julho entre 17:02 UTC e 21:31 UTC, os testes do macOS 14 não foram iniciados nos centros de dados EUA-Oeste e UE-Centro. Esta questão foi resolvida. Todos os serviços estão totalmente operacionais.

2. 11 Eost 2026 da 20:37 UTCpostmortem

   ### **Dates:** Wednesday July 1st 2026, 17:02 UTC - 21:31 UTC ### **What happened:** macOS 14 tests in the US West and EU Central data centers were unable to start. ### **Why it happened:** An internal datasource was unavailable due to a missing configuration entry. ### **How we fixed it:** The missing entry was replaced. ### **What we are doing to prevent it from happening again:** Safeguards have been put in place to prevent in-use datasource removal from configuration.

Traduzido automaticamente da atualização oficial do incidente.

1. 1 Gouere 2026 da 12:12 UTCinvestigating

   Estamos tendo um problema com o acesso do inspetor de Appium nos centros de dados US-West-1, EU-Central-1 e US-East-4. Estamos a investigar.

2. 1 Gouere 2026 da 12:58 UTCresolved

   Identificámos a causa raiz e implantámos uma solução para esta questão. Todos os serviços estão totalmente operacionais.

3. 20 Gouere 2026 da 20:34 UTCpostmortem

   Datas: Quarta-feira, 1 de Julho de 2026, 09:37 – 12:46 UTC O que aconteceu? O recurso Appium Inspector, usado durante sessões reais de testes ao vivo, tornou-se indisponível após uma implantação programada de UI. Os usuários que tentaram usar o recurso durante um teste ao vivo foram apresentados com um erro de 500, interrompendo sua sessão de teste ativo. Oleodutos de teste automatizados não foram afetados. Porque aconteceu Uma atualização de rotina de uma biblioteca frontend central introduziu uma incompatibilidade com a lógica de renderização do Inspetor Appium. A versão anterior da biblioteca tolerava o padrão, mas a versão atualizada não tolerava. A questão não foi captada antes da implantação devido à cobertura de teste de ponta a ponta insuficiente para esta característica específica. Como o consertamos Retrocedemos a UI para a última versão de trabalho conhecida para restaurar o recurso imediatamente, e então implantamos uma correção direcionada para a incompatibilidade. O que estamos a fazer para evitar que volte a acontecer Estamos adicionando cobertura de teste de ponta a ponta para o recurso Appium Inspector para garantir que ele seja validado automaticamente antes de futuras implementações.

Traduzido automaticamente da atualização oficial do incidente.

1. 24 Mezheven 2026 da 16:39 UTCresolved

   No dia 24 de junho, entre 08:28 UTC e 15:32 UTC, experimentamos falhas na sessão de teste de dispositivos reais impactando a API Appium e Access nos data centers dos EUA East, US West e EU. Esta questão foi resolvida. Todos os serviços estão totalmente operacionais.

2. 22 Gouere 2026 da 20:49 UTCpostmortem

   Datas: Quarta-feira, 24 de Junho de 2026, 09:42 UTC - 15:45 UTC. O que aconteceu? Sessões de teste de dispositivos reais usando Appium e Access API experimentaram taxas de erro aumentadas nos centros de dados dos EUA East, US West e UE. As sessões foram cronometradas após aproximadamente 90 segundos ou ficando presa no estado de "Conectação", impedindo-os de serem fechadas. Porque aconteceu Um defeito de produto foi introduzido, fazendo com que as sessões de teste do Real Device não começassem e impedissem que as sessões ativas fossem fechadas. Como o consertamos Voltar para uma versão estável. O que estamos a fazer para evitar que volte a acontecer Melhore o monitoramento e alerta e melhore a validação pós-implantação.

Traduzido automaticamente da atualização oficial do incidente.

1. 18 Mezheven 2026 da 05:30 UTCinvestigating

   Por volta das 3:51 AM UTC começamos a experimentar menor disponibilidade de dispositivos iOS no centro de dados EU-Central. Nossa equipe está ativamente investigando a causa raiz e trabalhando para uma resolução.

2. 18 Mezheven 2026 da 06:12 UTCmonitoring

   Temos experimentado indisponibilidade do dispositivo em nosso centro de dados EU-Central-1 e identificamos a causa raiz. Tomámos medidas correctivas e estamos actualmente a acompanhar.

3. 18 Mezheven 2026 da 06:59 UTCresolved

   Depois de tomar medidas corretivas, estamos vendo dispositivos reais disponíveis para testes no centro de dados EU-Central-1. Todos os serviços estão totalmente operacionais.

4. 6 Gouere 2026 da 17:17 UTCpostmortem

   Datas: Quinta-feira 18 de Junho de 2026, 03:45 UTC - 06:45 UTC. O que aconteceu? Aproximadamente 7% dos dispositivos iOS no nosso centro de dados da UE estavam temporariamente indisponível para as sessões de teste do cliente após uma perda de energia para o rack hosting-los. Porque aconteceu O circuito elétrico que alimentava o rack afetado excedeu sua capacidade, e um disjuntor protetor tropeçou para proteger a energia de corte de linha dos dispositivos naquele rack até que o circuito fosse restaurado. Como o consertamos O circuito afetado foi reiniciado e a energia restaurada no rack, devolvendo os dispositivos ao serviço ao cliente. O que estamos a fazer para evitar que volte a acontecer Estamos redistribuindo carga de energia através de racks afetados, adicionando monitoramento de capacidade com alertas de alerta precoce antes dos limites do circuito e introduzindo uma revisão de capacidade antes de novos dispositivos serem implantados em rack.

Traduzido automaticamente da atualização oficial do incidente.

1. 1 Mezheven 2026 da 14:02 UTCinvestigating

   We are experiencing an issue where the US-EAST data center is currently unavailable. We are investigating.

2. 1 Mezheven 2026 da 14:55 UTCinvestigating

   We are continuing to investigate this issue.

3. 1 Mezheven 2026 da 15:30 UTCinvestigating

   We have identified the root cause and are working on implementing a fix.

4. 1 Mezheven 2026 da 16:02 UTCmonitoring

   Access has been restored to the US-EAST data center. We are monitoring.

5. 1 Mezheven 2026 da 18:24 UTCresolved

   After taking remedial action, access to the US-EAST data center is fully restored. This incident is resolved.

6. 15 Mezheven 2026 da 23:32 UTCpostmortem

   ### **Dates:** Monday, June 1st 2026, 13:06 UTC -  15:42 UTC. ### **What happened:** Customers served by the US-EAST-4 region were unable to authenticate or start new test sessions because an incomplete TLS certificate chain was deployed to the core directory services.  ### **Why it happened:** A certificate extraction script defect silently truncated the certificate chain after the certificate authority transitioned to a longer hierarchy.  ### **How we fixed it:** Reverted the certificate rotation and re-applied the previous known ,good certificate to restore authentication services. ### **What we are doing to prevent it from happening again:** Implementing pre-deployment certificate chain validation, adding active monitoring, and fixing the script's chain-length limitations.

1. 21 Mae 2026 da 14:53 UTCresolved

   Between May 13th 18:47 UTC and May 15th 9:23 UTC, we experienced test details not appearing in the dashboard of the US-East Data Center. This issue has been resolved. All services are fully operational.

2. 26 Mae 2026 da 15:07 UTCpostmortem

   ### **Dates:** Wednesday, May 13th 2026, 18:47 UTC - Friday, May 15th 2026, 09:23 UTC. ### **What happened:** Customers were unable to access test run summaries for Real Device Cloud \(RDC\) jobs because events stopped publishing to the jobs Kafka topic in US-EAST. ### **Why it happened:** An authentication key used by the message producer unexpectedly lost its permissions during an account cleanup. ### **How we fixed it:** Manually restored the required permissions to re-establish the connection and resume service. ### **What we are doing to prevent it from happening again:** Migrating to a permanent service account, implementing a Dead Letter Queue \(DLQ\) for the jobs Kafka topic, and replaying the missing events to restore customer data.

1. 24 Ebrel 2026 da 16:33 UTCresolved

   Between April 23rd 22:44 and April 24th 15:25 UTC, there was a technical issue that affected video recordings for tests running on macOS 15 and iOS within our EU and US-West Data Center. We identified the issue and deployed a fix. All systems are now fully operational.

2. 13 Mae 2026 da 15:05 UTCpostmortem

   ### **Dates:** Thursday, April 23rd 2026, 22:43 UTC - Friday, April 24th 2026, 15:29 UTC ### **What happened:** Video assets were missing for virtual iOS simulator tests on ARM and macOS ARM desktop tests in the US-West and EU data centers. ### **Why it happened:** A product defect was introduced resulting in a screen capture failure. ### **How we fixed it:** We performed a rollback to a stable version. ### **What we are doing to prevent it from happening again:** We are improving monitoring & alerting to enhance our post deployment validation.

1. 16 Ebrel 2026 da 10:10 UTCresolved

   Between 02:00 and 11:15 CEST, live and automated tests on iOS 17.0 simulators were failing to start in the EU and US-West Data Center. We executed a deployment rollback, which restored services. All systems are now fully operational.

2. 8 Mae 2026 da 18:42 UTCpostmortem

   ### **Dates:** Thursday, April 16th 2026, 00:00 UTC – 09:15 UTC ### **What happened:** Live and automated tests on iOS 17.0 simulators failed to start in both the EU and US-West data centers. Customers running tests on iOS 17.0 Intel-based simulators were unable to execute their tests for approximately 9 hours. ### **Why it happened:** A deployment introduced an incompatibility affecting iOS 17.0 on Intel-based infrastructure. The issue was not caught prior to release due to insufficient post-deployment test coverage for that specific simulator configuration. ### **How we fixed it:** We performed a rollback to the previous deployment, which restored full iOS 17.0 simulator functionality. ### **What we are doing to prevent it from happening again:** We are reviving and expanding automated post-deployment tests to cover a broader range of simulator configurations, including legacy Intel-based iOS versions, to catch incompatibilities before they reach production.

1. 7 Ebrel 2026 da 15:02 UTCinvestigating

   We are currently investigating reports of test failures affecting users running tests using SauceCtl in our US-West-1 and EU-Central-1 Data Center. We are investigating.

2. 7 Ebrel 2026 da 16:13 UTCresolved

   We have identified the root cause and have deployed a fix for this issue. All services are fully operational.

3. 10 Ebrel 2026 da 21:56 UTCpostmortem

   ### **Dates:** Monday April 7th 2026, ~11:00 – 15:55 UTC ### **What happened:** Some customers experienced 503 errors when running tests via saucectl. The test-composer service was intermittently unavailable, preventing framework-based test execution. ### **Why it happened:** A stale Docker image was deployed to the test-composer service due to a packaging issue that arose during an internal container registry migration. This caused service pods to crash. ### **How we fixed it:** We identified the stale image and redeployed the correct version, restoring the service. ### **What we are doing to prevent it from happening again:** We are hardening our image deployment pipeline and adding validation checks to ensure container registry migrations do not result in stale or incorrect images being deployed to production.

1. 24 Meurzh 2026 da 17:36 UTCresolved

   Between 09:32 and 15:13 UTC, we identified a technical issue affecting iOS tests when running with network capture enabled. We've resolved the underlying cause and tests are working as expected. All services are fully operational.

2. 10 Ebrel 2026 da 23:06 UTCpostmortem

   ### **Dates:** Tuesday, March 24th 2026, 09:32 UTC – 15:13 UTC ### **What happened:** Network calls failed on iOS devices during Real Device Cloud sessions where network capture was enabled. Approximately 12-13% of iOS sessions were affected. Android was not impacted. ### **Why it happened:** A deployment introduced a DNS resolution change that was incompatible with the iOS platform, causing network capture to break. ### **How we fixed it:** Rolled back the deployment to restore service. ### **What we are doing to prevent it from happening again:** Adding synthetic tests to catch network capture regressions before production, and implementing monitoring alerts for faster detection after deployments.

1. 19 Meurzh 2026 da 09:51 UTCinvestigating

   Around 4:45 AM UTC we started experiencing lower iOS device availability in the US-West data center. Our team is actively investigating the root cause and working toward a resolution.

2. 19 Meurzh 2026 da 10:54 UTCresolved

   This incident has been resolved and our services are fully operational.

3. 10 Ebrel 2026 da 22:59 UTCpostmortem

   ### **Dates:** Wednesday, March 19 2026, 04:45 UTC - 10:47 UTC. ### **What happened:** Approximately 15% of iOS devices in our US-West data center were temporarily unavailable for customer test sessions due to failed internet connectivity checks. ### **Why it happened:** An automated wireless network optimization feature adjusted transmit power levels on access points serving the affected devices, degrading wireless connectivity and causing devices to fail their availability checks. ### **How we fixed it:** The affected access points were identified and restarted, restoring normal wireless connectivity. ### **What we are doing to prevent it from happening again:** Evaluation of the automated optimization tools and a monitoring improvement.

1. 18 Meurzh 2026 da 17:15 UTCresolved

   Between 14:43 and 15:11 UTC on March 13, a small subset of Real Devices (iOS and Android) became unavailable across all our data centers. After taking remedial action, the issue was identified and resolved. All services are fully operational.

2. 10 Ebrel 2026 da 22:57 UTCpostmortem

   ### **Dates:** Friday, March 13th 2026, 14:43 UTC - 15:11 UTC. ### **What happened:** Real Devices \(iOS and Android\) availability gradually decreased across all data centers. ### **Why it happened:** A product defect was introduced resulting in a small subset of Real Devices \(~10%\) failing to maintain required connectivity. ### **How we fixed it:** Rollback to a stable version. ### **What we are doing to prevent it from happening again:** Improve monitoring & alerting, enhance post deployment validation.

1. 10 Meurzh 2026 da 18:46 UTCinvestigating

   We are experiencing device unavailability in the US West 1, EU Central 1, and US East 4 data centers and have found that the issue is caused by a 3rd party service disruption. We are investigating.

2. 10 Meurzh 2026 da 23:34 UTCresolved

   This incident has been resolved.

3. 9 Ebrel 2026 da 21:07 UTCpostmortem

   ### **Dates:** Tuesday March 10th 2026, 17:52 - 23:34 UTC ### **What happened:** The majority of iOS devices across all regions became unavailable. ### **Why it happened:** Apple's [ppq.apple.com](http://ppq.apple.com) app verification endpoint was down, causing internal device monitoring checks to fail, bringing devices offline. ### **How we fixed it:** We temporarily disabled these device monitoring checks. ### **What we are doing to prevent it from happening again:** Improved external monitoring to catch outages of apple’s [ppq.apple.com](http://ppq.apple.com) endpoint, loosened device monitoring to not take down live iOS devices if [ppq.apple.com](http://ppq.apple.com) is down.

1. 6 Meurzh 2026 da 23:28 UTCresolved

   Between 21:38 UTC and 23:11 UTC, our virtual iOS and MacOS live and automated device tests were failing to start in the EU Data Center. We executed a deployment rollback, which restored services. All systems are now fully operational.

2. 10 Ebrel 2026 da 22:53 UTCpostmortem

   ### **Dates:** Friday, March 6th 2026, 21:38 UTC - 23:11 UTC ### **What happened:** During the incident timeline, customers running virtual iOS simulator tests on ARM or macOS ARM desktop tests in the EU Data Center were unable to start new sessions for either live or automated. ### **Why it happened:** There was a sequencing issue on the release of the ARM side disk images in the EU. ### **How we fixed it:** The image reference for the ARM side disk was rolled back to the previous reference to restore service. ### **What we are doing to prevent it from happening again:** The tests that run to validate the image syncing have been completed in each region.