Sauce Labs driftsforstyrrelser

1. 2. september 2026 kl. 09.39 UTCinvestigating

   Vi ser forhøjede fejlrater for Real Device tests i EU- Central-1 datacenter. Vi efterforsker det.

2. 2. september 2026 kl. 11.30 UTCidentified

   Den grundlæggende årsag til de forhøjede fejlrater for Real Device test i EU- Central-1 datacenter er blevet identificeret. Der er foretaget en sanering, og vi overvåger aktivt situationen.

3. 2. september 2026 kl. 11.53 UTCresolved

   Efter at have taget afhjælpende foranstaltninger, Real Device test fejlrater er vendt tilbage til normal i EU-Central-1 datacenter. Alle tjenester er fuldt operationelle.

Automatisk oversat fra den officielle hændelsesopdatering.

1. 25. august 2026 kl. 12.33 UTCinvestigating

   Vi ser i øjeblikket forhøjede fejlrater for macOS og iOS tests i US- West-1 datacenter. Vi er ved at undersøge sagen.

2. 25. august 2026 kl. 13.19 UTCresolved

   Efter at have taget afhjælpende foranstaltninger, er alle macOS og iOS test nu udføres som forventet i US- West-1 Data center. Alle tjenester er fuldt operationelle.

3. 3. september 2026 kl. 20.23 UTCpostmortem

   # # * * Datoer: * * Tirsdag den 25. august 2026, 11: 38 - 13: 10 UTC Hvad skete der? Kunder, der kører macOS og iOS tests i vores US- West region oplevet forringet service. Omkring 50% af den virtuelle Mac kapacitet i regionen stoppede acceptere nye tests, så test enten i kø eller undladt at starte. Resterende kapacitet kom under yderligere pres, da arbejdet skiftede til det, som udvidede starttider for både desktop og simulator test. Hvorfor skete det? En intern sikkerhed certifikat, der anvendes af vores Mac værter til at nå en understøttende cloud-service nåede sin udløbsdato. Når det er udløbet, værterne kunne ikke længere etablere en betroet forbindelse til denne tjeneste og stoppede levering af nye testmaskiner. Certifikatet var blevet udstedt manuelt og havde hverken automatiseret fornyelse eller udløbsvarsling. Hvordan vi ordnede det: Vi har udstedt og indsat en erstatning certifikat, som genoprettet forbindelse og returneret Mac kapacitet til normale niveauer. Hvad vi gør for at forhindre det i at ske igen: Vi flytter disse certifikater til automatiseret fornyelse og tilføjer alarmering, så de er erstattet godt før udløb, sammen med gennemgang af det omgivende værktøj til at gøre certifikathåndtering sikrere.

Automatisk oversat fra den officielle hændelsesopdatering.

1. 7. august 2026 kl. 12.08 UTCinvestigating

   Vi ser i øjeblikket forhøjede fejlrater for virtuelle desktop tests i US- West-1 datacenter. Vi er ved at undersøge sagen.

2. 7. august 2026 kl. 13.06 UTCresolved

   After taking remedial action, Virtual Desktop tests are starting successfully in the US-West-1 Data center. All services are fully operational. We are closely monitoring the situation

3. 14. august 2026 kl. 16.43 UTCpostmortem

   ### **Dates:** Friday, August 7th 2026, 11:00 UTC - 13:04 UTC. ### **What happened:** Windows and Intel Mac jobs in `us-west1` failed to start due to virtual machine \(VM\) allocation starvation. ### **Why it happened:** A service crash loop left VMs in an allocated but unclaimed state, while a cleanup bug prevented the system from releasing the orphaned capacity to boot new VMs. ### **How we fixed it:** Restored service stability and cleared stale allocations to resume VM provisioning and clear queued jobs. ### **What we are doing to prevent it from happening again:** Fixing allocator cleanup logic, strengthening deployment health checks, and improving capacity accounting for stale allocations.

Automatisk oversat fra den officielle hændelsesopdatering.

1. 16. juli 2026 kl. 20.33 UTCresolved

   Mellem 16. juli 13: 03 UTC og 16. juli 17: 33 UTC oplevede vi en afbrydelse af tjenesten, hvor symbolarkiver, der benytter flerdelte uploadprotokoller, ikke kunne behandles. Hændelsen er blevet løst, og alle tjenester er operationelle.

2. 14. august 2026 kl. 16.48 UTCpostmortem

   ### **Dates:** Thursday July 16th 2026, 13:03 – 17:33 UTC ### **What happened:** Symbol archives uploaded in multiple parts failed to process. All regions were affected. ### **Why it happened:** Our symbol processing service sent an upload-verification field that our cloud storage provider's API does not accept for multi-part uploads, so those uploads were rejected. A fix for this had already been developed, but it had not yet been included in a released build and the service was not configured to use it. As in the first incident, rejected uploads were retried and accumulated on local disk. ### **How we fixed it:** We deployed a build containing the fix and corrected the service configuration on the affected workers. A large backlog of uploads then processed, which briefly re-filled the disks before draining completely. ### **What we are doing to prevent it from happening again:** We are releasing the fix formally through our build pipeline and persisting the corrected configuration in our configuration management, so it can't be lost. The disk-utilization alerting added after the first incident also covers the disk-exhaustion pattern common to both.

Automatisk oversat fra den officielle hændelsesopdatering.

1. 16. juli 2026 kl. 20.30 UTCresolved

   Mellem 14. juli 23: 45 UTC og 15. juli 23: 05 UTC, oplevede vi et problem, hvor symbol arkiv uploads til Backtrace projekter mislykkedes med HTTP 400 fejl. Hændelsen er blevet løst, og alle tjenester er operationelle.

2. 14. august 2026 kl. 16.47 UTCpostmortem

   ### **Dates:** Tuesday July 14th 2026, 23:45 UTC – Wednesday July 15th 2026, 23:05 UTC ### **What happened:** Symbol archive uploads to Error Reporting \(Backtrace\) projects failed with HTTP 400 errors. All regions were affected. ### **Why it happened:** The credentials our symbol processing service used to write to cloud storage were no longer valid, so uploads could not be stored. Failed uploads were retried repeatedly and accumulated on local disk until the service ran out of space, at which point it also began rejecting new uploads. ### **How we fixed it:** We reissued the storage credentials, increased disk capacity on the affected workers, and restarted the service. The queued uploads then processed successfully, and we confirmed recovery with affected customers. ### **What we are doing to prevent it from happening again:** We've added disk-utilization monitoring and alerting to this service so we detect the condition ourselves before it affects uploads, rather than relying on customer reports.

Automatisk oversat fra den officielle hændelsesopdatering.

1. 14. juli 2026 kl. 17.19 UTCinvestigating

   Vi oplever et problem i EU Central 1 datacenter, hvor iOS ARM Simulator app tests mislykkes med infrastrukturfejl. Vi efterforsker det.

2. 14. juli 2026 kl. 18.05 UTCinvestigating

   App downloads tager længere tid end forventet, når disse infrastrukturfejl ses. Vi fortsætter med at undersøge.

3. 14. juli 2026 kl. 18.56 UTCmonitoring

   Vi har udsendt et fix for dette spørgsmål. Test bør nu udføres som forventet i EU 's centrale datacenter. Vi overvåger.

4. 14. juli 2026 kl. 19.11 UTCmonitoring

   Vi fortsætter med at overvåge for yderligere spørgsmål.

5. 14. juli 2026 kl. 20.23 UTCresolved

   Efter at have truffet afhjælpende foranstaltninger og overvåget situationen, vi ser iOS ARM applikationstest udføre som forventet i EU Data Center. Alle tjenester er fuldt operationelle.

6. 29. juli 2026 kl. 17.40 UTCpostmortem

   # # * * Datoer: * * Tirsdag den 14. juli 2026, 09: 00 - 18: 28 UTC Hvad skete der? iOS ARM test kører i vores EU datacenter mislykkedes. Hvorfor skete det? Der opstod en fejl hos vores primære netværksudbyder i vores EU-datacenter. Hvordan vi ordnede det: Vi fejlede over til en sekundær netværksudbyder fra vores EU datacenter. Hvad vi gør for at forhindre det i at ske igen: Vi har forbedret vores overvågning og alarmering for at fange problemer med tredjeparts netværksudbydere.

Automatisk oversat fra den officielle hændelsesopdatering.

1. 1. juli 2026 kl. 21.37 UTCresolved

   Den 1. juli mellem 17: 02 UTC og 21: 31 UTC, macOS 14 test var ikke at starte i US- West og EU-Central datacentre. Dette spørgsmål er blevet løst. Alle tjenester er fuldt operationelle.

2. 11. august 2026 kl. 20.37 UTCpostmortem

   # # * * Datoer: * * Onsdag den 1. juli 2026, 17: 02 UTC - 21: 31 UTC Hvad skete der? MacOS 14 tests i USA West og EU Central datacentre var ude af stand til at starte. Hvorfor skete det? En intern database var ikke tilgængelig på grund af en manglende konfiguration indgang. Hvordan vi ordnede det: Den manglende post blev erstattet. Hvad vi gør for at forhindre det i at ske igen: Der er blevet indført beskyttelsesforanstaltninger for at forhindre, at de anvendte data fjernes fra konfigurationen.

Automatisk oversat fra den officielle hændelsesopdatering.

1. 1. juli 2026 kl. 12.12 UTCinvestigating

   Vi oplever et problem med adgang til Appium Inspector i US- West-1, EU- Central-1, og US- East-4 datacentre. Vi efterforsker det.

2. 1. juli 2026 kl. 12.58 UTCresolved

   Vi har identificeret den grundlæggende årsag og har indsat en rettelse til dette spørgsmål. Alle tjenester er fuldt operationelle.

3. 20. juli 2026 kl. 20.34 UTCpostmortem

   # # * * Datoer: * * Onsdag den 1. juli 2026, 09: 37 - 12: 46 UTC Hvad skete der? Funktionen Appium Inspector, der blev brugt under virkelige live-testsessioner, blev ikke tilgængelig efter en planlagt UI implementering. Brugere, der forsøgte at bruge funktionen under en live test blev præsenteret for en 500 fejl, forstyrre deres aktive test session. Automatiske testrørledninger blev ikke påvirket. Hvorfor skete det? En rutinemæssig opgradering af en kerne frontend bibliotek indført en uforenelighed med Appium inspektørs rendering logik. Den tidligere biblioteksversion tolererede mønsteret, men den opdaterede version gjorde det ikke. Spørgsmålet blev ikke fanget før ibrugtagning på grund af utilstrækkelig end-to-end test dækning for denne specifikke funktion. Hvordan vi ordnede det: Vi rullede tilbage UI til den sidst kendte arbejdsversion for at gendanne funktionen straks, og derefter indsat en målrettet fix for uforenelighed. Hvad vi gør for at forhindre det i at ske igen: Vi tilføjer end- to- end test dækning for Appium Inspector funktion for at sikre, at det valideres automatisk før fremtidige installationer.

Automatisk oversat fra den officielle hændelsesopdatering.

1. 24. juni 2026 kl. 16.39 UTCresolved

   Den 24. juni mellem 08: 28 UTC og 15: 32 UTC, oplevede vi Real Device test session fejl påvirker Appium og Access API i USA East, US West og EU datacentre. Dette spørgsmål er blevet løst. Alle tjenester er fuldt operationelle.

2. 22. juli 2026 kl. 20.49 UTCpostmortem

   # # * * Datoer: * * Onsdag den 24. juni 2026, 09: 42 UTC - 15: 45 UTC. Hvad skete der? Real Device test sessioner med Appium og Access API oplevede øgede fejlrater i USA øst, USA vest og EU datacentre. Sessioner var timing ud efter ca. 90 sekunder eller at blive hængende i "Connecting" tilstand, forhindrer dem i at blive lukket. Hvorfor skete det? En produktdefekt blev indført, hvilket får Real Device testsessioner til ikke at starte og forhindre aktive sessioner i at blive lukket. Hvordan vi ordnede det: Rollback til en stabil version. Hvad vi gør for at forhindre det i at ske igen: Forbedre overvågning og alarmering og forbedre validering efter ibrugtagning.

Automatisk oversat fra den officielle hændelsesopdatering.

1. 18. juni 2026 kl. 05.30 UTCinvestigating

   Omkring 3: 51 UTC begyndte vi at opleve lavere iOS-enhed tilgængelighed i EU-Central datacenter. Vores team undersøger aktivt årsagen og arbejder hen imod en løsning.

2. 18. juni 2026 kl. 06.12 UTCmonitoring

   Vi har oplevet enhedens manglende tilgængelighed i vores EU- Central-1 datacenter og har identificeret årsagen hertil. Vi har truffet afhjælpende foranstaltninger og overvåger i øjeblikket.

3. 18. juni 2026 kl. 06.59 UTCresolved

   Efter at have taget afhjælpende foranstaltninger, er vi nu se reelle enheder til rådighed til test i EU-Central-1 datacenter. Alle tjenester er fuldt operationelle.

4. 6. juli 2026 kl. 17.17 UTCpostmortem

   # # * * Datoer: * * Torsdag den 18. juni 2026, 03: 45 UTC - 06: 45 UTC. Hvad skete der? Omkring 7% af iOS-enheder i vores EU datacenter var midlertidigt ikke tilgængelige for kundetest sessioner efter et tab af strøm til rack hosting dem. Hvorfor skete det? Strømkredsløbet fodring den berørte rack overskredet sin kapacitet, og en beskyttende breaker tripped for at beskytte linjen - skære magt til de enheder på rack indtil kredsløbet blev genoprettet. Hvordan vi ordnede det: Det berørte kredsløb blev nulstillet og strøm genoprettet til rack, returnere enheder til kundeservice. Hvad vi gør for at forhindre det i at ske igen: Vi omfordeler strømbelastningen på tværs af berørte stativer, tilføjer kapacitetskontrol med tidlige advarsler forud for kredsløbsgrænser, og indfører en kapacitetskontrol, før nye enheder sættes på en rack.

Automatisk oversat fra den officielle hændelsesopdatering.

1. 1. juni 2026 kl. 14.02 UTCinvestigating

   We are experiencing an issue where the US-EAST data center is currently unavailable. We are investigating.

2. 1. juni 2026 kl. 14.55 UTCinvestigating

   We are continuing to investigate this issue.

3. 1. juni 2026 kl. 15.30 UTCinvestigating

   We have identified the root cause and are working on implementing a fix.

4. 1. juni 2026 kl. 16.02 UTCmonitoring

   Access has been restored to the US-EAST data center. We are monitoring.

5. 1. juni 2026 kl. 18.24 UTCresolved

   After taking remedial action, access to the US-EAST data center is fully restored. This incident is resolved.

6. 15. juni 2026 kl. 23.32 UTCpostmortem

   ### **Dates:** Monday, June 1st 2026, 13:06 UTC -  15:42 UTC. ### **What happened:** Customers served by the US-EAST-4 region were unable to authenticate or start new test sessions because an incomplete TLS certificate chain was deployed to the core directory services.  ### **Why it happened:** A certificate extraction script defect silently truncated the certificate chain after the certificate authority transitioned to a longer hierarchy.  ### **How we fixed it:** Reverted the certificate rotation and re-applied the previous known ,good certificate to restore authentication services. ### **What we are doing to prevent it from happening again:** Implementing pre-deployment certificate chain validation, adding active monitoring, and fixing the script's chain-length limitations.

1. 21. maj 2026 kl. 14.53 UTCresolved

   Between May 13th 18:47 UTC and May 15th 9:23 UTC, we experienced test details not appearing in the dashboard of the US-East Data Center. This issue has been resolved. All services are fully operational.

2. 26. maj 2026 kl. 15.07 UTCpostmortem

   ### **Dates:** Wednesday, May 13th 2026, 18:47 UTC - Friday, May 15th 2026, 09:23 UTC. ### **What happened:** Customers were unable to access test run summaries for Real Device Cloud \(RDC\) jobs because events stopped publishing to the jobs Kafka topic in US-EAST. ### **Why it happened:** An authentication key used by the message producer unexpectedly lost its permissions during an account cleanup. ### **How we fixed it:** Manually restored the required permissions to re-establish the connection and resume service. ### **What we are doing to prevent it from happening again:** Migrating to a permanent service account, implementing a Dead Letter Queue \(DLQ\) for the jobs Kafka topic, and replaying the missing events to restore customer data.

1. 24. april 2026 kl. 16.33 UTCresolved

   Between April 23rd 22:44 and April 24th 15:25 UTC, there was a technical issue that affected video recordings for tests running on macOS 15 and iOS within our EU and US-West Data Center. We identified the issue and deployed a fix. All systems are now fully operational.

2. 13. maj 2026 kl. 15.05 UTCpostmortem

   ### **Dates:** Thursday, April 23rd 2026, 22:43 UTC - Friday, April 24th 2026, 15:29 UTC ### **What happened:** Video assets were missing for virtual iOS simulator tests on ARM and macOS ARM desktop tests in the US-West and EU data centers. ### **Why it happened:** A product defect was introduced resulting in a screen capture failure. ### **How we fixed it:** We performed a rollback to a stable version. ### **What we are doing to prevent it from happening again:** We are improving monitoring & alerting to enhance our post deployment validation.

1. 16. april 2026 kl. 10.10 UTCresolved

   Between 02:00 and 11:15 CEST, live and automated tests on iOS 17.0 simulators were failing to start in the EU and US-West Data Center. We executed a deployment rollback, which restored services. All systems are now fully operational.

2. 8. maj 2026 kl. 18.42 UTCpostmortem

   ### **Dates:** Thursday, April 16th 2026, 00:00 UTC – 09:15 UTC ### **What happened:** Live and automated tests on iOS 17.0 simulators failed to start in both the EU and US-West data centers. Customers running tests on iOS 17.0 Intel-based simulators were unable to execute their tests for approximately 9 hours. ### **Why it happened:** A deployment introduced an incompatibility affecting iOS 17.0 on Intel-based infrastructure. The issue was not caught prior to release due to insufficient post-deployment test coverage for that specific simulator configuration. ### **How we fixed it:** We performed a rollback to the previous deployment, which restored full iOS 17.0 simulator functionality. ### **What we are doing to prevent it from happening again:** We are reviving and expanding automated post-deployment tests to cover a broader range of simulator configurations, including legacy Intel-based iOS versions, to catch incompatibilities before they reach production.

1. 7. april 2026 kl. 15.02 UTCinvestigating

   We are currently investigating reports of test failures affecting users running tests using SauceCtl in our US-West-1 and EU-Central-1 Data Center. We are investigating.

2. 7. april 2026 kl. 16.13 UTCresolved

   We have identified the root cause and have deployed a fix for this issue. All services are fully operational.

3. 10. april 2026 kl. 21.56 UTCpostmortem

   ### **Dates:** Monday April 7th 2026, ~11:00 – 15:55 UTC ### **What happened:** Some customers experienced 503 errors when running tests via saucectl. The test-composer service was intermittently unavailable, preventing framework-based test execution. ### **Why it happened:** A stale Docker image was deployed to the test-composer service due to a packaging issue that arose during an internal container registry migration. This caused service pods to crash. ### **How we fixed it:** We identified the stale image and redeployed the correct version, restoring the service. ### **What we are doing to prevent it from happening again:** We are hardening our image deployment pipeline and adding validation checks to ensure container registry migrations do not result in stale or incorrect images being deployed to production.

1. 24. marts 2026 kl. 17.36 UTCresolved

   Between 09:32 and 15:13 UTC, we identified a technical issue affecting iOS tests when running with network capture enabled. We've resolved the underlying cause and tests are working as expected. All services are fully operational.

2. 10. april 2026 kl. 23.06 UTCpostmortem

   ### **Dates:** Tuesday, March 24th 2026, 09:32 UTC – 15:13 UTC ### **What happened:** Network calls failed on iOS devices during Real Device Cloud sessions where network capture was enabled. Approximately 12-13% of iOS sessions were affected. Android was not impacted. ### **Why it happened:** A deployment introduced a DNS resolution change that was incompatible with the iOS platform, causing network capture to break. ### **How we fixed it:** Rolled back the deployment to restore service. ### **What we are doing to prevent it from happening again:** Adding synthetic tests to catch network capture regressions before production, and implementing monitoring alerts for faster detection after deployments.

1. 19. marts 2026 kl. 09.51 UTCinvestigating

   Around 4:45 AM UTC we started experiencing lower iOS device availability in the US-West data center. Our team is actively investigating the root cause and working toward a resolution.

2. 19. marts 2026 kl. 10.54 UTCresolved

   This incident has been resolved and our services are fully operational.

3. 10. april 2026 kl. 22.59 UTCpostmortem

   ### **Dates:** Wednesday, March 19 2026, 04:45 UTC - 10:47 UTC. ### **What happened:** Approximately 15% of iOS devices in our US-West data center were temporarily unavailable for customer test sessions due to failed internet connectivity checks. ### **Why it happened:** An automated wireless network optimization feature adjusted transmit power levels on access points serving the affected devices, degrading wireless connectivity and causing devices to fail their availability checks. ### **How we fixed it:** The affected access points were identified and restarted, restoring normal wireless connectivity. ### **What we are doing to prevent it from happening again:** Evaluation of the automated optimization tools and a monitoring improvement.

1. 18. marts 2026 kl. 17.15 UTCresolved

   Between 14:43 and 15:11 UTC on March 13, a small subset of Real Devices (iOS and Android) became unavailable across all our data centers. After taking remedial action, the issue was identified and resolved. All services are fully operational.

2. 10. april 2026 kl. 22.57 UTCpostmortem

   ### **Dates:** Friday, March 13th 2026, 14:43 UTC - 15:11 UTC. ### **What happened:** Real Devices \(iOS and Android\) availability gradually decreased across all data centers. ### **Why it happened:** A product defect was introduced resulting in a small subset of Real Devices \(~10%\) failing to maintain required connectivity. ### **How we fixed it:** Rollback to a stable version. ### **What we are doing to prevent it from happening again:** Improve monitoring & alerting, enhance post deployment validation.

1. 10. marts 2026 kl. 18.46 UTCinvestigating

   We are experiencing device unavailability in the US West 1, EU Central 1, and US East 4 data centers and have found that the issue is caused by a 3rd party service disruption. We are investigating.

2. 10. marts 2026 kl. 23.34 UTCresolved

   This incident has been resolved.

3. 9. april 2026 kl. 21.07 UTCpostmortem

   ### **Dates:** Tuesday March 10th 2026, 17:52 - 23:34 UTC ### **What happened:** The majority of iOS devices across all regions became unavailable. ### **Why it happened:** Apple's [ppq.apple.com](http://ppq.apple.com) app verification endpoint was down, causing internal device monitoring checks to fail, bringing devices offline. ### **How we fixed it:** We temporarily disabled these device monitoring checks. ### **What we are doing to prevent it from happening again:** Improved external monitoring to catch outages of apple’s [ppq.apple.com](http://ppq.apple.com) endpoint, loosened device monitoring to not take down live iOS devices if [ppq.apple.com](http://ppq.apple.com) is down.

1. 6. marts 2026 kl. 23.28 UTCresolved

   Between 21:38 UTC and 23:11 UTC, our virtual iOS and MacOS live and automated device tests were failing to start in the EU Data Center. We executed a deployment rollback, which restored services. All systems are now fully operational.

2. 10. april 2026 kl. 22.53 UTCpostmortem

   ### **Dates:** Friday, March 6th 2026, 21:38 UTC - 23:11 UTC ### **What happened:** During the incident timeline, customers running virtual iOS simulator tests on ARM or macOS ARM desktop tests in the EU Data Center were unable to start new sessions for either live or automated. ### **Why it happened:** There was a sequencing issue on the release of the ARM side disk images in the EU. ### **How we fixed it:** The image reference for the ARM side disk was rolled back to the previous reference to restore service. ### **What we are doing to prevent it from happening again:** The tests that run to validate the image syncing have been completed in each region.