Expo outage history

The issue has been identified and a fix is being implemented.

This incident has been resolved.

We're seeing an increased rate of error response statuses coming from App Store Connect API ("500 Internal Server Error").

This incident has been resolved.

Around 6:30 AM California Time, our API server began experiencing elevated error rates for EAS submissions and build processing. We've identified the issue and are implementing a fix.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

We're experiencing an elevated level of API errors and are currently looking into the issue.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

Due to high database load, the Expo website and several services including submitting CI/CD jobs, build jobs, and store submissions were partially unavailable. The issue has subsided as of 1:30PM PT and we are continuing to diagnose the root cause and monitor the health of the services.

The primary database was under high CPU load from 1:11PM-1:29PM PT. The impact of this is that our website and API were partially unavailable \(more precisely, at 80-90% availability\) during that time. ### Scope What was affected: This affected most services including the website, builds, submissions, CI/CD jobs, and publishing updates. What wasn’t affected: Serving updates to end users maintained 100.000% availability. App performance measurements sent to Observe were also unaffected. ### Root cause analysis An internal orchestrator service used to update the state of CI/CD jobs, including build jobs and store submission jobs, was updated to retry upon application-level failures. We are investigating more thoroughly and currently believe this led to a cascade of retries that caused more failures once the database was under more load than it could handle. ### Remediation The commit that changed retry behavior was rolled back and we are clearing the queue of retries. For a more robust solution, the internal orchestrator must gracefully handle backpressure from the application server and database, and to apply more backoff.

We are currently investigating this issue.

The issue has been identified and a fix is being implemented.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

We're again seeing an elevated rate of iOS submissions failing. Our logs still point to a failing App Store Connect API (502 bad gateway response status code).

The errors from the 3rd party API have stopped. Submissions look healthy!

This incident has been resolved.

We're seeing an elevated rate of iOS submissions failing. Our logs point to a failing App Store Connect API (502 bad gateway response status code).

It looks like the 3rd party API is healthy again. We're going to keep monitoring.

This incident has been resolved.

We are currently investigating this issue.

The issue has been identified and a fix is being implemented.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

We are investigating a backlog in the iOS queue for Expo Notifications, resulting in delayed sending of notifications and generation of push receipts.

This incident has been resolved.

We are currently investigating this issue

This incident has been resolved.

Users may experience extended queue times or failed submissions due to an issue with downloading build artifacts. This is related to a current outage with R2 storage https://www.cloudflarestatus.com/incidents/ftvf8c3m4mv5

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

Queue times for Android builds and other Linux workflows are currently elevated due to GCP regional capacity constraints. We are currently enabling capacity in additional regions. Jobs in queue are continuing to progress.

Queue times for Android builds and other Linux workflows are currently elevated due to upstream provider issues affecting the provisioning of VM's. GCP has confirmed the issue and is working on resolving.

The upstream issue with creating VM's has been resolved by GCP, Linux worker queues are draining, and queue times are returning to expected levels.

This incident has been resolved.

We are currently monitoring degraded performance on the NPM registry. The degradation may cause package installs to time out, and EAS Builds to fail when using NPM. We will provide updates from NPM when available.

This incident has been resolved. NPM has restored registry performance. Package installs and EAS Builds using NPM should now complete as expected.

Maintenance on the cache instance serving Maven artifacts to Android builds has rendered the cache unavailable. We're implementing fixes.

This incident has been resolved.

Queue times are elevated for Linux workers for EAS Build and Workflows. Android build jobs may be delayed. We are currently investigating the issue.

We have determined that some of our package caches are under high load, and this is causing elevated build times and in turn causing higher Linux worker queue times. We are currently working on a fix. In-progress builds are continuing to complete and enqueued builds are continuing to be processed.

A fix was deployed at 2:50pm PT/5:50pm ET. The job queue has been running as expected. The root cause of this incident is that a new version of a cache needed more vCPUs than it was given, and we have increased the CPU count as the immediate fix along with additional monitoring for CPU usage on these cache instances.

We are seeing problems in our cloud provider which cause API and website availability disruption on top of delays when processing GitHub events.

The issue has been identified and a fix is being implemented.

As of 10:06am PT we rescheduled our services to run outside of the affected availability zone that is still having some networking issues. In short, Expo's services are no longer affected but we are monitoring the cloud provider's own status updates until the incident is fully resolved and we can reschedule services across all availability zones again.

Our cloud provider's network is fully operational in all of the availability zones Expo uses.

We are experiencing availability problems with our background job processing. This affects broad set of operations across EAS. We are investigating the issue and will provide updates as soon as we have more information.

A fix has been implemented and we are monitoring the systems as they are recovering.

Systems have mostly recovered.

This incident has been resolved.

Users may experience errors from Apple upon uploading to App Store Connect, resulting in failed submissions. We are continuing to monitor.

This incident has been resolved.

User's may experience iOS submissions delayed with long build uploads to App Store Connect. Apple’s API is reporting prolonged pending or processing states, which will increase submission queue times

This incident has been resolved.