Render outage history

Transactional emails for platform events, including but not limited to new account sign-up email confirmations, service build/deploy failure notifications, password resets, etc. are currently failing to send. We are currently addressing the issue.

We have applied a fix and see messages successfully delivering. We are monitoring for any further issues.

This incident has been resolved.

We have identified degraded networking performance linked to an upstream provider - we will continue to provide updates here.

We are continuing to work on a fix for this issue.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

If cloning fails, the deploy will fail. Deploy logs will provide errors related to that failure. Retrying a deploy may succeed. We are currently investigating this issue.

The issue has been identified and a fix is being implemented.

We are continuing to work on a fix for this issue.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

We've identified an issue with an upstream cloud provider that is affecting real-time updates on Render. Since approximately 19:10 UTC: - Dashboard: Deploy status, service events, and service and task run statuses may not update automatically. Refresh the page to see the current state. - SDK calls that wait for a task run to finish may fail with a 500 error after about 60 seconds. The task runs themselves keep running and complete normally. To check a task run's status, use GET /v1/task-runs/{taskRunId}. Deploys, running services, databases, and logs are not affected.

Our upstream provider has resolved the issue, and affected Render services have recovered. We’re continuing to monitor.

This incident has been resolved.

We are currently investigating this issue.

A fix has been implemented and we are monitoring the results.

We are continuing to monitor for any further issues.

We are temporarily pausing deploys to free services.

The issue has been identified and a fix is being implemented.

Builds are failing to start again across all regions, and our team is still working to address the root cause. Deploys for free services remain paused. We'll share another update as soon as we have more information.

A fix has been implemented, and we are monitoring the results. Deploys and builds for services running on free instances continue to be paused.

This incident has been resolved.

### Summary On October 14, 2026, between 12:12 and 15:08 PDT \(19:12 - 22:08 UTC\), builds and deploys across all regions failed to start, stalled, or were significantly delayed during two periods of impact totaling about 2 hours and 20 minutes. The cause was a database schema migration run as part of a planned version upgrade of our internal job scheduler that coordinates builds, deploys, and other critical platform operations. The migration led to high load and a longer-than-expected recovery time of this system’s database. We will change how we perform upgrades and modifications of this system in a highly available fashion in the future so that a similar change cannot cause an outage again. ### Impact There were two periods of impact. During this time, live user services were not affected, but builds and deploys across all regions failed to start, failed to finish, or were significantly delayed. **Period 1: 12:12 - 12:40 PDT \(19:12 - 19:40 UTC\), about 28 minutes** * New builds and deploys did not start. Deploys in progress stalled. * Some requests to create a deploy through the Dashboard, API, and CLI returned errors \(HTTP 500\). **Partial recovery: 12:40 - 13:18 PDT \(19:40 - 20:18 UTC\)** * Builds and deploys ran, but deploys took about 20 seconds to start instead of a few hundred milliseconds. **Period 2: 13:18 - 15:08 PDT \(20:18 - 22:08 UTC\), about 1 hour 50 minutes** * Builds and deploys were delayed, stalled in progress \(including in pre-deploy\), or failed. From 13:51 PDT, almost no new builds were able to start. * Some queued deploys did not start after recovery, and cancelling them through the API had no effect. Users needed to retry these deploys. ### Root Cause Render uses an internal job scheduler system to coordinate builds, deploys, and other critical platform operations. We were rolling out improvements to how it distributes work, and began a planned version upgrade of this system in production after a successful practice run in a test environment. The upgrade included a database schema migration that locked a large table and drove database CPU utilization to 100%. This database stores both the system’s core job state and secondary data used for search and reporting. The system could not read or write job state reliably, so builds and deploys stopped or slowed. Engineers were alerted to the issue quickly and reverted the upgrade. However, three factors extended the impact: * A pre-upgrade step to create a database clone for rollback could not be completed due to an error from the underlying cloud provider, so a fast rollback was not available. * The migration continued to run on the database after the client performing the upgrade was stopped and the upgrade reverted. It was found and terminated at 13:49 PDT, about 1.5 hours after it started. * An attempt to add capacity to the database failed due to an error from the underlying cloud provider. ### Timeline All times are PDT on 2026-10-14 \(UTC-7\). * **12:13** - The upgrade is started, including the database schema migration. Database load rises and builds and deploys stop starting. * **12:24** - Impact to builds and deploys identified. Incident escalated. * **12:33** - Status page incident opened. * **12:34** - Engineers revert the upgrade and stop the client performing the schema migration. * **12:37** - Job scheduler system recovers. Builds and deploys resume, with slower deploy start times. End of Period 1. * **13:17** - Engineers restart one unhealthy instance of the system to rebalance load. Database connections and errors increase across the system. Start of Period 2. * **13:37** - Free-tier deploys paused to reduce load. * **13:49** - Engineers find the schema migration still running on the database and terminate it. * **14:16** - Database restarted. An attempt to add database capacity fails because of an error from the underlying cloud provider. * **14:27** - Remaining components of the system restarted. * **14:30** - Builds and deploys start to recover, with slower than usual build and deploy times as the backlog is processed. * **15:08** - Deploy start times return to normal and database load returns to pre-incident levels. End of Period 2. * **15:11** - Free-tier deploys and spin-ups gradually re-enabled. * **15:18** - Incident resolved. ### Mitigations **Completed during the incident** * Reverted the job scheduler component upgrade and configuration change \(12:34 PDT\). * Paused free-tier deploys and spin-ups to reduce load, then re-enabled them gradually after recovery. * Terminated the schema migration that was still running on the database \(13:49 PDT\). * Restarted the database and the job scheduler system to restore normal operation. * Paused the upgrade in production until the upgrade process is changed. **Ongoing \(started during the incident\)** * This database is being upgraded to support cross-zone High Availability. **Planned** * Implement a new upgrade and deployment practice for this component: we will implement blue-green deploys to avoid upgrading this system under load in the future. * Adjust the underlying storage mechanisms for the job scheduler to be better suited to this workload type, after we have implemented blue-green deploys.

We are currently investigating this issue.

We are continuing to investigate this issue.

Impacted components are recovering. We are monitoring the situation.

This incident has been resolved.

We are currently investigating this issue.

Our upstream provider has begun recovering. Services in Oregon are improving, and we're continuing to monitor.

All services, except for the free web services tier, have recovered. We continue to monitor.

This incident has been resolved.

We are currently investigating this issue.

Builds and deploys failed for a subset of services in Oregon. Services using a prebuilt Docker image and static sites were unaffected.

Around 23:31 UTC on 2026-09-23, Key Value services created prior to 2024-03-05 were inadvertently restarted due to a configuration change on our end. No further restarts are planned or expected as a result of that change.

We are currently investigating this issue.

We are continuing to investigate this issue.

The issue has been identified and a fix is being implemented.

This incident has been resolved.

We are currently investigating this issue.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

Some web services in Oregon were unavailable between 19:20 and 19:30 UTC.

We are investigating service instability affecting some web services in our Singapore region. Customers may experience unexpected service restarts or intermittent availability. Our team is actively investigating and working to restore normal operation. We will provide an update as more information becomes available.

We've observed that deploys are primarily affected by this.

Inbound traffic to Singapore services is impacted. Free service spin-ups (from idle) in Singapore have been temporarily disabled while we continue to investigate.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

We are currently investigating this issue.

An issue has been identified with our upstream provider, Google Cloud. Free Web Services: Builds/deploys and spin-up have been temporarily disabled while we work to mitigate the issue.

We are observing recovery for builds and deploys in all regions. Builds, deploys, and spin-up for free web services have been re-enabled. Some customers may continue to see elevated latency while recovery progresses. We are continuing to monitor.

We have observed full recovery. We are continuing to monitor for any further issues.

This incident has been resolved.

Requests through the dashboard may be slower than usual. We're currently investigating the issue.

We are continuing to investigate this issue.

The REST API is also experiencing degraded performance. As a result, requests to it may experience timeouts and you may need to retry your request. We are continuing to investigate.

We've implemented a fix and are monitoring the results. Deploy failure reasons will be temporarily unavailable. Deploy failure events will still be generated. We will provide an update once deploy failure reasons are re-enabled.

No further issues have been observed since our fixes. The Dashboard and REST API remain fully operational. Deploy failure reasons are still disabled. We will update this incident once they are re-enabled. There are no further updates expected for today (2026-07-27, UTC-7).

We've continued to observe no further issues and deploy failure reasons have been re-enabled.

We are investigating an infrastructure issue affecting services hosted in our Oregon region. Some services may be unreachable or experience failed requests. Our engineering team is actively working to identify the cause and restore normal operation.

We've observed recovery across all services and are now monitoring the situation.

We believe the issue was caused by an upstream provider. We're continuing to monitor the situation.

This incident has now been resolved.

We are currently investigating this issue.

GitHub continues to report an outage, https://www.githubstatus.com. This is causing authentication failures when trying to deploy from a GitHub repository.

We are now seeing a recovery in our build failure rate. We are continuing to monitor the issue.

Builds failure rates have recovered. Resolving.

We have received an elevated number of reports about git clone failures resulting in a 403 error, aborting the deployment. We are investigating.

Services failing to deploy are those deployed in Frankfurt. Running services are not affected, only the deploy process is failing. We are still working to determine the cause of this error for those services.

Users with services hosted in Frankfurt that rely on GitHub repositories for deployments continue to see 403 errors in their logs causing the deployment to fail. We are continuing to work on this issue within our own systems and have reached out to GitHub.

We are continuing to work towards mitigating this issue, while also working with GitHub to resolve these deployment failures.

Deployments have been consistently completing since approximately 08:40, and are returning to normal levels. We are monitoring for any other impacts.

This incident has been resolved.

We are currently investigating this issue.

We are continuing to investigate this issue.

GitHub has updated its status page to report degraded REST API availability: https://www.githubstatus.com/incidents/gxycch3076xk

GitHub reports that they have mitigated their incident and are monitoring to ensure stability.

This incident has been resolved.

Some users enrolled in 2FA report login loops when attempting to access the Render dashboard. We're investigating.

The issue has been identified and a fix is being implemented.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.