Flyio outage history

We're aware of a problem affecting a subset of Sprites users. We are investigating the source of the issue.

Error rates have decreased. We are continuing to monitor the API health.

This incident has been resolved.

We have observed an upstream network issue in LAX. Connections to some destinations may see elevated latency and packet loss. We're working with our upstream to resolve this issue.

We have put in some temporary mitigations along with our providers. However since the root cause of this issue lies within a bigger upstream transit provider, you may continue to see some elevated latency and connection issues in/around affected regions. We're still working closely with them to resolve the root cause.

We're updating the affected region list to also include SJC since this seems to be a wider upstream issue in US West Coast.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

We are investigating an issue with the background job runner for our API. Actions that require a background job, such as creating apps, assigning IP addresses, or creating/renewing certificates, may fail at this time.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

A configuration update caused temporary failures for incoming HTTP/2 traffic for Fly Machines located on a subset of hosts for a few minutes. This incident has since been resolved. Managed Postgres depends on HTTP/2 and some control plane ops may have been affected as well. However, downstream Postgres connections were unlikely to have been affected by this since they do not use the HTTP2 handler.

Due to an upstream provider, we are seeing ~50% packet loss on a subset of hosts in ORD. Some MPG clusters in ORD are slow to replicate as a result.

Packet loss in ORD is improving and impacted services are recovering; we’re continuing to monitor for intermittent issues

This incident has been resolved.

We saw Sprite deletion jobs failing between 21:18 and 22:05 UTC. This issue has been resolved.

We are investigating networking issues impacting some hosts in GRU (São Paulo, Brazil) region. Some apps in GRU may experience increased latency or packet loss.

Networking performance in GRU has normalized and we are no longer seeing issues. We are continuing to monitor to ensure a full recovery.

We are seeing a recurrance in networking issues in GRU. Some apps in the region may experience increased latency or packet loss. We are working with our upstream networking provider to resolve.

Our upstream provider has implemented a fix. Network performance in GRU has normalized.

This incident has been resolved.

We are currently investigating this issue.

This incident has been resolved.

We are investigating issues with our WireGuard gateways. Some CLI commands like `flyctl ssh console` or `flyctl proxy` may not work at this time. Apps continue to run.

A fix has been implemented and we are monitoring the results.

Our testing and monitoring indicates gateways should be back to normal; if you are still having problem using `flyctl ssh console`, try restarting the `flyctl` agent by `flyctl agent restart`.

This incident has been resolved.

We are currently experiencing some metrics lag on servers in some regions. We are provisioning more metric processing instances to accommodate the backlog and catch up.

All hosts have caught up with metrics and we're monitoring the situation

This is now resolved

We are investigating network issues in the Los Angeles region. Apps may experience higher latency or be unreachable at this time.

Upstream networking issues have resolved.

This incident has been resolved.

A BGP configuration error caused our Anycast DNS to route to some nodes without the proper DNS infrastructure. The issue was temporary and was resolved as soon as we removed that node from BGP.

We have identified an issue causing authentication errors for some operations from `flyctl`. These operations are failing with an error like: `This endpoint no longer accepts legacy OAuth tokens (starting with `fo1_`). Please use a macaroon token (starting with `fm2_`) instead. We have identified the issue and are rolling out a fix

A fix has been deployed and this error should no longer be occurring. We're monitoring to ensure full recovery.

This incident has been resolved.

We had an issue with the ord-0 Fly Kubernetes cluster, and many MPG clusters are failing to restart. Our MPG team is actively working on it.

We are continuing to investigate this issue.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

6PN networking issues between some machines in YYZ during a rollout which was rolled back once we noticed errors. During this time some machines were unable to talk to internal resources like other DBs, other apps or MPG clusters.

New machines may fail to create in ARN because we lack capacity.

The capacity issue in the ARN region has been resolved.

We are working to recover our secrets service after a failed deployment. Apps continue to run, but it is not possible to create new apps or update secrets at this time.

We have failed over the secrets database to a replica, and the Machines API appears healthy now. We are monitoring for any further issues.

This incident has been resolved.

We are currently investigating degraded ipv6 networking on a subset of hosts

We are continuing to investigate this issue.

The issue has been identified and a fix is being implemented.

A fix has been implemented and we are monitoring the results.

We are continuing to monitor for any further issues.

This incident has been resolved.

We are currently investigating app-not-found errors returned by Machines API calls made shortly after creating new applications.

We have identified the issue as failed insertions in a subset of Corrosion batches. These failures trigger retries, which can cause timeouts for other batches.

We’ve applied a mitigation to reduce the impact from Corrosion batch insertion retries and are continuing to monitor while affected nodes catch up.

We’ve deployed an additional mitigation to further reduce Corrosion retry pressure and are seeing improvement; we’re continuing to monitor while remaining affected nodes catch up.

A fix has been implemented and we are monitoring the results

This incident has been resolved.

High CPU pressure on a shared etcd instance is causing lags on MPG creation in the IAD region

Etcd is stable. Services are back to normal.