Fortrabbit undefined outage history

Since ~08:00 we see issues with a number of Apps hosted in the EU region. It's about 70 Apps and as far as we can see, there is a DDoS attack going on. It's wide spread and thus hard to get in control.

We are working on a fix, but unfortunately it's not so easy to contain it.

We are still working on it, but ETA yet. Most Apps are online already. But we keep monitoring.

The issue is not 100% resolved. We are still monitoring. As most Apps (except one) are online again, we close this incident for now.

We will upgrade underlying systems for the Dashboard and object-storage on old platform. The Dashboard and object-storage might be unavailable for a few minutes.

We need to roll out some security patch fixes for the EU region. Expect a 30 minute downtime per App, somewhere within the noted time frame. The deploy service will have some additional downtime. Thanks for your understanding.

Maintenance completed

We need to roll out some security patch fixes for the US region. Expect a 30 minute downtime per App, somewhere within the noted time frame. The deploy service will have some additional downtime. Thanks for your understanding.

We are upgrading all databases to MySQL 8.4. This will cause up to an hour of downtime for each app. We cannot give you exact times for when specifically your app will be down, we can only say that it will be down some time during this maintenance window. For more details see our blog post: https://blog.fortrabbit.com/mysql-8-4-upgrade-plan

We are upgrading all databases to MySQL 8.4. This will cause up to an hour of downtime for each app. We cannot give you exact times for when specifically your app will be down, we can only say that it will be down some time during this maintenance window. This extra maintenance window will upgrade the remaining Universal Apps that we did not have time to finish yesterday. For more details see our blog post: https://blog.fortrabbit.com/mysql-8-4-upgrade-plan

We are upgrading all databases to MySQL 8.4. This will cause up to an hour of downtime for each app. We cannot give you exact times for when specifically your app will be down, we can only say that it will be down some time during this maintenance window. For more details see our blog post: https://blog.fortrabbit.com/mysql-8-4-upgrade-plan

We will roll out some security patches during the maintenance window. This will not affect customer facing software version changes. The expected downtime per app is under a minute. We run this in batches for both regions US and EU.

We are looking into issues for Let's Encrypt certificates that have been recently created or renewed.

We think to have resolved the certificate issue. It was related to a change in Let's Encrypt root chain certificate. This is something, that is not happening on a regular basis, so it was not included with general security monitoring of external sources. It affected only a small number of newly created or recently updated certificates. Contact us if you still see SSL issues.

We are looking into web delivery issues for about 30 Apps hosted in the EU region of the old platform. The issues may have started around 06:00 UTC. Not all Apps are down, just some, they return random errors or timeouts.

Apps should be back for now. But we are still investigating.

Post mortem Universal Apps run shared host systems. These host systems have - of course - limited file storage. Initially we enforced hard file system quotas, but an underlying bug in BTRFS caused issues with that. So we lifted the hard limits and allowed exceeding the including file system limits. For Universal Apps, there is no automatic scaling, so that solution is actually better for clients too. We overbook storage, roughly by the factor of three, which would allow clients to use as twice as storage as the largest Uni plan. We regularly check for Apps that are exceeding the limits and write mails. Lately we have to do that more often. Most commonly, excessive file storage usage is attributed to code/config misconfiguration. Most often find a lot of old local backups files (maybe some plugin?), but also a lot of cache files. Specifically the later is getting an issue more and more. Maybe because of bots more aggressivly crawling websites, that offer an unlimited number of possible pages. We have health checks in place, alerting us on storage issues. In this case, the storage filled quickly, causing all kind of random errors. We pro-actively removed cache folders, which caused higher load on the host system, when Apps refilled the caches. This architecture is of course not prefect. It's one of the many reasons we decided to create a new platform from scratch. The new platform solves this on multiple levels: - Larger storage plans - Optional "Autoscaling": either capped, or upgrade After monitoring, we consider the issue now resolved.

We will roll out some security patches during the maintenance window. This will not affect customer facing software version changes. The expected downtime per app is under a minute. We run this in batches for both regions US and EU. BTW: We have migrated our status tool and cleaned up a bit. So this notification may look different now.