Fasterize 장애 기록

We currently have some issues on our European infrastructure. Impact on acceleration. Some pages can have some slowdowns. We're investigating but seems to be due to our hosting provider partial outage https://status.scaleway.com/incidents/qkj49g6g3ykp

Everything is back to normal but we're still monitoring. Root cause seems to be a power issue in a datacenter.

We are continuing to monitor for any further issues.

Incident is closed since 11:30am but we keep on monitoring as the incident on our hosting provider is not fully resolved.

We are currently investigating an issue affecting website status checks in the console. Some websites may incorrectly appear with the status “Ping blocked” or “Sonde bloquée”. This does not necessarily mean that your website is unavailable. We will share an update as soon as possible.

This incident has been resolved.

We are currently fixing an issue affecting flush operations through the API and dashboard. Our team is working to restore normal behaviour as quickly as possible. We will share an update once the issue is resolved.

The issue affecting flush operations through the API and dashboard has been fixed. Flush actions are now working normally again.

We experienced some issues with our European infrastructure. Service has been restored since 00:20. This has affected loading speeds. Some pages or websites have experienced slowdowns.

We have identified an issue affecting the accuracy of INP data collected between Thursday, 5 February and Monday, 9 February. During this period, INP metrics were incorrectly reported for Safari on iPhone when the navigation type was back-forward cache (BFCache). This behaviour is related to a known issue in WebKit, which causes inflated INP values in this specific scenario. More details are available in the related Safari bug report: https://bugs.webkit.org/show_bug.cgi?id=305251 The issue has been identified and mitigated on our side. Only the INP metric is impacted; other performance metrics remain unaffected. We apologise for any confusion this may have caused and recommend caution when analysing INP data for Safari iPhone traffic over this time window. If you have any questions, our team remains available.

The Fasterzie Console is currently slow. We’re investigating.

The issue has been identified, our team is working on resolving it

The incident has been resolved, everything is now operating normally

Our platform is experiencing an outage, where possible traffic is automatically routed to origin to mitigate the incident. We identified the issue and are working on a fix to restore availability

We are still investigating and trying to restore the failing component.

The failing component is gradually recovering, and we are monitoring it until full restoration

A fix has been implemented and we are monitoring the results.

Configuration updates are not possible from the Fasterize console. We are currently investigating.

Acceleration is now fully operational. Configuration updates are still unavailable, and we are investigating the issue. Cache purges are also currently unavailable.

The issue has been identified and a workaround is being implemented. In the meantime, configuration updates and cache purges are unfortunately still unavailable. The workaround will unfortunately degrade the API performance but it should be temporary until we rollout a definitive solution. As soon as the fix is delivered, we'll monitor it to be sure that everything's ok. The postmortem will follow, as we also need to gather further elements from our provider. We deeply apologize for this incident and we're actively working to solve this issue as soon as possible.

This incident has been resolved. The API and the dashboard are now able to handle correctly configuration updates and cache purges. As mentioned earlier, a postmortem will be written and available here.

An in-memory database cluster failure occurred leading to service unavailability across multiple Fasterize components — primarily the **Optimisation Engine** and the **API**. At **09:48**, an in-memory database cluster failure happens after restarting multiple nodes to release a new engine version. The database cluster began an automatic failover sequence, but each time a new node was promoted as primary, it **crashed under excessive connection load**. This cluster serves as a **cache layer** providing access to configurations. During the outage, Engine instances attempted to reconnect at a very high frequency and fell back to retrieving data directly from the main database.This fallback mechanism worked as intended until **10:32**, allowing our optimization engine to continue operating in **degraded mode**. At **10:32**, however, the **proxy layer** of our optimisation engine became **saturated in network resources**, rendering it unavailable from the front layer. When the proxy layer is unreachable, the platform automatically **unplugs the CDN and sites are served directly from their origin servers**, without Fasterize optimizations. Working with our hosting provider, we **reduced the Optimisation Engine cluster size at 11:45** to limit reconnection attempts to Redis. By **12:16**, the Redis cluster had stabilised and full service was restored. Later, at **14:15**, we detected that the **API was unable to write to the Redis cluster**. The root cause was a **security patch applied by the hosting provider**, which **restricted the use of some commands** in the cluster.The API was patched to remove usage of these commands, and full functionality was restored by **20:15**. ‌ ## **Impact** * **Duration:** 10h32 – 12:16 \(outage\), API issue until 20:15 * **Affected components:** Optimisation Engine, API * **User impact:** Most websites temporarily served unoptimised content directly from origin; Some websites experienced unavailability due to an dysfunctional DNS fallback mechanismAPI write operations failed, blocking configuration update. ## **Resolution Timeline** ‌ ## Action plan Short term: * Improve **alerting and visibility** on in-memory database cluster health and failover events. * Review the in-memory database connection logic of the engine to avoid too many reconnection attempts in case of disconnections and avoid a case preventing the start of the process in case of an in-memory database outage.  * Adjust failover DNS logic to avoid redirection to the origin when the CDN/fronts are still able to accept the traffic. * Upscale in-memory database cluster in order to accept more connections Medium term:  * Review and test **disaster recovery procedures** for in-memory database cache clusters, including the ability to quickly activate a passive cluster. Long term: * Re-architecture the engine to reduce the number of connections on the itn-memory database cluster

A network maintenance caused a regression in the service responsible for extracting the True-Client-IP header from the X-Forwarded-For header. Around 6% of the platform trafic has been impacted. The issue has now been resolved. We apologize for the inconvenience.

Between 14:49 and 14:52 (Paris time), our services were briefly disrupted due to a Distributed Denial of Service (DDoS) attack. The issue has been fully resolved, and systems are operating normally. We appreciate your patience and understanding. Sorry for the inconvenience.

This incident has been resolved.

Between 11:35 a.m. and 11:55 a.m. (Paris time), the platform experienced performance slowdowns following a scheduled update impacting the load balancers.

# **🛠️ Incident Description** **On Tuesday, September 9th, 2025, Fasterize carried out a scheduled maintenance to modernize part of its private network infrastructure.** **During this operation, an unexpected configuration issue temporarily disrupted communication between some components of the platform. This resulted in an increased error rate on part of the traffic between 11:43 and 11:50 \(Paris time\).** # **📅 Timeline** * **11:40 – Update started on the first batch of machines** * **11:43 – Anomaly detected in communication between proxies and fronts** * **11:50 – Issue mitigated, service returned to normal** # **✅ Immediate Actions** * **Affected machines were taken out of rotation** * **Healthy machines were reactivated to absorb the traffic** * **Service stabilized within minutes of the anomaly** # **🚀 Next Steps** * **Strengthening our validation procedures during network migrations** * **Adding automated connectivity tests to detect similar issues earlier** * **Improving documentation and standardization of network configurations to minimize future risks**

On September 9, Fasterize performed a planned network maintenance to replace an obsolete private network. As part of this operation, a new private IP address was introduced for our load balancers while keeping the existing ones in place. Later that afternoon, a DDoS attack led to the saturation and restart of one of the load balancers. Following this restart, the load balancer started using the new private IP address to communicate with the backend servers. Since this IP had not yet been registered as a trusted proxy, some requests were processed incorrectly.

## **🛠️ Incident Report** **Date:** Tuesday, September 9, 2025 **Duration:** 14:50 – 18:30 \(Paris time\) **Impact:** ~23% of traffic ### **📌 Summary** On September 9, Fasterize performed a planned network maintenance to replace an obsolete private network. As part of this operation, a new private IP address was introduced for our load balancers while keeping the existing ones in place. Later that afternoon, a DDoS attack led to the saturation and restart of one of the load balancers. Following this restart, the load balancer started using the new private IP address to communicate with the backend servers. Since this IP had not yet been registered as a trusted proxy, some requests were processed incorrectly. ### **📈 Impact** * ~23% of traffic was affected between **14:50 and 18:30**. * Some users encountered **403 errors**. * In certain cases, client IP addresses were not correctly identified in the headers. ### **📅 Timeline** * **10:25** – New private IP address added to the load balancers. * **14:49** – DDoS attack causes one load balancer to restart. * **14:50** – First incorrect headers observed \(undetected at this stage\). * **16:02** – Support receives a ticket mentioning 403 errors. * **17:16** – Ticket escalated to the platform team. * **18:16** – Root cause identified and linked to the network update. * **18:30** – New IP address added to the trusted list; incident resolved. **Total duration of impact:** 3h40 ### **🔍 Root Cause** The new private IP address introduced during the maintenance was not yet included in the trusted proxy configuration. When one load balancer restarted, it began using this IP, leading to incorrect handling of client headers. ### **🚀 Next Steps** * All load balancer IPs have now been added to the trusted configuration across environments. * Documentation has been updated to include this step in future network changes. * Additional monitoring will be set up to detect unexpected private IPs in client headers. * Escalation procedures between support and platform teams will be reviewed to ensure faster response times.

Slowdowns on the platform since 10 a.m.

This incident has been resolved.

We currently have some issues on one of our european DC. Being fixed. Trafic is interrupted for a large portion of the customers. Really sorry for the inconvenience.

The failover mechanism didn't trigger. We trigger it manually.

This incident is now resolved. Post-mortem will follow but to sum up: the root cause is a change in a DNS record. During that change, the record pointing to our DC took a temporary wrong value that was captured by some edge servers and stored for one hour. This affected only a subset of edge servers and only a subset of the health checkers responsible for triggering the failover mechanism. This explains why the failover mechanism wasn't fully triggered.

**🧾 Incident Summary** As part of preparations for the summer sales and in response to a load handling issue observed on June 12, the tech team has planned an update of a DNS record pointing to the platform’s frontend layer. The goal was to review the way CDN Edge servers send traffic to the platform. Since the DNS record could not be edited in place, it was deleted and immediately recreated with a new routing policy. During the brief deletion \(~10 seconds\), some DNS resolvers fell back to a default wildcard record with a 3600s TTL, pointing to decommissioned IPs instead of returning NXDOMAIN. As a result, several CDN Edge servers cached these incorrect IPs and continued to use them for up to an hour, causing a global service outage on the affected POPs. Some health check probes also resolved to the wrong IPs but not enough of them failed at once to trip the health check thresholds \(which require consecutive failures to switch to a degraded state\). ‌ ### **📅 Timeline** ### **🧩 Root Cause Analysis** This outage was not due to an infrastructure failure but to a human error during maintenance without sufficient safeguards. We consider this as a major governance failure in infrastructure change management. The affected DNS zone was created over 12 years ago and currently holds legacy records. Over time, several have become outdated or unused. However, due to the risk of accidental deletion and unexpected impact, no regular cleanup has been performed. This lack of maintenance allowed a misconfigured wildcard record to persist, which was unintentionally triggered during the temporary deletion of a critical DNS record. We’ve now initiated a full audit of this DNS zone to identify, document, and progressively remove obsolete records. A cross-validation policy will be enforced before any future changes. Although a staging environment exists and is used to validate infrastructure changes, the specific scenario here — related to active CDN traffic and DNS behavior during the few seconds of deletion — was not anticipated. Due to the distributed and asynchronous nature of CDN DNS propagation and caching \(per POP\), the issue could hardly be replicated in a staging environment. ### **✅ Immediate Fixes** * Re-creation of the DNS record with the correct configuration * Manual checks to ensure resolvers and CDN POPs are now resolving to the correct origin ### **🔒 Preventive countermeasures** **Short-term:** * Freeze on infrastructure changes for two weeks **Medium-term:** * Improve the staging environment to better simulate CDN-specific issues * Clean up outdated records in the platform's DNS zone * Formalize an emergency fallback procedure * Introduce more logical zones to avoid widespread impact across clients ‌ ### **Conclusion** This incident highlights how even short-lived DNS misconfigurations can cause major disruptions in distributed systems like CDNs.Rigorous TTL management and better anticipation of critical DNS usages are essential to avoid similar outages in the future. Despite this incident, the technical and customer success team remains fully committed to delivering a smooth and successful sales period for our clients.A temporary change freeze is in place, and additional planning and capacity measures are being taken to ensure high availability and reliability throughout this peak traffic period.

We currently have some issues on our european infrastructure. We're investigating. Speeding-up is disable. Traffic should be redirected to origins.

Issue has been identified. Some load balancers in one of our datacenter have crashed and have been restarted. Traffic was interrupted at 3:31PM and is back to normal since 3:42PM CET.

We're still monitoring.

Incident is now closed. Sorry for the inconvenience. A post-mortem will follow.

## **Summary** On June 12 2025, Fasterize experienced a service disruption for a subset of customers due to a large-scale DDoS attack. From 15:31 to 15:42 \(UTC\+2\) the surge in traffic overloaded the zone’s load balancers, resulting in a temporary fallback to origin servers for affected sites. Total outage time was 11 minutes, with full platform stability restored at 15:42. ## **Timeline \(UTC\+2\)** * **15:31 – 15:42**: DDoS attack peaks at ~8 million requests overall, spiking to 50 000 req/s. Load balancers in the zone restart repeatedly. * **15:34**: Internal alerts detect the saturation \(detection time: 3 min\). Health-check system routes traffic directly to origin servers. * **15:42:** Traffic normalises; services in the affected zones recover \(time to full resolution: 11 min\). ## **What Happened** The volumetric DDoS flooded the load balancers, driving CPU utilisation to 100 %. Although our automatic health-check mechanism successfully redirected traffic to customer origin servers \(for those customers' whose stack permits this automatic redirection\), the repeated load-balancer restarts caused an 11-minute service degradation for sites relying on that zone. We had increased load-balancer capacity after a similar event on May 15, but this proved insufficient for the size of yesterday’s attack, highlighting a design limitation in relying on provider-managed load balancers. ‌ ## Impact * **Severity level:** 1 — Unplanned downtime affecting multiple production websites * **Detection time:** 3 minutes * **Time to full recovery:** 11 minutes During the incident, visitors to impacted sites experienced 502 errors until they were served directly from origin infrastructure, potentially experiencing higher latency. ## **What We're Doing** ### Short-term improvements \(in progress\) * Re-architecture to avoid load balancer issues ### Long-term improvements * **Additional zones:** Bringing online new zones to spread traffic and isolate attacks more effectively. * **Rate-limit hardening:** Evaluating two third-party CDNs that offer native volumetric-attack rate limiting.

What happened: This morning at 8:52, we made planned adjustments to our infrastructure in order to improve the reliability of the connection between our platform and CDN. During the transition, the service experienced a short disruption due to unexpected load balancer restart. Impact: The site was briefly unavailable for approximately one minute generating error 502. Resolution: The situation was quickly identified and resolved by 08:52 (GMT+2) . Everything is now back to normal and fully operational. We’re sorry for the inconvenience and are taking steps to prevent this from happening again.

One of our datacenter has been unavailable between 21h21 and 21h47. We are investigating the incident.

We experienced a service disruption caused by a Distributed Denial of Service (DDoS) attack. The issue has now been resolved. A full post-mortem will follow. Thank you for your patience and understanding.

## **Summary** On May 14, 2025, Fasterize experienced a partial service disruption affecting a subset of customers. The issue was caused by a large-scale DDoS attack targeting a website accelerated by our platform. The incident lasted approximately 25 minutes, with service fully restored at 21:47. ## **Timeline \(UTC\+2\)** * **21:22 – 21:30**: Our systems registered an abnormally high volume of requests — over 37 million in total, peaking at 350,000 requests per second. * **21:47**: Traffic stabilized and all services were back to normal. ## **What Happened** The DDoS attack overwhelmed several load balancers, leading to repeated restarts. Under normal circumstances, our failover system automatically routes traffic directly to the origin servers if a platform zone becomes unhealthy. However, the DNS health checks tied to certain zones were misconfigured. They continued to report the zone as healthy despite the outage, preventing failover from triggering correctly. ## **Impact** * **Severity Level:** 1 \(Unplanned downtime affecting multiple production websites\) * **Detection time:** 12 minutes * **Time to full recovery:** 25 minutes ## **What We're Doing** ### **Immediate fixes** * Corrected the failover configuration to ensure accurate health checks. ‌ ### **Short-term improvements** * Tuned load balancer settings for better resilience under high traffic. * Improved alerting on health check anomalies. ### **Medium-term improvements** * Increasing infrastructure redundancy to distribute traffic more effectively. * Evaluating native rate-limiting solutions to mitigate volumetric attacks.

Acceleration was disabled between 10:45 AM GMT+1 and 11 AM GMT+1 due to a misconfiguration of the global platform health checks. These health checks are currently being reviewed (but the platform is actually healthy).

Root cause has been identified. A post-mortem is being written. We're gonna set up more robust global platform health checks.

Fix has been deployed, everything is back to normal (the platform is properly monitored again). We're still monitoring.

Incident is now closed. Sorry for the inconvenience. A post-mortem will follow.

# Healthcheck in error Date of post mortem : 25/02/2025 Participants of the post mortem : Anthony Barré # Description of the incident Following a change in the Fasterize API, **the health checks of the platform turned red**, indicating an unavailability of the platform while it was actually healthy. These health checks were configured to query a URL associated with **a domain name that was no longer present** in the configuration database. ### **Immediate corrective action** * **Temporary deactivation** of health checks. * **Rollback** of the API configuration change. # Facts & Timeline All times are UTC\+1. # Analysis ## **Technical context** * The health checks validate the availability of the platform through several layers \(proxy, load balancer, workers, etc …\) * The failure of the health checks was due to errors at the proxy level, a symptom of the absence of configuration associated with the domain used.  * At the proxy level, the health check response is adapted by region to respond to the health checks associated with each region. * An outdated process, that was still running despite no longer being required, caused issues when it updated the configuration used by proxies with incorrect data. ## **Root cause analysis** The API configuration change caused an unstable state. Indeed, an API update did not take place immediately. Thus, some API pods were reading the health check configuration from one environment and others from another environment. This corrupted the database used by the proxies of our main environment. The proxies no longer responded with 200 but with error code to health check requests. ## **Impact of incident** 🏠 **Affected customers** : **All customers** were impacted by the lack of acceleration. 🔴 **Specific issue** : **Two customers** experienced major disruptions because their origins did not allow them to receive traffic directly from the Internet. 📊 **Incident metrics** * **Severity**: **1** \(service shutdown impacting a large number of users\). * ⏱ **Detection time**: **4 minutes**. * 🛠 **Resolution time**: **20 minutes**. # Action Plan  ## Short term \(immediate - 1 week\) * Review the platform's health checks so that they are more robust  * Disable and delete the obsolete monitoring process ✅ * Fix the deployment of configmaps in the helm charts. ## Medium term * Review the API configuration to clarify the use of region-related fields and avoid side effects.

We encountered a minor issue affecting the purge of image metadata and CSS/JS bundles. While the purge process experienced some disruptions, this had no noticeable impact on website functionality. Pages and static files were purged correctly.

Timeframe: 9:30 AM to 10:15 AM During this period, pages were served without optimization by the platform. There was no downtime, and traffic continued to be served seamlessly with original (non-optimized) pages. Cause: The issue was triggered by a recent release that unexpectedly introduced a high load on the platform. Resolution: We promptly identified the issue and rolled back the release to restore normal operations. We sincerely apologize for any inconvenience this may have caused and appreciate your understanding.

The platform experienced an outage from 10:13 AM to 10:20 AM (Paris timezone) caused by a DDOS attack. Traffic was automatically routed to origins from 10:17AM to 10:20AM. The attack volume was around 30x the normal trafic. Even if our anti DDOS system blocked 97.6% of the attack, the attack was too heavy for our frontals servers.