Bunny 장애 기록

We are currently investigating a Storage API incident on our German nodes. API Upload/List/Deletions may be affected as a result.

We identified an failing database process on the Storage layer, which has now been resolved by our Engineering team.

We are currently investigating reports of issues affecting the dashboard and certain services. Our engineering team is actively investigating the situation. We will share more information as soon as it becomes available.

Between 18:34 and 18:57 UTC, a transient connectivity incident affecting one of our upstream providers impacted Storage services and the Core API in Europe. The issue has been resolved. No data loss occurred, as the incident was limited to network connectivity. We are actively monitoring the situation and are working with our upstream provider to gather additional details about the incident.

This incident has been resolved.

We are currently investigating an issue affecting Magic Containers and Databases. Our engineering team is actively working to identify the cause and restore normal service. Further updates will be provided as they become available.

Service has been restored and affected systems are operating normally. We are continuing to monitor the platform closely to confirm stability and ensure there are no remaining impacts.

This incident has been resolved.

We identified intermittent issues affecting the Shield API that may have caused occasional errors for some users. A fix has been implemented, and the service has returned to normal operation. We are currently monitoring to ensure continued stability.

This incident has been resolved.

We have identified an issue with free certificate issuance and renewals for custom CDN domains.

We identified that this is due to an issue currently experienced by the Let's Encrypt service. Existing certificates on custom domains will continue to function as normal - however new and cert renewals via our free certificate option may experience errors to the certificate issuance. You can follow https://letsencrypt.status.io/ for more details.

As per Lets Encrypt status page, this has been resolved - we similarly have observed no degradation/failures on certificate renewals since their update went out.

We are currently investigating reports of delays affecting synchronization and the propagation of configuration changes. Our team is actively investigating the issue and working to determine its scope and impact. We will provide additional updates as soon as more information becomes available.

Our team has identified the underlying cause of the issue and has implemented a fix. The affected processes are now recovering, and pending changes are continuing to propagate. As a result, some synchronization and configuration propagation delays may still be observed while the platform returns to normal operation. We are actively monitoring the recovery and will provide another update once processing has normalized.

All pending configuration changes have propagated across the CDN.

We have identified an issue on free cert issuance and renewals for custom CDN domains. We have observed a delay from Lets Encrypt API endpoint causing these delays. Existing certificates on custom domains will continue to function as normal - however new and cert renewals via our free certificate option may experience a delay to the certificate issuance.

We no longer see delays stemming from Lets Encrypt and the issue is considered resolved.

We are currently investigating a service degradation affecting Stream. Users may experience delays with video uploads, video fetching operations, and transcoding jobs. Playback, delivery, and management of existing videos are not affected and continue to operate normally. Our engineering team is actively working to identify the cause and restore full service. Further updates will be provided as they become available.

We are seeing normal processing resume for newly submitted video uploads and transcoding jobs. Some videos that entered processing during the incident may remain delayed. Our team is actively reviewing and working through the remaining affected jobs while continuing to monitor the service. Further updates will be provided as they become available.

This incident has been resolved.

We are currently investigating an issue affecting Magic Containers and Databases. Our engineering team is actively working to identify the cause and restore normal service. Further updates will be provided as they become available.

Service has been restored and affected systems are operating normally. We are continuing to monitor the platform closely to confirm stability and ensure there are no remaining impacts.

This incident has been resolved.

Processing Delays and Intermittent Timeouts for Magic Containers On June 1, Magic Containers experienced degraded service for approximately one hour due to a network connectivity issue in our hosting provider's environment. Customers running workloads may have seen increased processing delays, intermittent timeouts, temporary disruption, or reduced availability as some workloads were automatically rescheduled. The issue was resolved after the underlying network conditions stabilized, allowing scheduling activity to resume normally. We are improving the resilience of Magic Containers so degraded connectivity can be detected and handled more effectively, not only complete outages. We are also adding safeguards to reduce unnecessary or duplicate workload rescheduling during degraded conditions, helping limit customer impact if similar infrastructure issues occur again.

We're currently investigating an issue causing the Optimizer to serve a number of 5XX errors. We're investigating this as a matter of priority and we will provide updates here as soon as we can.

This incident has been resolved.

We've been observing 25 to 30 nodes in various PoPs around the world exhibiting constant delays with syncing their configuration since yesterday evening. We've depooled all impacted nodes as a temporary workaround and working on a permanent fix.

A fix is being staged on a subset of affected nodes. Impacted nodes are since identifying the issue isolated from CDN traffic and only receiving test traffic. Further updates will be provided after testing concludes and deployment to all affected nodes is complete.

The fix has been deployed to all nodes. We will be readding the missing nodes to the pool today.

As a continuation of our DNS issues last night, we are currently investigating an issue when newly CDN accelerating a DNS record results in CDN 502s for the domain in question. Existing pull zones and pull zones not created via CDN Acceleration are not affected. Existing CDN Accelerated zones are also not affected.

We have identified a discrepancy in our backend process that has caused zone id mismatch on new CDN Accelerated zones. The team are currently working to deploy a hotfix to address the problem.

A fix has been applied and we're now working through the backlog to process all pending updates. We expect processing to be back to real-time in the next 6 hours.

This incident has been resolved.

We've identified an issue with the configuration of one of our primary domains, bunny.net, impacting most of our services. We're working to fix it.

A fix has been implemented and all impacted services have recovered. We're monitoring.

An internal process failure related to the management of our bunny.net domain with one of our domain registrars led to the temporary suspension of the domain, causing multiple services under bunny.net to become unavailable. After identifying and confirming the issue, we worked with the registrar to restore the domain and all affected services are now recovering as DNS changes propagate globally. We have identified the root cause internally and are implementing additional safeguards and escalation procedures to ensure this cannot happen again. During the outage, products and services under bunny.net experienced varying levels of disruption depending on DNS propagation and their dependency on bunny.net infrastructure and Bunny DNS. Services including the Website, Dashboard, API, DNS, and related tools experienced widespread availability issues. Other products such as Stream and Magic Containers saw limited disruption and remained largely operational for most users, though some functionality was impacted depending on DNS resolution and service dependencies. Core services such as CDN delivery and Storage remained operational throughout the incident. We will publish a full public RCA once the incident is fully resolved, including additional details around the timeline, service impact, root cause, and the preventative measures being implemented. We're deeply sorry for the disruption and inconvenience caused. We expect the situation to fully resolve over the next few hours as DNS propagation completes worldwide. If you have any questions, please contact support@bunny.net.

We are currently experiencing a network issue in our Singapore region, and as a result there may be degraded performance on CDN, Storage, Magic Containers and Database from the region. Our Engineering team are investigating the problem.

We are continuing to investigate this issue.

We identified an outage in relation to an upstream network provider that resulted in partial degradation in Singapore, this is now resolved. We are currently monitoring the situation.

This incident has now been cleared.

We are currently investigating issues related to delayed configuration of Pull Zones.

We have identified the core issue at hand and our Engineering team are currently reviewing the situation. Configuration delays/syncs are slowly recovering.

The configuration delay has now been resolved. At around 00:15, we observed a large scale domain sync that caused configuration delays. This was resolved at 06:00 UTC, before the backlog of configuration changes were cleared by 09:30 UTC. Our Engineering team have identified the core issue and will take action to prevent future recurrence of the problem.

We are currently looking into isolated 502s within the NA region for CDN endpoints utilizing Magic Containers and Databases. This is primarily affecting specific regions in North America. Our Engineering team are investigating the issue currently.

We are continuing to investigate this issue.

We have made some routing adjustments to mitigate the timeouts on Database and is fully operational again. Our Engineering team is continuing to investigate the Magic Containers issue.

Our Engineering team are still investigating the root cause for isolated 502s in the NA region.

We are monitoring improvements related to the resilience of routing configuration failover on edge servers during temporary control plane connectivity interruptions.

We identified and resolved an issue that could cause endpoint updates to experience delayed propagation after CDN services had been running for an extended period of time. The issue was related to endpoint synchronization between Magic Containers and the CDN routing component, where slower processing connected to sticky session invalidation could result in stale endpoint data remaining in use longer than expected.

We are working on an issue with our Statistics API endpoint. There are increased response times probing CDN and DNS statistics, resulting in some delayed reports, or timeouts. This affects both dashboard and direct API requests for statistics.

Our Engineering team has identified the issue of the statistics response delays, and are preparing a fix

Our Engineering team identified an overloaded process with on our statistics endpoint. This was rectified and monitored, and the issue has now been resolved.

We are aware of and working on an issue with our Statistics API endpoint. There are increased response times probing CDN statistics, resulting in some delayed reports, or timeouts. This affects both dashboard and direct API requests for statistics

A fix has been implemented in our statistics backend and we're monitoring its performance.

This incident has been resolved.

We are currently experiencing high demand for our free encoding services. Video processing times may be longer than usual. Premium encoding, existing videos, and delivery are not affected. Our team is actively monitoring the situation, thank you for your patience.

We have expanded our transcoding capacity, transcoding delays have now cleared, and processing has normalized.

Our Magic Containers team are currently investigating an issue with application updates. There may be deployment delays or failures on the platform as a result.

Our Containers team found that a recent deployment had resulted in an unintended CPU throttling on new pod or app deployments. This has now been resolved, and the team has made mitigating changes to ensure this does not resurface.