Bunny outage history

Our Engineering team are currently investigating an issue on our Stream platform of playback failures on our embedded players, as well as upload failures to our platform.

We have identified a backend database issue which had caused a timeout across our video servers (to include our player and uploading process). The problem was rectified and we are now monitoring the situation.

We are observing some isolated issues on the back of the fix. We are working to fully restore the process on our platform.

We have confirmed that all services have fully resumed. Our Engineering team are currently monitoring the situation. Playback has now fully resumed without issue. New uploads during the outage have failed and thus would need re-uploading. Existing transcoding videos were not lost, however the status reported on some of these videos may be inaccurate as a result.

This issue is now fully resolved. Our Engineering team have identified additional guardrails to prevent this specific issue from happening in future.

We are currently observing performance degradation with a Tier 1 transit provider in the Los Angeles area due to multiple fiber cuts on their network. This has impacted connectivity between the Bunny Optimizer LA cluster and a small number of US CDN PoPs. Traffic has been rerouted to minimize the impact.

This incident has been resolved.

We're investigating reports of widespread network issues in Los Angeles that seem to be impacting our services as well. We are working with our providers in the area to understand the issue at hand and mitigate.

A wider network incident in Los Angeles impacted Bunny Services hosted in the region. Affected network providers have mitigated the incident and all impacted services have remained stable since then.

We have identified an issue with our Database controller, that our Engineering team are working to restore. Databases already online are not affected and will continue operating normally, but idle Databases or Database management is currently unavailable.

Our Engineering team are continuing to work on the issue. Additionally, we are aware of some working Databases failing during this outage. This is related and our team are also working to address this.

The issue has been resolved and the affected services have been restored. Our Engineering team will continue monitoring the platform to ensure everything remains stable.

After monitoring the affected services, we have confirmed that Database management and idle Database activation are operating normally.

We are currently investigating elevated error rates on the Storage API in our London, UK region. Some API operations may be affected as a result.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

We are currently investigating intermittent playback failures on Stream libraries, specifically with Basic DRM enabled. This does not impact existing underlying storage, or other video library types (Enterprise DRM or non DRM).

We have identified a process on a single server causing Basic DRM transformation failures, and was the cause of intermittent video freezes/500s. This has been rectified by the Engineering team.

We are currently investigating an issue where users are unable to log in to the Bunny.net dashboard. Services themselves are not disrupted.

The login issue has now been resolved following a fix deployed by our Frontend team.

We are currently investigating a Storage API incident on our German nodes. API Upload/List/Deletions may be affected as a result.

We identified an failing database process on the Storage layer, which has now been resolved by our Engineering team.

We are currently investigating reports of issues affecting the dashboard and certain services. Our engineering team is actively investigating the situation. We will share more information as soon as it becomes available.

Between 18:34 and 18:57 UTC, a transient connectivity incident affecting one of our upstream providers impacted Storage services and the Core API in Europe. The issue has been resolved. No data loss occurred, as the incident was limited to network connectivity. We are actively monitoring the situation and are working with our upstream provider to gather additional details about the incident.

This incident has been resolved.

We are currently investigating an issue affecting Magic Containers and Databases. Our engineering team is actively working to identify the cause and restore normal service. Further updates will be provided as they become available.

Service has been restored and affected systems are operating normally. We are continuing to monitor the platform closely to confirm stability and ensure there are no remaining impacts.

This incident has been resolved.

We identified intermittent issues affecting the Shield API that may have caused occasional errors for some users. A fix has been implemented, and the service has returned to normal operation. We are currently monitoring to ensure continued stability.

This incident has been resolved.

We have identified an issue with free certificate issuance and renewals for custom CDN domains.

We identified that this is due to an issue currently experienced by the Let's Encrypt service. Existing certificates on custom domains will continue to function as normal - however new and cert renewals via our free certificate option may experience errors to the certificate issuance. You can follow https://letsencrypt.status.io/ for more details.

As per Lets Encrypt status page, this has been resolved - we similarly have observed no degradation/failures on certificate renewals since their update went out.

We are currently investigating reports of delays affecting synchronization and the propagation of configuration changes. Our team is actively investigating the issue and working to determine its scope and impact. We will provide additional updates as soon as more information becomes available.

Our team has identified the underlying cause of the issue and has implemented a fix. The affected processes are now recovering, and pending changes are continuing to propagate. As a result, some synchronization and configuration propagation delays may still be observed while the platform returns to normal operation. We are actively monitoring the recovery and will provide another update once processing has normalized.

All pending configuration changes have propagated across the CDN.

We have identified an issue on free cert issuance and renewals for custom CDN domains. We have observed a delay from Lets Encrypt API endpoint causing these delays. Existing certificates on custom domains will continue to function as normal - however new and cert renewals via our free certificate option may experience a delay to the certificate issuance.

We no longer see delays stemming from Lets Encrypt and the issue is considered resolved.

We are currently investigating a service degradation affecting Stream. Users may experience delays with video uploads, video fetching operations, and transcoding jobs. Playback, delivery, and management of existing videos are not affected and continue to operate normally. Our engineering team is actively working to identify the cause and restore full service. Further updates will be provided as they become available.

We are seeing normal processing resume for newly submitted video uploads and transcoding jobs. Some videos that entered processing during the incident may remain delayed. Our team is actively reviewing and working through the remaining affected jobs while continuing to monitor the service. Further updates will be provided as they become available.

This incident has been resolved.

We are currently investigating an issue affecting Magic Containers and Databases. Our engineering team is actively working to identify the cause and restore normal service. Further updates will be provided as they become available.

Service has been restored and affected systems are operating normally. We are continuing to monitor the platform closely to confirm stability and ensure there are no remaining impacts.

This incident has been resolved.

Processing Delays and Intermittent Timeouts for Magic Containers On June 1, Magic Containers experienced degraded service for approximately one hour due to a network connectivity issue in our hosting provider's environment. Customers running workloads may have seen increased processing delays, intermittent timeouts, temporary disruption, or reduced availability as some workloads were automatically rescheduled. The issue was resolved after the underlying network conditions stabilized, allowing scheduling activity to resume normally. We are improving the resilience of Magic Containers so degraded connectivity can be detected and handled more effectively, not only complete outages. We are also adding safeguards to reduce unnecessary or duplicate workload rescheduling during degraded conditions, helping limit customer impact if similar infrastructure issues occur again.

We're currently investigating an issue causing the Optimizer to serve a number of 5XX errors. We're investigating this as a matter of priority and we will provide updates here as soon as we can.

This incident has been resolved.

We've been observing 25 to 30 nodes in various PoPs around the world exhibiting constant delays with syncing their configuration since yesterday evening. We've depooled all impacted nodes as a temporary workaround and working on a permanent fix.

A fix is being staged on a subset of affected nodes. Impacted nodes are since identifying the issue isolated from CDN traffic and only receiving test traffic. Further updates will be provided after testing concludes and deployment to all affected nodes is complete.

The fix has been deployed to all nodes. We will be readding the missing nodes to the pool today.

As a continuation of our DNS issues last night, we are currently investigating an issue when newly CDN accelerating a DNS record results in CDN 502s for the domain in question. Existing pull zones and pull zones not created via CDN Acceleration are not affected. Existing CDN Accelerated zones are also not affected.

We have identified a discrepancy in our backend process that has caused zone id mismatch on new CDN Accelerated zones. The team are currently working to deploy a hotfix to address the problem.

A fix has been applied and we're now working through the backlog to process all pending updates. We expect processing to be back to real-time in the next 6 hours.

This incident has been resolved.

We've identified an issue with the configuration of one of our primary domains, bunny.net, impacting most of our services. We're working to fix it.

A fix has been implemented and all impacted services have recovered. We're monitoring.

An internal process failure related to the management of our bunny.net domain with one of our domain registrars led to the temporary suspension of the domain, causing multiple services under bunny.net to become unavailable. After identifying and confirming the issue, we worked with the registrar to restore the domain and all affected services are now recovering as DNS changes propagate globally. We have identified the root cause internally and are implementing additional safeguards and escalation procedures to ensure this cannot happen again. During the outage, products and services under bunny.net experienced varying levels of disruption depending on DNS propagation and their dependency on bunny.net infrastructure and Bunny DNS. Services including the Website, Dashboard, API, DNS, and related tools experienced widespread availability issues. Other products such as Stream and Magic Containers saw limited disruption and remained largely operational for most users, though some functionality was impacted depending on DNS resolution and service dependencies. Core services such as CDN delivery and Storage remained operational throughout the incident. We will publish a full public RCA once the incident is fully resolved, including additional details around the timeline, service impact, root cause, and the preventative measures being implemented. We're deeply sorry for the disruption and inconvenience caused. We expect the situation to fully resolve over the next few hours as DNS propagation completes worldwide. If you have any questions, please contact [email protected].