Mux outage history

We are experiencing some latency in collecting beacons, beginning 22:04 UTC. Investigation is underway.

The system has recovered at 22:28 UTC and we are monitoring the results.

This incident has been resolved. For a subset of customers, our ingestion pipeline was unresponsive to incoming beacons at 22:04 UTC. While our SDKs are programmed to retry, there is the potential for delayed or lost beacons at and around that time. Delayed/lost beacons would disrupt our realtime metrics and alerting. Missing beacons may also result in some missing event data for views being generated at that time.

We’ve identified an issue tha is affecting on-demand (VOD) video playback for some customers. Some viewers may see failed or slow playback. Live streaming is not affected.

The issue has been identified and a fix is being implemented.

A fix has been implemented and we are monitoring the results.

We are continuing to monitor for any further issues.

This incident has been resolved.

From 22:10 to 22:50 UTC we experience elevated latency for video routes, we are currently investigating the root cause.

We are continuing to investigate this issue.

The issue has been identified and it was related to our cloud provider hosting our API. API latency has returned to normal levels, and we’re continuing to monitor stability. We apologize for the disruption.

This incident has been resolved.

We are investigating elevated error rates affecting Direct Uploads, thumbnails, gifs, subtitles, storyboards. Some customers may have experienced failures when creating or uploading to direct upload URLs

A fix has been implemented and we are monitoring the results.

We are continuing to monitor for any further issues.

This incident has been resolved.

We are currently investigating elevated webhook delivery times. Beginning at 16:40 UTC, some webhooks are being delivered with a delay of approximately 5 minutes. Video processing, playback, and API requests are not affected, and no webhook events are being dropped — delayed events will still be delivered. We will provide an update as soon as we have more information.

We have identified the cause of the elevated webhook delivery times. An issue in one of our internal database clusters introduced a delay in the pipeline that queues webhook events, resulting in delivery delays beginning at 16:40 UTC. Video processing, playback, and API requests are unaffected, and no events have been lost — delayed webhooks are still being delivered. Delivery times are beginning to recover.

Webhook delivery times have returned to normal. Between 16:40 UTC and 17:36 UTC, webhook events were delivered with delays averaging around 5 minutes, with some events delayed up to 15 minutes. Video processing, playback, and API requests were unaffected, and no events were lost — all delayed webhooks have been delivered.We have addressed the underlying issue in one of our internal database clusters and are monitoring to confirm.

This incident has been resolved. Between 16:40 UTC and 17:36 UTC, webhook events were delivered with delays averaging around 5 minutes, with some events delayed up to 15 minutes. Video processing, playback, and API requests were unaffected, and no events were lost — all delayed webhooks have been delivered.The underlying issue in one of our internal database clusters has been addressed, and delivery times have been stable since 17:36 UTC. We apologize for the disruption.

We are investigating an issue where CSV exports are delayed beginning 8/13 to today.

The issue has been identified and we have started the process of running delayed exports jobs.

Export jobs are being run for the next 24 hours, beginning with 8/13. We will notify here when the jobs are complete and exports are available.

This incident has been resolved, export jobs are complete

We are currently investigating slowness in asset creation that affects limited number of assets in one of our North America cloud locations.

A fix has been implemented and we are monitoring the results.

The fix resolved the issue and asset creation back to operational.

Customers may experience issues with video live and/or playback. Root cause has been identified and fix is being worked on. More updates to come as we have it.

A fix has been implemented and we are monitoring the results.

We have identified service disruption in underlying cloud infrastructure in a single availability zone, which impacted a portion of video ingestion and playback. We have routed traffic away from the affected infrastructure and are continuing to monitor while we work with the service provider on resolution.

All customer impact has been mitigated. Traffic remains directed away from the affected infrastructure while we work with our cloud provider on the underlying issue. We will remain in this state until we have fully validated the issue is resolved and we can safely restore normal operation. Once normal operation is restored, we will update and close this incident accordingly.

This incident has been resolved.

We are investigating reports of video playback failures. Some customers may experience playback errors or timeouts.

The issue has been identified and a fix is being implemented.

We are continuing to work on a fix for this issue.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

We are investigating an issue affecting live streams. Some viewers may experience dropped or interrupted live streams. We will provide updates as more information becomes available.

We are continuing to investigate this issue.

The issue has been identified and a fix is being implemented.

A fix has been implemented and we are monitoring the situation

This incident has been resolved.

We are investigating an issue affecting live streams. Some viewers may experience dropped or interrupted live streams. Our team has identified the underlying infrastructure issue and is actively working on a resolution. We will provide updates as more information becomes available.

The issue has been identified and a fix is being implemented.

A fix has been implemented and we are monitoring the results.

We are continuing to monitor for any further issues.

This incident has been resolved.

We are investigating an issue with collecting Mux data beacons. Beginning 22:12 UTC, view session data will be delayed.

A fix has been implemented and we are monitoring results. Delayed data was for a subset of customers between 22:12 UTC - 22:44 UTC. No data loss.

This incident has been resolved.

Between 16:15 and 16:20 UTC a small number of live streams were disconnected. Active live streams were able to reconnect and resume, and there is no long term impact to DVR or VOD content from affected live streams other than resuming on a new Asset ID. All systems are currently operational and we are monitoring the live ingest systems.

No further issues observed during monitoring period. Live streams are operational.

We are investigating an incident which caused the disconnection of multiple live streams at 13:11 UTC.

A small number of live streams were disconnected around 13:10 UTC, ending the corresponding Asset. Active live streams were able to reconnect and resume, and there is no long term impact to DVR or VOD content from affected live streams other than resuming on a new Asset ID. All systems are currently green.

We are experiencing increased errors with delivery across Live and VOD in Europe regions  and are actively investigating.

We've identified an issue with a CDN provider in the impacted regions. We are seeing recovery, and continuing to monitor.

Video delivery for Live and VOD in the impacted region has returned to normal operations.  The impacted time frame was 9:30 - 10:30 UTC.  We are continuing to monitor.

This incident has been resolved.

We are investigating an issue causing an increase in errors in the Mux Data dashboard and API.

We have identified the issue and are implementing a fix.

A fix has been implemented and we are monitoring.

This incident has been resolved.

From 21:02 UTC to 22:55 UTC some live streams may have experienced elevated rebuffering. We have identified and mitigated the issue.

We are seeing high webhook delivery latency. We are actively investigating

We have identified the issue and have deployed a fix. We are continuing to monitor.

This incident has been resolved.

From 03:08UTC to 03:14UTC we saw increased request error rates and webhook delivery latency increases. The issue is mitigated and systems are operating normally.

We are currently investigating errors in Mux Data historical metrics beginning at approximately 22:08 UTC. We apologize for the inconvenience and will share an update once we have more information.

Our team has identified the cause of elevated Data Historical metrics errors and is actively working on a fix. We’ll continue to share updates as we have them.

Our team has implemented a fix and we are monitoring the results. We’ll post an update as soon as the issue is fully resolved.

This incident has been resolved