Mux outage history

We are investigating an issue where CSV exports are delayed beginning 8/13 to today.

The issue has been identified and we have started the process of running delayed exports jobs.

Export jobs are being run for the next 24 hours, beginning with 8/13. We will notify here when the jobs are complete and exports are available.

This incident has been resolved, export jobs are complete

We are currently investigating slowness in asset creation that affects limited number of assets in one of our North America cloud locations.

A fix has been implemented and we are monitoring the results.

The fix resolved the issue and asset creation back to operational.

Customers may experience issues with video live and/or playback. Root cause has been identified and fix is being worked on. More updates to come as we have it.

A fix has been implemented and we are monitoring the results.

We have identified service disruption in underlying cloud infrastructure in a single availability zone, which impacted a portion of video ingestion and playback. We have routed traffic away from the affected infrastructure and are continuing to monitor while we work with the service provider on resolution.

All customer impact has been mitigated. Traffic remains directed away from the affected infrastructure while we work with our cloud provider on the underlying issue. We will remain in this state until we have fully validated the issue is resolved and we can safely restore normal operation. Once normal operation is restored, we will update and close this incident accordingly.

This incident has been resolved.

We are investigating reports of video playback failures. Some customers may experience playback errors or timeouts.

The issue has been identified and a fix is being implemented.

We are continuing to work on a fix for this issue.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

We are investigating an issue affecting live streams. Some viewers may experience dropped or interrupted live streams. We will provide updates as more information becomes available.

We are continuing to investigate this issue.

The issue has been identified and a fix is being implemented.

A fix has been implemented and we are monitoring the situation

This incident has been resolved.

We are investigating an issue affecting live streams. Some viewers may experience dropped or interrupted live streams. Our team has identified the underlying infrastructure issue and is actively working on a resolution. We will provide updates as more information becomes available.

The issue has been identified and a fix is being implemented.

A fix has been implemented and we are monitoring the results.

We are continuing to monitor for any further issues.

This incident has been resolved.

We are investigating an issue with collecting Mux data beacons. Beginning 22:12 UTC, view session data will be delayed.

A fix has been implemented and we are monitoring results. Delayed data was for a subset of customers between 22:12 UTC - 22:44 UTC. No data loss.

This incident has been resolved.

Between 16:15 and 16:20 UTC a small number of live streams were disconnected. Active live streams were able to reconnect and resume, and there is no long term impact to DVR or VOD content from affected live streams other than resuming on a new Asset ID. All systems are currently operational and we are monitoring the live ingest systems.

No further issues observed during monitoring period. Live streams are operational.

We are investigating an incident which caused the disconnection of multiple live streams at 13:11 UTC.

A small number of live streams were disconnected around 13:10 UTC, ending the corresponding Asset. Active live streams were able to reconnect and resume, and there is no long term impact to DVR or VOD content from affected live streams other than resuming on a new Asset ID. All systems are currently green.

We are experiencing increased errors with delivery across Live and VOD in Europe regions  and are actively investigating.

We've identified an issue with a CDN provider in the impacted regions. We are seeing recovery, and continuing to monitor.

Video delivery for Live and VOD in the impacted region has returned to normal operations.  The impacted time frame was 9:30 - 10:30 UTC.  We are continuing to monitor.

This incident has been resolved.

We are investigating an issue causing an increase in errors in the Mux Data dashboard and API.

We have identified the issue and are implementing a fix.

A fix has been implemented and we are monitoring.

This incident has been resolved.

From 21:02 UTC to 22:55 UTC some live streams may have experienced elevated rebuffering. We have identified and mitigated the issue.

We are seeing high webhook delivery latency. We are actively investigating

We have identified the issue and have deployed a fix. We are continuing to monitor.

This incident has been resolved.

From 03:08UTC to 03:14UTC we saw increased request error rates and webhook delivery latency increases. The issue is mitigated and systems are operating normally.

We are currently investigating errors in Mux Data historical metrics beginning at approximately 22:08 UTC. We apologize for the inconvenience and will share an update once we have more information.

Our team has identified the cause of elevated Data Historical metrics errors and is actively working on a fix. We’ll continue to share updates as we have them.

Our team has implemented a fix and we are monitoring the results. We’ll post an update as soon as the issue is fully resolved.

This incident has been resolved

We are currently investigating an issue where playback may experience elevated rebuffering rates and playback failures.

The issue has been identified and a fix is being implemented.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

We are currently investigating the issue

The issue has been identified and a fix is being implemented.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

Mux Data Views for EU customers are delayed beginning 22:53 UTC. We are investigating

The issue has been identified and a fix is being implemented.

Our pipeline latency is resolved and we are caught up as of 1:50 UTC

This incident has been resolved.

We are seeing some latency in both Monitoring and Historical views starting 15:53 UTC for EU customers and are currently investigating

We are monitoring for additional latency. EU saw elevated latency 15:50 UTC to 16:04 UTC. Monitoring data was inaccurate during that window and views were delayed. No data was lost.

This incident has been resolved.

Small percentage of static rendition downloads. We are currently investigating this issue.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.