7digital outage history

Dear Clients, We are currently experiencing an outage affecting our HLS streaming endpoints: GET /stream/subscription.m3u8 GET /offline/subscription.m3u8 GET /stream/catalogue.m3u8 Progressive streaming and download endpoints are not affected. Our on-call support engineers are investigating and working to restore stability. We will send a further updates in due course. If you have any questions, please contact the Client Success Team. Kind regards, Massive Music Client Success Team

Following our earlier notice, we have identified the cause of the outage affecting our HLS streaming endpoints. Our CDN provider is investigating an issue with its On-the-Fly Packaging service, which generates our HLS streams. Our engineers are in close contact with the CDN provider, monitoring progress and working to restore service as quickly as possible. We will send a further update once service is restored.

The issue affecting our HLS streaming endpoints has been resolved, and we are monitoring the service. The outage was caused by a problem with our CDN provider’s On-the-Fly Packaging service, which generates our HLS streams. While the problem lasted, requests to the following endpoints returned 503 errors. The issue affected service between 16:36 and 18:51 UTC on Monday 26 October. We are continuing to monitor the platform closely and will review the incident.

This incident has been resolved.

Dear Clients, We are currently investigating a streaming outage affecting the us-east-1 region. Our engineering team is actively working on a resolution, and we will provide further updates as they become available. Further notices will be sent regarding this incident as they become available. With best regards, 7digital Client SuccessTeam

A fix has been implemented for the streaming issues in us-east-1. We are currently monitoring the results and will provide a final update shortly.

This incident has been resolved.

Dear Clients, Between 10:00 AM and 10:25 AM UTC, we experienced elevated latency/error rates on our streaming APIs. Our engineering team identified the root cause and implemented a fix. Service has returned to normal operations, and all systems are now monitored and stable. With best regards, 7digital Client SuccessTeam

We're currently experiencing an outage with our Catalogue Search. This impacts users of the search API and the 7digital playlist tool with search currently unavailable within the playlist tool. The 7digital engineering team are working to resolve the issue and we'll provide further updates as soon we have them. Regards, 7digital Client Success Team

A fix has been implemented to restore service to search and we're continuing to monitor the issue. Regards, 7digital Client Success Team

This incident has been resolved.

As of ~08:00 BST we started to observe errors in our Catalogue Events product which meant we were unable to push out any event messages. We've identified the issue and are implementing a fix to resolve the impact on Events. Once the service is restored we'll follow up with further updates and information.

On further investigation we've identified that an issue with a 3rd party supplier is impacting our ingestion platform. Despite attempting to implement a resolution internally, this has resulted in an outage impacting our ability to ingest content from suppliers. All catalogue updates will be delayed until our ingestion platform is restored to full health. To clarify after the initial status message, our Catalogue Events product is operational but due to the underlying issue with our ingestion platform we won't be generating any catalogue updates through Events as a result.

Our 3rd party supplier has updated us that they're seeing significant signs of recovery to their service. As a result some of the impacted services on 7digital's platform are also starting to restore. We'll continue to monitor progress and have taken the decision to leave catalogue ingestion switched off until we're confident the platform is restored to full health. At this stage we plan to switch on catalogue ingestion shortly after 13:00 UTC if things continue to progress as expected. We'll update this incident at that time.

Catalogue ingestion has now been switched on and the platform has been restored to full health. The team will continue to monitor and be on hand to react to any further incidents.

Dear Clients, Due to unforeseen circumstances with our data warehouse, the incremental artist feed for 20240304 has been generated, however, it will not contain any data. To ensure you have access to the most up-to-date catalogue, we advise continuing as normal by ingesting the incremental artist feed for 20240305, which will cover data for both 20240304 and 20240305. We apologise for any inconvenience this may cause and want to assure you that we are actively working to resolve this issue and prevent future occurrences. If you have any questions, please contact your Client Success Manager. If you are a client who does not ingest our feeds then please ignore this email. With best regards, 7digital Client Success Team

This incident has been resolved.

We're observing an elevated error rate against stream requests to the platform. The 7digital engineering team are working to resolve the issue and we'll provide further updates as we have them.

A fix has been implemented to deal with the elevated error rate and we're now seeing error rates returning to a normal level. We're continuing to monitor the platform and will take any further action as necessary.

This incident has now been resolved. We'll continue to monitor the platform and we'll continue to have engineering support staff on hand should any further issue occur.

We're currently observing an unprecedented volume of demand on the platform, and as a result multiple areas of the API are currently unavailable. We're in the process of scaling up the platform to support this increased load and restore availability. As soon as we have further updates they'll be posted here.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

Dear Clients, During this time, we have noticed a higher than usual error rate on calls to various APIs and longer response times. Our team of on-call support engineers is currently investigating the issue and taking steps to improve the stability of the platform. We will provide you with additional details in a further notice once we have completed our analysis of the problem. If you have any questions, please create a Service Desk ticket with the Client Success Team. With best regards, 7digital Client Success Team

Dear clients, The platform degradation/outage experienced today is now closed. The platform is stable and the error rate has dropped. Our Tech team will continue to investigate. If you continue to experience issues with the 7digital platform create a Service Desk ticket with the Client Success Team. With best regards, 7digital Client Success Team

This incident has been resolved.

We are currently experiencing an outage on our content ingestion systems. We last processed new content and updates on Saturday morning (UK time). We are working towards a resolution now and plan to provide an update shortly.

We've identified the fault and are currently in the process of restoring our ingestion platform. Once the ingestion platform is functional again we can begin ingesting pending deliveries and will provide a further update at that time.

A fix has been implemented to solve the problem, allowing us to resume ingestion. The backlog will be cleared gradually as we ramp up processing capacity.

This incident has been resolved.

We're currently investigating an issue with the download API where most, or all download attempts are failing. As soon as we have further information we'll provide additional updates.

We've rolled back our Download API to a previous state which has restored downloads successfully. We'll continue to monitor to ensure the issue is resolved and will follow up with further information once we've compiled a post incident report.

Dear Clients, We are currently seeing a degradation of service across most APIs. This issue is only affecting uncached media as cache hits are served from our CDN and are therefore unaffected. Our on call support engineers are currently investigating the issue and taking action to further stabilise the platform. Once we have completed analysis of the issue we'll send an additional notice out with further details. Further notices will be sent regarding this incident as they become available. With best regards, 7digital Client SuccessTeam

Dear clients, The platform degradation. The platform is stable and the error rate has now returned to normal performance levels. Our Tech team will continue to investigate and monitor performance. If you continue to experience issues with the 7digital platform create a Service Desk ticket with the Client Success Team. With best regards, 7digital Client Success Team

This incident has been resolved.

At 13:46 BST we were alerted to an issue with one of our two network lines. After investigation, at 14:04 BST we removed the network at fault from service, directing all traffic to our secondary line which restored normal service to the platform. We are continuing to investigate what caused the failure and will restore network redundancy as soon as possible.

Dear clients, The 7digital Playlist Tool is currently unavailable. Once we have further updates we'll share them with additional announcements. You can subscribe to updates via email, webhooks and RSS feed on our statuspage (insert link). If you would like to receive SMS updates, please create a Service Desk ticket with the Client Success Team. If you have any questions, please create a Service Desk ticket with the Client Success Team. With best regards, 7digital Client Success Team

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

We are currently investigating a suspected platform outage. Currently all endpoints on 7digital's API are unavailable, 7digital engineers are investigating the cause and we'll provide further information as soon as it's available.

We've now isolated the cause of the outage to our DB cluster. We're observing an issue between the primary and replica node which is hindering our ability to bring the cluster back online. We're continuing to investigate the issue and will provide further updates as they're available.

We have been able to successfully bring back online the affected DB cluster, however we're still experiencing problems keeping the DB cluster online permanently, causing the platform to be unavailable. We are continuing to investigate and will provide further updates shortly.

We are continuing to work on a fix for the issue and will provide an update again shortly. We can also confirm that cached streams continue to be served throughout the incident. We're reviewing activity to identify other areas of the platform which are only partially unavailable and will communicate updates with further information shortly.

We are continuing to work to resolve the ongoing outage. The cause has been isolated to our SQL Server cluster, which is currently not able to keep certain databases online - choosing to take them down for as of yet, an unknown reason. Our efforts to force a single node to host the databases have so far not been effective at solving the current issue. We believe at this stage, that our cluster configuration has a non-trivial problem, and we are moving to bring up databases separately outside of the high-availability cluster to restore service as soon as we can. Following the return of stability, we will look to restore high-availability as soon as we can.

As per the previous update a fix has been implemented and service has been restored to the platform, with the exception of ALC purchasing, locker and ALC/permanent download endpoints. We are continuing to monitor the fix and will also be working on restoring high-availability to the platform.

Full service has been restored to all components of the platform and monitoring will continue. We're confident service has been restored and we will follow up with an incident report next week once we've been able to fully evaluate the issue and the actions taken to remedy it.

The incident report for this outage is now available and can be found [here](https://drive.google.com/file/d/1MK4JVxFuKtHnnvekUzSk2aT61Z0zo43G/view).

Dear Clients, The Download APIs are currently down due to an outage. Streaming and Media Transfer endpoints are not affected. A new EC2 instance is being set up by the Tech team. Further updates will be announced as soon as they become available. If you have any questions, please create a Service Desk ticket with the Client Success Team. With best regards, 7digital Client Success Team

The download outage experienced today is now closed. The endpoint is back to normal. If you continue to experience issues with the 7digital platform create a Service Desk ticket with the Client Success Team. With best regards, 7digital Client Success Team

This incident has been resolved.

Dear Clients, Since 9.20am GMT we saw a degradation across all APIs. Engineers on-call are currently investigating the issue and are taking action to further stabilise the platform. A further update will be communicated in due course. If you have any questions, please create a Service Desk ticket with the Client Success Team. With best regards, 7digital Client Success Team

As of 10:50 GMT full service has resumed but we're continuing to monitor the platform.

Dear Clients, We are currently experiencing a delay with processing UMG content. A bug was detected in the UMG Teleporter application causing a backlog of content to develop on their side. Therefore, as a result, content will be delayed from entering the daily feeds. A fix has been deployed and UMG have been notified of the issue. We are currently processing the backlog which is estimated to take 24 hours. If you have any questions, please create a Service Desk ticket with the Client Success Team. With best regards, 7digital Client Success Team

We are currently experiencing a delay with processing UMG content. At 8pm 01/07 a bug was detected in the UMG Teleporter application causing a backlog of content to develop on their side. Therefore as a result, content delivered between 8pm 01/07 and 11am 02/07 may be delayed from entering the daily feeds. A fix has been deployed and UMG have been notified of the issue. We are currently processing the backlog.

A fix has been deployed and UMG have been notified of the issue. We are currently processing the backlog.

This incident has been resolved.

Dear Clients, We are currently seeing a degradation of service with our CDN provider. Further notices will be sent regarding this incident as they become available. With best regards, 7digital Client SuccessTeam

The CDN outage is affecting all 7digital API and Media Delivery endpoints. Our on-call support engineers are currently investigating the issue and in contact with our CDN provider. You can subscribe to updates via email, webhooks and RSS feed on our statuspage (https://status.7digital.com/). If you would like to receive SMS updates, please create a Service Desk ticket with the Client Success Team. Once we have further updates we'll share them with additional announcements. If you have any questions, please create a Service Desk ticket with the Client Success Team.

We are continuing to work on a fix for this issue.

The error rate has dropped since 11:53 GMT. Our Tech team will continue to monitor the platform before we close this incident. For the most current updates, you can follow Fastly Status page here: https://status.fastly.com/

Dear Clients, The CDN outage experienced today is now closed. The platform is back to normal and the error rate has completely dropped since 13:37. With best regards, 7digital Client Success Team