Störungsverlauf von Drmtoday

Due to the ongoing outage of AWS in the us-east-1 region we are currently redirecting all license traffic to other regions at 7:40 UTC. While we were not experiencing errors in regards to license delivery, monitoring for the region was affected and we decided to cut off the region as a precaution. We keep monitoring the situation and will take us-east-1 into operation again once AWS resolves all issues. (This message comes delayed as the AWS outage also affected Atlassian Statuspage operations.)

AWS has reported that the issues have been resolved. We have enabled the us-east-1 region again.

We are currently investigating this issue.

Multiple faulty instances were replaced and our services no longer show excessive error rates. We are monitoring the situation.

This incident has been resolved.

A detailed Root Cause Analysis \(RCA\) is available for this incident and can be provided upon request. To obtain a copy, please submit a ticket via our helpdesk. The report includes an overview of the incident timeline, contributing factors, and the corrective measures implemented.

We are working on an issue with license delivery logs. A cluster node of the delivery log primary storage has been replaced. We are currently working on restoring data. Please be aware that data shown in the Dashboard or fetched from the reporting API may be incomplete or invalid. License delivery is not affected.

Missing data has been restored. License delivery data on the Dashboard and from the reporting API is now complete again.

We are working on an issue with license delivery logs. Please be aware that data shown in the Dashboard or fetched from the reporting API may be incomplete or invalid. License delivery is not affected.

Missing data has been restored. License delivery data on the Dashboard and from the reporting API is now complete again. We will continue with upgrading the underlying infrastructure.

This incident has been resolved.

We are working on an issue with license delivery logs. A cluster node of the delivery log primary storage has been replaced. We are currently working on restoring data. Please be aware that data shown in the Dashboard or fetched from the reporting API may be incomplete or invalid. License delivery is not affected.

Missing data has been restored. License delivery data on the Dashboard and from the reporting API is now complete again.

We identified and remediated an issue with license delivery logs. Please be aware that delivery log data generated between 03:30 and 04:30 AM (UTC) is incomplete. This affects both, the Dashboard as well as the reporting API. Missing data will be restored during business hours on Monday. Once this step is complete we will update this incident page. License delivery is not affected.

Missing data has been restored and is now complete again.

We are currently investigating this issue.

Currently a major portion of the requests to googles widevine services are failing. We are currently in contact with google to resolve the incident.

Google is currently working to address the issue.

Google has resolved the issue with their license servers in all regions.

We are currently investigating an increases error rate on our ingest APIs, config APIs, and Dashboard. License delivery is not affected.

This incident has been resolved. The increased error rate occurred between 01:28 - 02:05 (UTC) and was caused by high system load and a misconfigured rate limiting setting. We apologize for the inconvenience.

An update has been deployed on our staging system to fix an issue with failing Google Widevine requests.

We were seeing an interruption of the license delivery service in ap-northeast-1. A DNS failover to all other regions was initiated automatically.

The automated fail-over started at 7:03 UTC, following a sharp increase in requests to the ap-northeast-1 region around 7:01. We've increased capacity to handle the additional load in the region. We are monitoring the situation.

The incident has been resolved. The region recovered at 7:07: the automated fail-over to other regions ended and traffic from the area was routed again back to ap-northeast-1.

We are investigating an issue with license delivery logs. Please be aware that data shown in the Dashboard or fetched from the reporting API may be incomplete or invalid. License delivery is not affected.

A cluster node of the delivery log database was replaced. We are currently working on restoring data.

The data have been restored and the backlog has been worked off. We will verify data in the cluster during business hours over the next days and update this page in case there are any findings such as missing license log data.

We're seeing connection issues with Google Widevine service in the us-west-1 region. License delivery services are currently routed to other regions.

The incident has been resolved. Traffic in the area is routed to the region again.

We were seeing an interruption of the license delivery service in ap-northeast-1. A DNS failover to all other regions was initiated automatically.

The failover started at 12:05 UTC, following a sharp increase in requests in ap-northeast-1 around 12:03:50. The region and the whole system recovered automatically and became fully operational again at 12:15. During the time of the failover, we observed degraded perfomance.

The failover was caused by a sudden increase in traffic in the ap-northeast-1 region over the course of a few seconds which caused the running instances to handle too much load. New instances were started automatically but their startup took longer than the increase in traffic, which triggered the failover. The DNS failover spread the requests over multiple regions, which handled the peak with a much slighter increase in load and after the startup of the new instances, ap-northeast-1 recovered in the aforementioned timeframe. As a countermeasure, we increased the number of instances running in ap-northeast-1 which now handle the peaks without degraded performance. A note to the previous status message: We added a small correction in the time. We actually recovered at 12:15 UTC instead of 14:15 (timezone error).

Seemingly google started enforcing permanent device revocations today. As this affects quite a large number of requests, we are in discussion with Google to find out if this is intended.

Google provided the following information: 9.00am PST update Widevine has identified the issue that's causing these intermittent license failures to many Android devices, due to a revoked L3 certificate. We are correcting this asap. Next update in 2 hours.

Google is taking care about the situation and will provide an update in 1.5hrs.

Google has started rolling out a fix. Most requests are already working again.

The issue has been resolved by the Google Widevine team. We logged the last error for this incident at 18:41 UTC. Google Widevine provided the following statement: A Level 3 DRM certificate was revoked in the Widevine ecosystem between June 22, 11pm PST to June 23, 2am PST. This revoked certificate resulted in high licensing error rates for a wide variety of Android mobile devices. SystemID: 8158 Name: Android L3 ARM for Q (10) and R (11) This action has been reverted back to its unmodified original state.

We are currently investigating increased ingest error rates in us-west-1.

The issue was caused by excessive CPU usage on the key storage database due to a failing automated maintenance task. License delivery was not affected. We apologize for the inconvenience.

We're currently investigating an incident in the eu-west-1 region. License delivery was temporarily routed to other regions.

Performance of license delivery in eu-west-1 was degraded between 17:59:40 and 18:04:10. Traffic was routed to other regions between 18:02 and 18:05. (All times UTC.) Root cause: One of our internal services did not scale fast enough to handle an unusually high increase in load. We will address this issue by increasing our proactive scaling for European prime time.

We are investigating an issue with license delivery logs. Please be aware that data shown in the Dashboard or fetched from the reporting API may be incomplete or invalid. License delivery is not affected.

There was an issue with shard allocation on the database cluster that has been resolved. We're now processing the backlog and start restoring missing data. Please be aware that data shown in the Dashboard or fetched from the reporting API is still incomplete.

The backlog has been worked off. We will verify data in the cluster during business hours in the next days and will provide updates on our status page. Please be aware that license delivery data in the Dashboard and from the reporting API may still be incomplete or inconsistent until this verification has been completed.

Verification showed no issues. License delivery data in the Dashboard and from the reporting API is complete and consistent.

We are investigating an issue with license delivery logs. Please be aware that data shown in the Dashboard or fetched from the reporting API may be incomplete or invalid. License delivery is not affected.

There's an issue with one of the database cluster nodes where license delivery logs are stored. We're currently trying to recover the faulty node.

We've replaced the failed node and started to restore data. Please be aware that data shown in the Dashboard or fetched from the reporting API is still incomplete.

Data until around midnight UTC has been restored from the latest snapshot. We're now indexing the backlog of live-data from today.

The backlog has been worked off and missing data has been restored. License delivery data in the Dashboard and from the reporting API is now complete and consistent again.

We are currently investigating issues in the ap-northeast-1 region. License delivery is currently routed to other regions.

We were seeing an interruption of the license delivery service in ap-northeast-1. The underlying cause was a database failover at 17:20. Between 17:22 and 17:29 as well as between 17:34 and 17:38 a part of the incoming license requests was rerouted to other DRMtoday regions.

We are currently investigating an issue with our frontend services affecting key ingest, the dashboard, and configuration API.

The issue was caused due to unexpected high load on our ingest API. We're currently scaling up instances to handle the increased load. We are also applying rate limiting for ingest requests for the moment to help the system stabilize.

We have currently disabled the US region for maintenance of the key database. License requests from the area are routed to other regions now.

Maintenance of the key database in the US region has been completed. License requests are processed in the region again.

This incident has been resolved.