25 Eventive incidents · July 2019 — official updates, affected components, duration and resolution details.
An API failure occurred during a planned API scale operation due to a configuration error with the scheduling software. This has since been fixed and the root cause has additionally been mitigated.
We are currently experienced delays with the sending of emails and mailings. All other functionality is working fine. We will update as we have further information
This incident has been resolved. All email functionality has returned to normal.
We're experiencing an elevated level of API errors due to a widespread Amazon Web Services outage. Actions may continue to work intermittently. If an action fails, please wait a moment and try again.
We are continuing to investigate this issue.
This incident has been resolved.
We are experiencing a platform wide outage and are actively investigating
The platform was down from 4:42 CDT to 5:28 CDT due to an issue with our server farm. This issue has been resolved.
There was a 9 minute service outage from 9:34am - 9:42am CDT involving an API service that failed to start. It was quickly resolved.
There was a short incident with the ordering database which causes a brief outage from 8:32pm EDT to 8:52pm EDT. This issue has been resolved and service has returned to normal
The issue has been resolved and payment processing has returned to normal.
This incident has been resolved.
Partial outage - a webhook outage resulted in some successful transcodes not updating status in the dashboard, resulting in an extended delay in the content ingestion process. The issue has been resolved and all new uploads are processing as expected. For any assets still stuck in an extended (over 3 hour) VOD transcoding state, please remove the asset and try uploading again.
We are currently investigating this issue.
This issue was caused by an upstream service provider DNS outage. The incident has been resolved by the upstream provider and we are monitoring results.
This incident has been resolved.
Our video encoding provider has reported an issue with cloud server architecture that is causing virtual content trancode failures. The cause has been identified and a fix is in progress. All affected uploads will have to be re-submitted once the fix is complete. Estimated resolution by 09:00 UTC on 12 September.
We are continuing to work on a fix for this issue.
This incident has been resolved.
The Eventive platform is currently inaccessible due to a networking related incident at one of our DNS providers. We will provide an update within 30 minutes.
One of our service providers, Heroku, is currently experiencing a DNS-related incident that is preventing users from accessing the Eventive platform. They are actively working to mitigate the issue. We will provide another update within 45 minutes.
Heroku is still actively working to mitigate the issue. We will provide another update within 45 minutes.
We have implemented a temporary mitigation to bypass the Heroku DNS issues until they resolve the underlying incident. As a result of our temporary mitigation, the Eventive platform should now be accessible again.
This incident has been resolved.
We are currently investigating an issue impacting access to the Eventive organizer dashboard. We will provide another update within 30 minutes.
One of our service providers, Heroku, is currently experiencing a widespread platform outage that is impacting accessibility of the Eventive platform. The Eventive platform may be inaccessible to users. We will provide an update within 30 minutes.
At this time we are observing that accessibility to the Eventive platform is gradually being restored. Heroku has indicated that a gradual restoration is in progress. We will provide another update within 30 minutes.
This incident has been resolved.
We are currently investigating an elevated rate of Premium Livestream failures. Failures may manifest as only a black screen visible (instead of expected content) in preview and/or playback mode. As a temporary remediation while we continue our investigation, we recommend utilizing Basic Livestreams, which do not appear impacted by this issue.
In collaboration with our primary premium livestreaming partner Microsoft Azure, we continue to investigate the issue impacting Premium Livestreams. At this time, Basic Livestreams are recommended to ensure this error scenario is avoided.
The issue has been identified and a fix is being implemented. We will provide an update within 1 hour.
We have resolved the issue impacting Premium Livestream playback. All functionality has been restored. We sincerely apologize for the service disruption, and we will conduct an extensive review to ensure such an issue does not occur again.
We are currently investigating an elevated rate of encoding failures for newly uploaded assets. At this time, please do not retry failed uploads. We will provide an update within 60 minutes.
At this time we are continuing to investigate the elevated rate of failures impacting new asset encodings on the Eventive platform. We will provide another update within 90 minutes.
An issue at an external vendor that is causing some new asset encodings to fail has been identified. The vendor is working towards a remediation now. We apologize for the inconvenience, and will provide another update within 3 hours.
This incident has been resolved. All affected assets are being automatically queued for re-encoding. No further action should be necessary. We apologize again for the inconvenience.
We are again observing an increase in DRM license key delivery failures. A current widespread Amazon Web Services outage impacting platforms including Disney+, Netflix, and others, is also affecting the Eventive DRM licensing system. Viewers may be unable to view protected content. This incident will remain open until the issue is exhaustively confirmed resolved by AWS.
The incident has been resolved. Playback success has returned to normal levels, and AWS has confirmed a resolution of the underlying networking issue.
This incident has been resolved.
A current widespread Amazon Web Services outage impacting platforms including Disney+, Netflix, and others, is also affecting the Eventive DRM licensing system. Viewers may be unable to view protected content. We will provide another update soon.
A fix has been implemented by Amazon, and DRM playback success is returning to normal levels.
A Microsoft Azure DNS outage affecting a number of top internet properties is currently preventing playback of some assets. Azure is investigating the incident. We will provide an update soon.
Microsoft continues to investigate the issue impacting Azure DNS. Although their primary status page is impacted by the issue, updates are available at https://status2.azure.com/ We will provide an additional update soon.
Microsoft has implemented a fix and we are beginning to see traffic return to normal levels. Please try viewing content again. We will continue to monitor the resolution process.
This incident has been resolved.
One of our hosting providers, Vercel, is currently experiencing a partial outage that may impact the accessibility of the Eventive Virtual screening platform. Trying another device, browser, or wireless network may improve accessibility. We will provide another update shortly. You can follow along for updates here: https://www.vercel-status.com/
A mitigation has been put in place by our hosting provider and access should soon be fully restored for affected customers.
This incident has been resolved by our upstream provider.
Our primary DRM provider, Microsoft Azure, is currently experiencing an outage impacting DRM license key delivery. Viewers may be unable to view protected content. We will provide another update soon.
The issue affecting DRM license key delivery has been identified and a fix is being implemented by Microsoft. We will provide another update soon.
A fix has been implemented and DRM-protected playback should now be restored for all customers and working properly. We will continue to monitor the resolution and will provide another update within an hour.
Playback has returned to normal levels. We are continuing to monitor for any further issues.
This incident has been resolved.
Our live chat provider, Intercom, is currently experiencing a partial outage that reduces our ability to provide live support. Please contact us via email, [email protected], with any urgent issues. https://www.intercomstatus.com/
Our live chat provider has implemented a fix for this issue and we are monitoring results.
This incident has been resolved.