История сбоев Streamtime

We are currently investigating this issue.

The authentication issue has been resolved.

Currently investigating this issue. We’ll have an update shortly.

We have identified and issue and are investigating a solution.

The incident has been resolved.

We have been made aware of some customer receiving a 'Your connection is not private' browser message when navigating to their Streamtime URL. Our team are investigating as a matter of urgency.

Our technical team have identified a domain issue that they are investigating a solution for.

Our continued monitoring of the propogation is showing a decline in the small number of customers that have been affected. We are hopeful that the changes made will continue to update DNS servers and resolve access for those still affected. We will continue to closely watch progress and exhaust all avenues available to us to fully resolve the issue as quickly as possible. Our team are available via email ([email protected]) should you have any additional concerns. We apologise and do appreciate the impact this continues to cause.

We’re continuing to see a steady decline in the number of customers affected. Our team are actively exploring every option and are monitoring closely to ensure progress continues. Thank you for your continued patience.

While there are no major changes to report just yet, we’re continuing to monitor things closely. Our team remains on hand to help so please reach out to [email protected] if there’s anything you need.

We have now exhausted all technical paths on our side. There are third parties involved that, unfortunately, we cannot control but we will be working closely with them to get this resolved as quickly as possible. We're still keeping an eye on things and are here if you need any support (feel free to reach out via [email protected]).

Our technical team have confirmed with all relevant 3rd parties that the change made at 10.10pm AEST yesterday evening did in fact resolve the core issue. The subsequent delay in the solution reaching some of the affected parties was caused by the propagation process. If your access to Streamtime remains affected, please contact [email protected] for our technical team to investigate your specific situation. Our focus now turns to reviewing the cause and effect of this issue so we can ensure any critical failure points are resolved and future scenarios are avoided. We will endeavor to post an update here in the coming days. Once again, we apologise for the inconvenience this undoubtedly caused a small number of customers. Michael O'Riley Head of Operations

# **Streamtime Technical Postmortem - Domain Suspension** **Incident ID:** STM-2025-0522-001 **Date:** May 27, 2025 **Duration:** 15\+ hours \(ongoing propagation\) **Severity:** High **Impact:** Customer access disruption due to domain suspension. ## **Summary** On May 22, 2025, at 18:00 AEST, customers began experiencing issues accessing Streamtime presenting as either a verification of domain request page, or an SSL certificate validation failure "Your connection is not private". The incident was caused by an incomplete ICANN registration update following domain technical contact changes during routine domain renewal. ## **Timeline \(AEST\)** * **May 7, 2025:** Domain renewal completed. Technical contact details updated to new email address. Key account owner details unchanged. * **May 10, 2025:** Automated email sent from domain provider to a generic, non-specific, non-technical email address asking for new technical contact details to be confirmed. No other contact received. * **May 22, 2025 18:00:** First customer report received of certificate validation issues. * **May 22, 2025 20:30:** Single email requesting confirmation of technical contact details found and actioned. Investigations continued for other possible root causes. * **May 23, 2025 06:00:** Technical support for domain provider available - discussions begin to confirm root cause. * **May 23, 2025:** DNS propagation ongoing. * **May 23, 2025:** Northern hemisphere timezone online with confirmation from majority of previously affected customers that the issue had resolved. Some customers remain affected. * **May 24, 2025:** Propagation process reaches 48hrs at 20:30 AEST. * **May 26, 2025:** Confirmation from all previously affected Southern Hemisphere customers that the issue is resolved. * **May 26, 2025:** No sign of domain related issues in our network monitoring system. * **May 26, 2025:** Confirmation from domain host that domain was suspended due to the updated technical contact email address not being verified. ## **Root Cause Analysis** The incident originated from an ‘incomplete’ domain registration update process: 1. Technical contact information was successfully updated in the domain host administration panel during renewal. 2. Automated email received in a generic email inbox asking for updated technical contact details to be confirmed. 3. Timeframe for validation of details expired without additional warning or contact from the domain provider. Domain suspended without notification. 4. Due to lack of confirmation, ICANN registration records were not updated, leading to a domain suspension. ## **Resolution** 1. Located and verified the legitimacy of the confirmation email from the domain host. 2. Completed the technical contact confirmation process. 3. ICANN registration records subsequently updated. 4. Initiated DNS propagation across global DNS infrastructure & monitored progress. ## **Customer Impact** Due to the distributed nature of DNS propagation, impact varies by customer location and ISP DNS configuration. Some customers may experience delayed resolution or temporary new occurrences of the issue as DNS updates propagate globally. It is generally accepted that global propagation can take up to 48hrs. ## **Immediate Actions Taken** * Troubleshooting of root cause. * Incident created at [https://streamtime.statuspage.io/incidents/hk5ysgq63z9y](https://streamtime.statuspage.io/incidents/hk5ysgq63z9y) with incident subscription option to receive updates automatically. * Consistent, proactive communication to customers on progress. * Confirmed technical contact details with domain registrar. * Confirmed the ICANN registration records would be automatically updated. * Established monitoring for continued customer reports. * Provided customer workaround guidance \(alternative network connections\). * Contacted all 3rd party stakeholders and continued to push for information to confirm root cause. ## **Process Improvements** 1. **Domain Management Consolidation:** We have initiated the process to transfer domain management from our existing provider to GoDaddy for 24/7 domain hosting support availability. 2. **Notification Enhancement:** Update notification systems to target both individual team members and shared support inboxes. ## **Next Steps** * Continue to monitor support channels for customer responses as to their connectivity. * Continue to monitor relevant errors received by customers via our network monitoring system. * A full review of all critical accounts to be scheduled and completed internally. Confirming correct notifications are enabled and are sent to monitored email accounts. * Review and ensure existing internal disaster recovery plan is updated based on any account changes.

We have had some reports of Streamtime not being accessible. Our team are aware of these reports and are working on it as our highest priority. More updates to follow.

The issue has been identified as a malformed URL. Steps to resolve: - in the URL that is currently returning the 404 page, remove all text AFTER ...streamtime.net - hit return, page should load Alternatively: - Visit streamtime.net - click login in upper right - you should be able to log in as normal

This incident has been resolved.

Our engineers are currently investigating an outage as our highest priority. We are aware of some customers unable to access their Streamtime account. We will update via this incident as/when we have further information or when the issue is resolved.

We are continuing to investigate this issue.

Our engineers identified and resolved the issue. Please contact Streamtime Support if you continue to experience issues. Our team will continue to investiage the outage to ensure we avoid any recurrence if at all possible.

We have been made aware of some interruptions to the Streamtime service in the last 30minutes. After immediate investigation, the issues appear to be linked to the AWS services currently affected by operational issues - https://health.aws.amazon.com/health/status. The operational issues may still be impacting a small number of customers due to browser caching of Streamtime's server details. We expect any outstanding 504 errors to be resolved with the AWS operational issues. Please contact us directly should you have any questions or continue to be affected.

The issue has been identified and we are monitoring the AWS status updates. https://health.aws.amazon.com/health/status

We are continuing to monitor the outstanding AWS oeprational issues.

AWS continue to work on their operational issue https://health.aws.amazon.com/health/status. We have had one additional instance of a 504 error in Streamtime which means connectivity is still being sporadically affected. When an issue is identified by our AWS server, a new server is created for Streamtime customers to connect and access Streamtime. Hitting a 504 error means you will be routed to the newly created AWS server asap. Trying to access Streamtime via an incognito browser window will avoid cache issues and likely have you connected to the new AWS server and Streamtime faster. We will continue to monitor the AWS issue progress.

AWS have resolved their operational incident - https://health.aws.amazon.com/health/status. We do not expect to see any further access errors.

We are aware of this issue and are currently working on a fix.

We are continuing to investigate this issue.

The issue has been identified and a fix is being implemented.

This incident has been resolved.

We have had some reports of Streamtime app not being accessible. Our engineering team are investigating as we speak. We will update ASAP.

Our team have identified the issue and implementing a solution to our AWS server machine.

We have resolved the issues affecting Streamtime and all services have been restored

Our engineering team is currently investigating multiple reports of poor performance whilst attempting to access Streamtime and in some cases users are completely unable to login. We will post further updates here ASAP.

We are continuing to investigate this issue.

Normal server operations are restored. We are closely monitoring the situation.

Our monitoring has not revealed any further problems. The issue was identified to be related to our Apache server and has now been rectified. To ensure the issue does not reoccur we have increased certain limits on our server and will monitor over the coming days. This issue is now marked as resolved.

We are currently experiencing widespread performance issues. Our engineering team are investigating and we hope to have a resolution shortly. Updates to come.

Investigation continues and is focused on the Apache server. We hope to confirm shortly and of course work on a resolution. Apologies for the disruption 🙏

Our engineering team have identified the cause of the problem and are working on a resolution to the issue.

A fix has been implemented and we are monitoring the results.

Monitoring and tests have completed successfully. This issue is now resolved. Again, apologies for the disruption caused this morning. Our engineering team will conduct a review into the cause of the issue and make any necessary changes to avoid similar occurrences in future. Any questions/issues please get in touch via our in-app/website messaging function.

We are currently investigating this issue.

All fixed. No longer an incident!