Streamtime Status

The incident has been resolved.

# **Streamtime Technical Postmortem - Domain Suspension** **Incident ID:** STM-2025-0522-001 **Date:** May 27, 2025 **Duration:** 15\+ hours \(ongoing propagation\) **Severity:** High **Impact:** Customer access disruption due to domain suspension. ## **Summary** On May 22, 2025, at 18:00 AEST, customers began experiencing issues accessing Streamtime presenting as either a verification of domain request page, or an SSL certificate validation failure "Your connection is not private". The incident was caused by an incomplete ICANN registration update following domain technical contact changes during routine domain renewal. ## **Timeline \(AEST\)** * **May 7, 2025:** Domain renewal completed. Technical contact details updated to new email address. Key account owner details unchanged. * **May 10, 2025:** Automated email sent from domain provider to a generic, non-specific, non-technical email address asking for new technical contact details to be confirmed. No other contact received. * **May 22, 2025 18:00:** First customer report received of certificate validation issues. * **May 22, 2025 20:30:** Single email requesting confirmation of technical contact details found and actioned. Investigations continued for other possible root causes. * **May 23, 2025 06:00:** Technical support for domain provider available - discussions begin to confirm root cause. * **May 23, 2025:** DNS propagation ongoing. * **May 23, 2025:** Northern hemisphere timezone online with confirmation from majority of previously affected customers that the issue had resolved. Some customers remain affected. * **May 24, 2025:** Propagation process reaches 48hrs at 20:30 AEST. * **May 26, 2025:** Confirmation from all previously affected Southern Hemisphere customers that the issue is resolved. * **May 26, 2025:** No sign of domain related issues in our network monitoring system. * **May 26, 2025:** Confirmation from domain host that domain was suspended due to the updated technical contact email address not being verified. ## **Root Cause Analysis** The incident originated from an ‘incomplete’ domain registration update process: 1. Technical contact information was successfully updated in the domain host administration panel during renewal. 2. Automated email received in a generic email inbox asking for updated technical contact details to be confirmed. 3. Timeframe for validation of details expired without additional warning or contact from the domain provider. Domain suspended without notification. 4. Due to lack of confirmation, ICANN registration records were not updated, leading to a domain suspension. ## **Resolution** 1. Located and verified the legitimacy of the confirmation email from the domain host. 2. Completed the technical contact confirmation process. 3. ICANN registration records subsequently updated. 4. Initiated DNS propagation across global DNS infrastructure & monitored progress. ## **Customer Impact** Due to the distributed nature of DNS propagation, impact varies by customer location and ISP DNS configuration. Some customers may experience delayed resolution or temporary new occurrences of the issue as DNS updates propagate globally. It is generally accepted that global propagation can take up to 48hrs. ## **Immediate Actions Taken** * Troubleshooting of root cause. * Incident created at [https://streamtime.statuspage.io/incidents/hk5ysgq63z9y](https://streamtime.statuspage.io/incidents/hk5ysgq63z9y) with incident subscription option to receive updates automatically. * Consistent, proactive communication to customers on progress. * Confirmed technical contact details with domain registrar. * Confirmed the ICANN registration records would be automatically updated. * Established monitoring for continued customer reports. * Provided customer workaround guidance \(alternative network connections\). * Contacted all 3rd party stakeholders and continued to push for information to confirm root cause. ## **Process Improvements** 1. **Domain Management Consolidation:** We have initiated the process to transfer domain management from our existing provider to GoDaddy for 24/7 domain hosting support availability. 2. **Notification Enhancement:** Update notification systems to target both individual team members and shared support inboxes. ## **Next Steps** * Continue to monitor support channels for customer responses as to their connectivity. * Continue to monitor relevant errors received by customers via our network monitoring system. * A full review of all critical accounts to be scheduled and completed internally. Confirming correct notifications are enabled and are sent to monitored email accounts. * Review and ensure existing internal disaster recovery plan is updated based on any account changes.

This incident has been resolved.

Our engineers identified and resolved the issue. Please contact Streamtime Support if you continue to experience issues. Our team will continue to investiage the outage to ensure we avoid any recurrence if at all possible.

AWS have resolved their operational incident - https://health.aws.amazon.com/health/status. We do not expect to see any further access errors.

This incident has been resolved.

This incident has been resolved.

We have resolved the issues affecting Streamtime and all services have been restored

Our monitoring has not revealed any further problems. The issue was identified to be related to our Apache server and has now been rectified. To ensure the issue does not reoccur we have increased certain limits on our server and will monitor over the coming days. This issue is now marked as resolved.

Monitoring and tests have completed successfully. This issue is now resolved. Again, apologies for the disruption caused this morning. Our engineering team will conduct a review into the cause of the issue and make any necessary changes to avoid similar occurrences in future. Any questions/issues please get in touch via our in-app/website messaging function.

The scheduled maintenance has been completed.

The scheduled maintenance has been completed.

The scheduled maintenance has been completed.

The scheduled maintenance has been completed.

The scheduled maintenance has been completed.

The scheduled maintenance has been completed.