Ticketleap outage history

The Epic Experiences Store is experiencing critical network connectivity issues affecting availability of service. Our team is investigating.

We are continuing to investigate this issue.

The issue is related to an outage on cloud infrastructure provider. Our team is following the situation closely and preparing potential additional mitigation actions.

The store is now online and services are progressively restoring. Our team will continue to monitor the situation.

This incident has been resolved.

Activations experienced a brief outage from 9:09 AM EDT to 9:36 AM EDT. This was caused by a network configuration mismatch during a security patch. Recovery was delayed due to a secondary technical block in our deployment pipeline. All systems are now stable

The Mobile CMS experienced brief outages at approximately 10:21 AM EST and 10:29 AM EST, each lasting a few seconds. The issue was due to a performance bottleneck in our user-state data model for ticket-based audiences. During these windows, CMS users may have encountered slowness or temporary downtime. We have identified and planned improvements to help prevent future occurrences.

The SSL certificate for several activation domains was removed from CloudFlare without notice and before its expiration date. We are working with CloudFlare support to determine the reason why the certificate was removed. This issue has been resolved.

An outage with our cloud delivery network (CDN) provider caused clients and customers to not be able to visit our pages. Issue was from 2:56AM EST to 3:12AM EST.

An underlying cloud service is experiencing a partial outage. Users may experience intermittent errors. Engineering is monitoring.

This incident has been resolved.

ShowClix had an 18 minute outage this morning. Between 11:24 and 11:42 AM EST engineering noted elevated error rates while processing credit card payments due to an underlying network issue.

An outage with our cloud provider is causing degraded performance on the system including but not limited to sending emails and loading files for some pages.

Application errors have largely subsided. Engineering will continue to monitor as cloud services catch up.

As of 6:53 PM EDT last night we reported full recovery.

At 8:20 PM EDT replication stopped on a read-only replica database which caused 404 responses when accessing data created or updated after that time. Engineering attempted to restart the database at 10:05 PM. However the process did not restart on its own, leading to connection failures to the replica database for a period of around 6 minutes. At 10:11 PM DEVOPS pulled the affected replica out of rotation which resolved all issues. Practically the effects were that customers and clients experienced “not found” errors when viewing some data (viewing orders, event configurations, etc.) between 8:20PM and 10:05 PM and widespread 500 errors (which render as the broken ticket page) when accessing the application (purchasing tickets, logging in, accessing the admin, etc). between 10:05 and 10:11 PM. This affected all ShowClix, Ticketbooth and white label clients.