Ticketleap outage history

In response to a recently disclosed critical security vulnerability affecting systems worldwide, we have proactively implemented additional security protections across our platforms. These measures may result in intermittent access issues or unexpected errors for some users. Our teams are actively monitoring and adjusting these protections to minimize disruption while maintaining platform security. We appreciate your patience as we work to resolve these.

This incident has been resolved.

Activations platform (APIs, Public Web Activations, Admin CMS) suffered from degraded availability between 8:08 - 8:52 AM EDT related to an internal DNS issue with our services. The availability issue was resolved, and investigation for complete root cause and additional mitigation is under way.

Activations platform (APIs, Public Web Activations, Admin CMS) suffered from degraded availability between 8:08 - 8:52 AM EDT related to an internal DNS issue with our services. The availability issue was resolved, and investigation for complete root cause and additional mitigation is under way.

The issue has been identified and a fix is being implemented.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

On September 16, Leap Ticketing experienced a service disruption for 11 minutes caused by a production configuration change related to our search infrastructure. The change resulted in an unexpected increase in database connections, which exhausted the available connection capacity and temporarily impacted application availability. Once the issue was identified, our engineering team disabled the affected configuration and restored normal service. We are reviewing the incident and will implement appropriate safeguards around connection limits, capacity monitoring, and production changes to prevent a similar issue from occurring in the future. We apologize for the disruption and appreciate your patience while our team worked to restore service.

The Epic Experiences Store is experiencing critical network connectivity issues affecting availability of service. Our team is investigating.

We are continuing to investigate this issue.

The issue is related to an outage on cloud infrastructure provider. Our team is following the situation closely and preparing potential additional mitigation actions.

The store is now online and services are progressively restoring. Our team will continue to monitor the situation.

This incident has been resolved.

Activations experienced a brief outage from 9:09 AM EDT to 9:36 AM EDT. This was caused by a network configuration mismatch during a security patch. Recovery was delayed due to a secondary technical block in our deployment pipeline. All systems are now stable

The Mobile CMS experienced brief outages at approximately 10:21 AM EST and 10:29 AM EST, each lasting a few seconds. The issue was due to a performance bottleneck in our user-state data model for ticket-based audiences. During these windows, CMS users may have encountered slowness or temporary downtime. We have identified and planned improvements to help prevent future occurrences.

The SSL certificate for several activation domains was removed from CloudFlare without notice and before its expiration date. We are working with CloudFlare support to determine the reason why the certificate was removed. This issue has been resolved.

An outage with our cloud delivery network (CDN) provider caused clients and customers to not be able to visit our pages. Issue was from 2:56AM EST to 3:12AM EST.

An underlying cloud service is experiencing a partial outage. Users may experience intermittent errors. Engineering is monitoring.

This incident has been resolved.

ShowClix had an 18 minute outage this morning. Between 11:24 and 11:42 AM EST engineering noted elevated error rates while processing credit card payments due to an underlying network issue.

An outage with our cloud provider is causing degraded performance on the system including but not limited to sending emails and loading files for some pages.

Application errors have largely subsided. Engineering will continue to monitor as cloud services catch up.

As of 6:53 PM EDT last night we reported full recovery.

At 8:20 PM EDT replication stopped on a read-only replica database which caused 404 responses when accessing data created or updated after that time. Engineering attempted to restart the database at 10:05 PM. However the process did not restart on its own, leading to connection failures to the replica database for a period of around 6 minutes. At 10:11 PM DEVOPS pulled the affected replica out of rotation which resolved all issues. Practically the effects were that customers and clients experienced “not found” errors when viewing some data (viewing orders, event configurations, etc.) between 8:20PM and 10:05 PM and widespread 500 errors (which render as the broken ticket page) when accessing the application (purchasing tickets, logging in, accessing the admin, etc). between 10:05 and 10:11 PM. This affected all ShowClix, Ticketbooth and white label clients.