Swapcard Status

We are providing a detailed post-mortem report regarding the service disruption that affected Swapcard Login on June 19th, 2026. This issue was caused by an upgrade of our cache cluster, which led to temporary login failures. The goal of this post-mortem is to share insights from our assessment and the steps taken to resolve the issue while providing transparency to our customers. **Summary** As part of a planned upgrade to increase the capacity of our cache cluster, we carried out a zero-maintenance-window upgrade, a rollout designed to require no scheduled downtime. During the rollout, a misconfiguration that was not identified in our pre-production environment affected the login service in production, resulting in a period during which users were unable to sign in. **Timeline (UTC)** [16:35] — Migration of production services to the upgraded cache cluster proceeds as part of the zero-downtime rollout. [16:38] — The login service begins failing to connect to the cache; sign-in errors start. [16:42] — Incident detected and investigation begins. [16:45] — Root cause identified: corrected configuration deployed to login. [16:48] — Login fully recovers; error rates return to normal. **Mitigation** We updated the login service to connect to the upgraded cache cluster using the correct configuration, restoring sign-in functionality. As a precaution, we reviewed all other production services and corrected one additional service that carried the same latent misconfiguration before it could cause further impact. **Root cause** During a planned upgrade to increase the capacity of our cache cluster, a misconfiguration was introduced that left the Login service unable to connect to the upgraded cluster. The issue was not identified in our pre-production environment and surfaced in production during the rollout, causing sign-in to fail until the configuration was corrected. **Next steps** We are adding automated checks to (1) detect any reference to a cache cluster scheduled for decommission and (2) verify configuration parity between our pre-production and production environments, so this class of issue is caught before reaching production in future upgrades.

### **Summary** Approximately two weeks ago, users began experiencing disruption when accessing analytics data within App Studio. Initial investigation did not immediately identify the root cause; a deeper analysis ultimately traced the issue to a routine infrastructure consolidation that was capping database capacity below the threshold required to handle specific event analytics queries spanning long date ranges. Full service has since been restored following a targeted increase in database resources. ### **Timeline** * **2026-03-11 19:32 UTC** — Incident created. Initial investigation launched following a rise in user-reported loading issues on analytics, despite no prior performance degradation. The initial investigation did not surface a clear root cause at this stage. * **2026-03-18 10:45 UTC** — Automated monitoring triggered alerts for elevated database error rates, corroborated by incoming user reports of analytics failures. * **2026-03-19 09:30 UTC** — Deeper analysis by the Engineering team identified the root cause: high latency and query failures isolated to specific event analytics queries over long date ranges, linked to the earlier infrastructure consolidation. * **2026-03-19 09:40 UTC** — Remediation plan executed: database resources scaled up to meet query load demands. * **2026-03-19 10:05 UTC** — Database performance stabilized; analytics functionality verified as fully restored. ### **Resolution** To restore service, database resources were immediately scaled up to accommodate the throughput required by analytics queries. Once additional capacity was provisioned, query timeouts ceased and application stability was reestablished. The system remained under active monitoring until no further degradation was observed, at which point the incident was marked resolved. ### **Root Cause Analysis** The incident was caused by a capacity mismatch introduced during a routine infrastructure consolidation. Such consolidations are a standard part of our infrastructure lifecycle and are conducted with active monitoring in place to ensure continued performance across all services. In this instance, however, the capacity allocated to the Analytics Database was capped at a level that, while adequate for standard workloads, could not sustain the computational demands of specific event analytics queries executed over long date ranges. The initial investigation did not immediately surface this connection; it was only through deeper analysis that the consolidation was identified as the underlying cause. When these queries were executed, they exceeded available processing capacity, resulting in database timeouts and repeated failures of the analytics service; conditions that persisted until capacity was restored. The investigation also surfaced a broader pattern: a gradual uptick in user-reported loading issues over the preceding two weeks, despite analytics having previously performed reliably. This indicated the capacity cap had been quietly eroding performance well ahead of the full outage. ### **Conclusion** We are straightening our processes to prevent recurrence. Monitoring coverage applied during database consolidations will be strengthened to detect query latency degradation earlier, and capacity planning will be refined to better account for the demands of specific event analytics queries over long date ranges. Looking further ahead, the team is actively working on delivering a fully revamped analytics pipeline to production later this year. This is not an incremental improvement; the new pipeline is being built to deliver deeper, richer metrics that give event organizers a comprehensive view of their event performance.

## Summary On March 11, 2026, we experienced a service disruption affecting the visibility of speaker and attendee profiles across multiple events. The incident was triggered by a timing issue during a scheduled infrastructure update — a data refresh was initiated before new field definitions had fully propagated to our Search Engine, causing the indexation system to be built from an outdated configuration. Our Search Engine relies on an indexation system to serve profile data quickly across events. When the refresh completed against the stale configuration, new fields were rejected due to strict schema enforcement, leaving profiles absent from search results and listings. The goal of this post-mortem is to share our assessment and the steps taken to resolve the issue, while providing full transparency to our customers. ## Timeline | Time \(UTC\) | Event | | --- | --- | | Mar 11 – 11:04 | Infrastructure update deployed to production. Changes confirmed as applied. | | Mar 11 – 13:21 | Data refresh initiated as part of the planned release process. Previous indexation remained fully functional during this period. | | Mar 11 – 13:42 | Release step complete. Data refresh running, expected duration ~3–4 hours. | | Mar 11 – 14:49 | Configuration mismatch errors first logged internally. | | Mar 11 – 15:33 | Data refresh reported as complete. Spot checks passed. Final release step initiated. | | Mar 11 – 16:52 | Reports of missing profiles across multiple events surfaced. Incident response team immediately activated. | | Mar 11 – 17:29 | Root cause identified in under few minutes: indexation system had been built before the new fields were fully available. Internal alerts corroborated the finding. | | Mar 11 – 18:08 | Analysis completed confirming the refresh did not produce an indexation with the expected new fields. | | Mar 11 – 18:39 | Corrective data refresh triggered with the correct configuration fully in place, within 30 minutes of initial reports. | | Mar 11 – 18:43 | New indexation confirmed to contain the correct fields. Full refresh in progress. | ## Resolution Once the root cause was identified, the team immediately initiated a new data refresh against the correct, fully propagated configuration. Profile visibility and search behavior were monitored throughout the process. Once the refresh completed, all profiles returned to normal search and listing behavior. ## Root Cause Analysis The root cause was a race condition between the deployment of new Search Engine field definitions and the triggering of a data refresh. The refresh started before the new configuration had fully propagated, causing the indexation system to be built from a stale schema. Due to strict schema enforcement, subsequent write operations referencing the new fields were rejected, leaving profiles absent from search results and listings. Several factors contributed to this incident: * **Ambiguous deployment signal:** A pipeline warning created uncertainty around whether the deployment had completed cleanly. * **Insufficient spot checks:** Post-refresh verification passed at a surface level, masking the underlying schema mismatch until live traffic exposed it. **Short-term improvements:** * Add a mandatory pre-refresh validation step to confirm the Search Engine schema matches the expected configuration before any refresh is initiated. * Update the release process to include explicit schema validation checkpoints for sub field, as a required step. ## Conclusion This incident stemmed from a timing edge case in our release process. The improvements listed above reflect our commitment to preventing similar incidents and maintaining a reliable platform for all our customers. For any questions or concerns regarding this incident, please reach out to our support team.

The internal DNS servers has receive unusual amount of traffic and didn’t succeed to handle the load. The replication in place didn’t succeed to balance the load due to miss-configuration and under sized instances. This cascade failure of our internal dns nodes leaded to a partial outage of the internal communicate between our internal APIs inside our private network. The root cause of the failure was hard to identify as the outage was not global and part of the DNS resolution was still treated properly due to some DNS cache configuration.

One of database cluster have receive an unusual amount of request (root cause of that increase of load is under investigation) that lead to partial outage of the database, the replication in place didn’t succeed to handle the load due to the volume of query, that lead to a cascade failure of our database nodes. This cascade failure of our Database nodes lead to a partial outage of the authentication and chat components.

We encounter an issue on the studio due to an improper version rolling update, we apologise for the disruption.

We have identified a configuration issues that has lead to 403 error on the endpoint. We apologise for the inconvenience

We have faced an increase of response time on our main caching server due to unexpected latency on the replication mechanism. Once the issue was identified, we worked on deploying new nodes on the caching cluster and switched some of our services. This led to a decrease in the response time. We’re currently working on a mitigation plan which will be conducted within the next few hours without any service interruption. This aim of this plan is to prevent the latency issue from happening again. We apologize for any inconvenience.

A fix has been implemented and we are monitoring the results.

We have identified a latency issues that has lead to 503 error on the endpoint. The root cause has been found and the problem has been solved. We apologise for the inconvenience