Swapcard 장애 기록

Sherlock is currently experiencing degraded performance due to an inference capacity issue. We are working with our inference providers to restore full capacity as quickly as possible.

A fix to increase capacity is currently under testing.

Inference capacity has been fully restored

The issue has been identified and a fix is being implemented.

This incident has been resolved.

We are providing a detailed post-mortem report regarding the service disruption that affected Swapcard Login on June 19th, 2026. This issue was caused by an upgrade of our cache cluster, which led to temporary login failures. The goal of this post-mortem is to share insights from our assessment and the steps taken to resolve the issue while providing transparency to our customers. **Summary** As part of a planned upgrade to increase the capacity of our cache cluster, we carried out a zero-maintenance-window upgrade, a rollout designed to require no scheduled downtime. During the rollout, a misconfiguration that was not identified in our pre-production environment affected the login service in production, resulting in a period during which users were unable to sign in. **Timeline (UTC)** [16:35] — Migration of production services to the upgraded cache cluster proceeds as part of the zero-downtime rollout. [16:38] — The login service begins failing to connect to the cache; sign-in errors start. [16:42] — Incident detected and investigation begins. [16:45] — Root cause identified: corrected configuration deployed to login. [16:48] — Login fully recovers; error rates return to normal. **Mitigation** We updated the login service to connect to the upgraded cache cluster using the correct configuration, restoring sign-in functionality. As a precaution, we reviewed all other production services and corrected one additional service that carried the same latent misconfiguration before it could cause further impact. **Root cause** During a planned upgrade to increase the capacity of our cache cluster, a misconfiguration was introduced that left the Login service unable to connect to the upgraded cluster. The issue was not identified in our pre-production environment and surfaced in production during the rollout, causing sign-in to fail until the configuration was corrected. **Next steps** We are adding automated checks to (1) detect any reference to a cache cluster scheduled for decommission and (2) verify configuration parity between our pre-production and production environments, so this class of issue is caught before reaching production in future upgrades.

### **Summary** Approximately two weeks ago, users began experiencing disruption when accessing analytics data within App Studio. Initial investigation did not immediately identify the root cause; a deeper analysis ultimately traced the issue to a routine infrastructure consolidation that was capping database capacity below the threshold required to handle specific event analytics queries spanning long date ranges. Full service has since been restored following a targeted increase in database resources. ### **Timeline** * **2026-03-11 19:32 UTC** — Incident created. Initial investigation launched following a rise in user-reported loading issues on analytics, despite no prior performance degradation. The initial investigation did not surface a clear root cause at this stage. * **2026-03-18 10:45 UTC** — Automated monitoring triggered alerts for elevated database error rates, corroborated by incoming user reports of analytics failures. * **2026-03-19 09:30 UTC** — Deeper analysis by the Engineering team identified the root cause: high latency and query failures isolated to specific event analytics queries over long date ranges, linked to the earlier infrastructure consolidation. * **2026-03-19 09:40 UTC** — Remediation plan executed: database resources scaled up to meet query load demands. * **2026-03-19 10:05 UTC** — Database performance stabilized; analytics functionality verified as fully restored. ### **Resolution** To restore service, database resources were immediately scaled up to accommodate the throughput required by analytics queries. Once additional capacity was provisioned, query timeouts ceased and application stability was reestablished. The system remained under active monitoring until no further degradation was observed, at which point the incident was marked resolved. ### **Root Cause Analysis** The incident was caused by a capacity mismatch introduced during a routine infrastructure consolidation. Such consolidations are a standard part of our infrastructure lifecycle and are conducted with active monitoring in place to ensure continued performance across all services. In this instance, however, the capacity allocated to the Analytics Database was capped at a level that, while adequate for standard workloads, could not sustain the computational demands of specific event analytics queries executed over long date ranges. The initial investigation did not immediately surface this connection; it was only through deeper analysis that the consolidation was identified as the underlying cause. When these queries were executed, they exceeded available processing capacity, resulting in database timeouts and repeated failures of the analytics service; conditions that persisted until capacity was restored. The investigation also surfaced a broader pattern: a gradual uptick in user-reported loading issues over the preceding two weeks, despite analytics having previously performed reliably. This indicated the capacity cap had been quietly eroding performance well ahead of the full outage. ### **Conclusion** We are straightening our processes to prevent recurrence. Monitoring coverage applied during database consolidations will be strengthened to detect query latency degradation earlier, and capacity planning will be refined to better account for the demands of specific event analytics queries over long date ranges. Looking further ahead, the team is actively working on delivering a fully revamped analytics pipeline to production later this year. This is not an incremental improvement; the new pipeline is being built to deliver deeper, richer metrics that give event organizers a comprehensive view of their event performance.

This incident has been resolved.

This incident has been resolved.

## Summary On March 11, 2026, we experienced a service disruption affecting the visibility of speaker and attendee profiles across multiple events. The incident was triggered by a timing issue during a scheduled infrastructure update — a data refresh was initiated before new field definitions had fully propagated to our Search Engine, causing the indexation system to be built from an outdated configuration. Our Search Engine relies on an indexation system to serve profile data quickly across events. When the refresh completed against the stale configuration, new fields were rejected due to strict schema enforcement, leaving profiles absent from search results and listings. The goal of this post-mortem is to share our assessment and the steps taken to resolve the issue, while providing full transparency to our customers. ## Timeline | Time \(UTC\) | Event | | --- | --- | | Mar 11 – 11:04 | Infrastructure update deployed to production. Changes confirmed as applied. | | Mar 11 – 13:21 | Data refresh initiated as part of the planned release process. Previous indexation remained fully functional during this period. | | Mar 11 – 13:42 | Release step complete. Data refresh running, expected duration ~3–4 hours. | | Mar 11 – 14:49 | Configuration mismatch errors first logged internally. | | Mar 11 – 15:33 | Data refresh reported as complete. Spot checks passed. Final release step initiated. | | Mar 11 – 16:52 | Reports of missing profiles across multiple events surfaced. Incident response team immediately activated. | | Mar 11 – 17:29 | Root cause identified in under few minutes: indexation system had been built before the new fields were fully available. Internal alerts corroborated the finding. | | Mar 11 – 18:08 | Analysis completed confirming the refresh did not produce an indexation with the expected new fields. | | Mar 11 – 18:39 | Corrective data refresh triggered with the correct configuration fully in place, within 30 minutes of initial reports. | | Mar 11 – 18:43 | New indexation confirmed to contain the correct fields. Full refresh in progress. | ## Resolution Once the root cause was identified, the team immediately initiated a new data refresh against the correct, fully propagated configuration. Profile visibility and search behavior were monitored throughout the process. Once the refresh completed, all profiles returned to normal search and listing behavior. ## Root Cause Analysis The root cause was a race condition between the deployment of new Search Engine field definitions and the triggering of a data refresh. The refresh started before the new configuration had fully propagated, causing the indexation system to be built from a stale schema. Due to strict schema enforcement, subsequent write operations referencing the new fields were rejected, leaving profiles absent from search results and listings. Several factors contributed to this incident: * **Ambiguous deployment signal:** A pipeline warning created uncertainty around whether the deployment had completed cleanly. * **Insufficient spot checks:** Post-refresh verification passed at a surface level, masking the underlying schema mismatch until live traffic exposed it. **Short-term improvements:** * Add a mandatory pre-refresh validation step to confirm the Search Engine schema matches the expected configuration before any refresh is initiated. * Update the release process to include explicit schema validation checkpoints for sub field, as a required step. ## Conclusion This incident stemmed from a timing edge case in our release process. The improvements listed above reflect our commitment to preventing similar incidents and maintaining a reliable platform for all our customers. For any questions or concerns regarding this incident, please reach out to our support team.

The internal DNS servers has receive unusual amount of traffic and didn’t succeed to handle the load. The replication in place didn’t succeed to balance the load due to miss-configuration and under sized instances. This cascade failure of our internal dns nodes leaded to a partial outage of the internal communicate between our internal APIs inside our private network. The root cause of the failure was hard to identify as the outage was not global and part of the DNS resolution was still treated properly due to some DNS cache configuration.

One of database cluster have receive an unusual amount of request (root cause of that increase of load is under investigation) that lead to partial outage of the database, the replication in place didn’t succeed to handle the load due to the volume of query, that lead to a cascade failure of our database nodes. This cascade failure of our Database nodes lead to a partial outage of the authentication and chat components.

We have faced an increase of response time on our main caching server due to unexpected latency on the replication mechanism. Once the issue was identified, we worked on deploying new nodes on the caching cluster and switched some of our services. This led to a decrease in the response time. We’re currently working on a mitigation plan which will be conducted within the next few hours without any service interruption. This aim of this plan is to prevent the latency issue from happening again. We apologize for any inconvenience.

A fix has been implemented and we are monitoring the results.

We are continuing to work on a fix for this issue.

The issue has been identified, we working on the remediations.

We are currently investigating this issue.

We have identified a configuration issues that has lead to 403 error on the endpoint. We apologise for the inconvenience

**What Happened** On Friday 23 October at 15:58 UTC, we suffered a ~10min latency increase due to a massive load on one of our main API endpoint for meetings between attendees. We received an automatic alert that the service started to be degraded at 15:55 UTC. **Resolution** We immediately turned off the endpoint to mitigate the impact on our services. At the same time we have assigned a specific engineering team to analyse the cause and to work on its resolution. We identified the root cause ~16:45 UTC and started to work on a long term mitigation plan. The team worked on optimising the specific endpoint to prevent any further issue. We ran tests to prevent any regression and to ensure the quality of the fix. The solution has been deployed and the endpoint has been re-activated. **Impact** A degraded response time and an increase of page loading on our event app \([app.swapcard.com](http://app.swapcard.com)\)

The issue has been identified and a fix is being implemented.

We are currently investigating this issue.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

We encounter an issue on the studio due to an improper version rolling update, we apologise for the disruption.

We have identified a latency issues that has lead to 503 error on the endpoint. The root cause has been found and the problem has been solved. We apologise for the inconvenience

The issue has been identified and a fix is being implemented.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

**What Happened** On Tuesday 9 March at 17:28 UTC, we experienced an increase in error rates for between 5 and 10min due to a massive load on one of our databases in eu-west datacenter. We received an automatic alert that the service started facing issues at 17:28 UTC. **Resolution** We immediately created an incident room with a support team and assigned a dedicated engineering team to analyse the cause and work on its resolution. We identified the root cause ~17:32 UTC and started to work on a mitigation plan. The team worked on optimising the affected database’s scaling capabilities to prevent any further issues. We ran tests to prevent regression and to ensure the quality of the resolution. The solution has been deployed and the service is running normally. **Impact** A degraded experience on our event app, showing error messages when accessing event content on our Event App. No other features were impacted and everything else remained functional during the incident.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

**What Happened** On Thursday 8 April at 10:00 UTC, we experienced an increase in error rates during **3 min** due to a miss configuration in one of our web application in eu-west datacenter. We received an automatic alert that the service started facing issues at 09:59 UTC. **Resolution** We immediately created an incident room with a support team and assigned a dedicated engineering team to analyse the cause and work on its resolution. We identified the root cause ~10:01 UTC and started to work on a mitigation plan. The team has worked on improving the configuration of that services to prevent any further issues. **Impact** A degraded experience on our event app, showing error messages when accessing Event App. No other features were impacted and everything else remained functional during the incident.

The issue has been identified and a fix is being implemented.

We are currently investigating this issue.

The issue has been identified and a fix is being implemented.

This incident has been resolved.

**What Happened** On Monday 12 April at 16:18 UTC, we suffered a ~15min outage due to an unresponsive api request on our main API endpoint for notifications. We received an automatic alert that the service started to be degraded at 16:15 UTC. **Resolution** We immediately turned off the api request responsible to mitigate the impact on our services and recover the situation, the turn off process took us around ~6-7 min to ensure proper execution and deployment. At the same time we have assigned a specific engineering team to analyse the cause and to work on its resolution. We identified the root cause ~16:20 UTC and started to work on a long term mitigation plan. The team worked on solving the specific query to prevent any further issue. We ran tests to prevent any regression and to ensure the quality of the fix. The solution has been deployed and the api request has been re-activated. We still investigate to under how this happen and improve our QA process to prevent this happening in the further. **Impact** An impossibility to access to on our event app \([app.swapcard.com](http://app.swapcard.com)\) and studio \([studio.swapcard.com](http://studio.swapcard.com)\) during ~15min.

A fix has been implemented and we are monitoring the results.

**What Happened** On May 5th at 12:00 UTC, we suffered to an duplicate notifications issues on our push notification feature on the studio. We received an automatic alert that some notification are being duplicate 12:12 UTC. **Resolution** We immediately turned off the organiser notifications feature and start disabling responsible notification to mitigate the impact on the end user and recover the situation, the turn off process took us some time to ensure proper execution and full cover of the issue. At the same time we have assigned a specific engineering team to analyse the cause and to work on its resolution. We identified the root cause ~13:00 UTC and started to work on a long term mitigation plan. The team worked on solving this specific feature to prevent any further issue. We ran tests to prevent any regression and to ensure the quality of the fix. The solution has been deployed and the feature has been re-activated. We understand how this happen and improve our QA process to prevent this happening in the further in addition of adding protection to prevent duplicate to happen on this notification type. **Impact** Duplicate notification sent by the organiser and duplicate show in the activities feed.

The 5 May on 12 UTC and during ~2h, we face a notification duplicate issues on organiser notifications, the problem has been identified and fixed. Duplicate notifications has been deleted and the activities feed is back to original state. Protection has been added to prevent this happening again in the future and quality process has been re-enforce to ensure further issues. We apologise for the trouble.

**What Happened** On Tuesday 8 June at 09:58 UTC, we experienced an increase in error rates for ~1h in the Exhibitor center and with customers using the vanity url \(\*.app.swapcard.com & [app.swapcard.com](http://app.swapcard.com) remained functional during the entire incident\) due to the major failure of our CDN provider Fastly \([https://status.fastly.com/incidents/vpk0ssybt3bj](https://status.fastly.com/incidents/vpk0ssybt3bj)\), which is used to provide vanity urls and worldwide access across vanity urls. The major issue was a global outage from Fastly provider, which also impacted most of the biggest tech companies and media websites including: Reddit, CNN, Paypal, Twitter, Amazon, Twitch, and so on. We received an automatic alert that the service started facing issues ~09:59 UTC. **Resolution** We immediately created an incident room with a dedicated engineering team and support team to mitigate the impact on the domain we can control \(meaning [team.swapcard.com](http://team.swapcard.com)\). In Swapcard we have the Multi CDN architecture for Swapcard domains. After ensuring our backup CDN was ready and configuration was okay, we immediately triggered the switch to use our emergency CDN \(Cloudfront\) for [team.swapcard.com](http://team.swapcard.com). In addition, we had provided transparent communications about the incident on our Twitter account and status page \([https://www.swapcardstatus.com/](https://www.swapcardstatus.com/)\). Following this incident, and in favour of providing the best service to our customers, we will evaluate a Multi CDN architecture for the vanity urls. Although this extremely rare issue was not under our control, it is important to ensure that in the future we can address this issue faster and hot switch from one CDN to another for vanity url. **Impact** A degraded experience in the exhibitor center \([team.swapcard.com](http://team.swapcard.com)\). For customers using vanity url, error messages were shown when accessing the event app \(vanity url only\) or exhibitor center. \*.app.swapcard.com & [app.swapcard.com](http://app.swapcard.com) remained functional during the entire incident.

Our current CDN (Fastly) is impacted by a world wide issue https://status.fastly.com/ that impact many websites around the world. EDIT : Fastly has implemented a fix. https://status.fastly.com/. team.swapcard.com is now working fine.

We are currently investigating this issue.

We are currently investigating this issue.

A fix has been implemented and we are monitoring the results.

Swapcard has observed recovery of all services and has resolved this incident. We publish a post-mortem in the following hours. We apologise for any inconvenience and we thank you for your understanding.

Please see our post-mortem below regarding a service delivery disruption that affected Swapcard customers from June 22th, 2021 at 08:30 UTC through to 08:49 UTC. It is our goal in this post-mortem to provide details on our initial assessment of the incident communicated on the Swapcard status page and to describe the remediation actions that we have taken to restore service. ## Incident summary On Tuesday, June 22 at 08:30 UTC, we experienced an outage due to a network access control list \(ACL\) change on our private cloud network \(VPC\). On Tuesday, June 22 at 08:29 UTC, the infrastructure team deployed an ACL adjustment on our private cloud network \(VPC\) which was planned for non-production environments. The same change was applied to our production account by our infrastructure as code tool overriding the initial plan \(with only non-production environments scoped\). This caused a significant disruption of traffic and an outage of Swapcard services. This change was applied in accordance with Swapcard standard infrastructure & security change and enhancement practices. A misconfiguration in our network module has caused this unplanned change in the production environment. This network rules configuration change caused an outage of all incoming traffic from our clients generating HTTP 5xx errors. Swapcard monitoring detected the traffic disruption at 08:30 UTC and activated the Swapcard Incident Response team. Swapcard’s team worked to triage and restore services to alleviate customer impact. In parallel, the cause of the issue was investigated and mitigations were put in place. ## Mitigation deployment At 08:35 UTC, Swapcard’s Engineering team identified the specific network rules causing the traffic disruptions, and by 08:40 UTC had reverted the network rule. Traffic disruption stopped as the update propagated through the network. Swapcard engineering then monitored all the services to ensure full and proper recovery by 08:42 UTC. As a result of restoring the network, customers would then see a reduction in the 5XX errors and error messages. Approximately ⅔ of customers would have experienced errors after the network recovered as that is the time it takes for the traffic to be absorbed by our services. At 08:50 UTC, Swapcard confirmed that the update was completed and capacity restored to pre-incident levels, ensuring that the traffic was back to the pre-incident rate. ## Event Outline ### Duration Summary Time alerted to the outage: 01 minutes Time to identify the source of disruption: 5 minutes Time to initiate recovery: 5 minutes Time to monitor and restore capacities pre-crash: 8 minutes ### Events of 2021 June 22 \(UTC\) \(08:30 UTC\) | Initial onset of global network disruption \(08:30 UTC\) | Global network disruption identified by Swapcard monitoring \(08:35 UTC\) | Swapcard Engineering identified the service configuration \(08:40 UTC\) | Swapcard status post is activated \(08:44 UTC\) | Impacted services began to recover \(08:47 UTC\) | Majority of services recovered, additional mitigation measures taken \(08:50 UTC\) | Incident Mitigated, pre-incident capacity restored \(08:59 UTC\) | Status post resolved Affected customers may have been impacted by varying degrees and with a shorter duration than as described above. ## Forward Planning Swapcard has deployed a permanent fix for this incident and implemented technical measures to ensure that planned scope is always respected by the infrastructure change toolchain. In accordance with our high standard in terms of deliverability, Swapcard will conduct an internal audit on the change process and policy to ensure that infrastructure modules are even more controlled and hardened. Procedure and control are already in place but today’s incident highlights the need for improvement. **We consider the likelihood of a recurrence of this issue to be extremely low and will further reduce the risk as we make future interventions and improvements to our infrastructure and procedures.**

The issue has been identified and a fix is being implemented.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

Please see our post-mortem below regarding a service delivery disruption that affected Swapcard customers from July 6th, 2021 at 05:58 UTC through to 6:18 UTC. It is our goal in this post-mortem to provide details on our initial assessment of the incident communicated on the Swapcard status page and to describe the remediation actions that we have taken to restore service. ## **Incident summary** On Tuesday, July 6th at 06:01 UTC, we experienced an increase error rate due to a rolling update \(capacity and security\) apply on our caching sever \(Redis\). On Tuesday, July 6th at 06:01 UTC, our private cloud provider has deployed an automatic update on our main caching server \(Redis\). This automatic change is applied in accordance with Swapcard standard infrastructure & security change and enhancement practices. Cluster are configured in High Availability/Failover mode to prevent any disruption of services and support our zero downtime strategy. An issue in our hot switch mechanism has prevent our services to take in consideration the primary/secondary switch has cause a significant disruption on the content delivery. Swapcard monitoring detected the disruption at 05:58 UTC and activated the Swapcard Incident Response team. Swapcard’s team worked to triage and restore services to alleviate customer impact. In parallel, the cause of the issue was investigated and mitigations were put in place. ## **Mitigation deployment** At 05:58 UTC, Swapcard’s Engineering team identified the cause, and by 06:08 UTC had deploy manually a configuration change to reflect the primary/secondary changes. Content delivery disruption stopped as the update propagated through our services. Swapcard engineering then monitored all the services to ensure full and proper recovery by 6:17 UTC. As a result of manually deploying this configuration change, customers would then see a reduction in the 5XX errors and error messages. At 06:23 UTC, Swapcard confirmed that the update was completed and capacity restored to pre-incident levels, ensuring that the traffic was back to the pre-incident rate. ## **Event Outline** ### **Duration Summary** Time alerted to the outage: 01 minutes Time to identify the source of disruption: 3 minutes Time to initiate recovery: 5 minutes Time to monitor and restore capacities pre-crash: 8 minutes ### **Events of 2021 June 22 \(UTC\)** \(05:58 UTC\) | Initial onset of content delivery disruption \(05:58 UTC\) | Disruption identified by Swapcard monitoring \(06:03 UTC\) | Swapcard status post is activated \(06:05 UTC\) | Swapcard Engineering identified the issue \(06:08 UTC\) | Swapcard Engineering initiate the manual configuration change \(06:13 UTC\) | Impacted services began to recover \(06:16 UTC\) | Majority of services recovered, additional mitigation measures taken \(06:17 UTC\) | Incident Mitigated, pre-incident capacity restored \(06:23 UTC\) | Status post resolved Affected customers may have been impacted by varying degrees and with a shorter duration than as described above. ## **Forward Planning** Swapcard has deployed a permanent fix for this incident and implemented technical measures to ensure proper automatic maintenance windows on this endpoint and improve the primary/secondary hot switch configuration. In accordance with our high standard in terms of deliverability, Swapcard will conduct an internal audit on the hot switch capacity and configuration. Control are already in place but today’s incident highlights the need for improvement on this part. **We consider the likelihood of a recurrence of this issue to be extremely low and will further reduce the risk as we make future interventions and improvements to our infrastructure and procedures.**

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented.

We are currently investigating this issue.

Please see our post-mortem below regarding the increase response time on our chat endpoint from August 10th, 2021 at 16:48 UTC through to 17:15 UTC. It is our goal in this post-mortem to provide details on our initial assessment of the incident communicated on the Swapcard status page and to describe the remediation actions that we have taken to restore service. ## **Incident summary** On Tuesday, August 10th at 16:48 UTC, we experienced an increase response time on our chat endpoint due to the pressure of two database queries \(Count unread channel & Number of active one to one conversation\) on our chat database. On Tuesday, August 10th at 16:48 UTC, our infrastructure team has been automatically alerted of an increase response time on our chat endpoint, requests was replying in an average of 300ms compare to the usual average of 4ms response time. Swapcard monitoring detected the start of disruption at ~16:47 UTC and activated the Swapcard Incident Response team. Swapcard’s team worked to triage and mitigate the responsible database queries to alleviate customer impact. In parallel, the cause of the issue was investigated and short & mid & long term plan were put in place. ## **Mitigation deployment** At 16:52 UTC, Swapcard’s Engineering team identified the cause, and by 16:57 UTC had immediately turned off the responsible database queries to mitigate the impact on our services and recover a proper chat endpoint response time, the turn off process took us around ~8 min to ensure proper execution and deployment. Chat increase response time stopped as the update propagated through our chat endpoints. Swapcard engineering then monitored all the chat endpoints to ensure full and proper recovery. As a result of the deployment of that change, customers would then see a reduction of the chat loading time and error messages. At 17:30 UTC, Swapcard confirmed that the update was completed and response time restored to pre-incident levels and ensuring that the traffic was back to the pre-incident rate. ## **Event Outline** ### **Duration Summary** Time alerted to the outage: 01 minutes Time to identify the source of disruption: ~5 minutes Time to initiate recovery: ~5 minutes Time to monitor and restore response time pre-crash: ~12 minutes ### **Events of 2021 August 10 \(UTC\)** \(16:46 UTC\) | Initial onset of the response time increase \(16:47 UTC\) | Disruption identified by Swapcard monitoring \(16:48 UTC\) | Swapcard status post is activated \(16:52 UTC\) | Swapcard Engineering identified the issue \(17:05 UTC\) | Swapcard Engineering mitigate the issue \(17:13 UTC\) | Impacted chat endpoint began to recover \(17:15 UTC\) | Incident Mitigated, pre-incident response time restored \(17:38 UTC\) | Status post resolved Affected customers may have been impacted by varying degrees and with a shorter duration than as described above. ## **Forward Planning** Swapcard has deployed a permanent fix for this incident by optimising the responsible database queries and implemented technical measures to ensure proper covering of other databases queries on the chat endpoint. In accordance with our high standard in terms of deliverability, Swapcard conducted during the 5 following days post incident a series of optimisation to prevent further similar issues on the chat endpoint. **We consider the likelihood of a recurrence of this issue to be extremely low and will further reduce the risk as we make improvements.**

The issue has been identified and a fix is being implemented.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

Please see our post-mortem below regarding the increase response time on our chat endpoint from September 2th, 2021 at 15:38 UTC through to 15:41 UTC. It is our goal in this post-mortem to provide details on our initial assessment of the incident communicated on the Swapcard status page and to describe the remediation actions that we have taken to restore service. ## **Incident summary** On Thursday, September 2th at 15:38 UTC, we experienced an increase response time on our event app due to one of our replicate database server being under pressure because of a specific query. On Thursday, September 2th at 15:37 UTC, our infrastructure team has been automatically alerted of an increase response time on our event app, requests were replying in a twice higher average time compare to the usual average response time. Swapcard monitoring detected the start of disruption at ~15:37 UTC and activated the Swapcard Incident Response team. Swapcard’s team worked to triage and mitigate the responsible database queries to alleviate customer impact. In parallel, the cause of the issue was investigated and short & mid & long term plans were put in place. ## **Mitigation deployment** At 15:39 UTC our monitoring system had immediately turned off the responsible database that was under pressure to mitigate the impact on our services and recover a proper event app response time. The turn off process took around ~15s. Event app increase response time stopped as the database turned off configuration propagated through our endpoints. Swapcard engineering team then monitored the event app endpoints to ensure full and proper recovery. As a result of the deployment of that change, customers would then see a reduction of the chat loading time and error messages. At 15:43 UTC, Swapcard confirmed that the update was completed and response time restored to pre-incident levels and ensuring that the traffic was back to the pre-incident rate. Swapcard’s Engineering team identified the cause, and by ~15:50 UTC has re-enabled the impacted database replication, the root cause has also been identified and solved. ## **Event Outline** ### **Duration Summary** Time alerted to the outage: 1 minutes Time to identify the source of disruption: ~2 minutes Time to initiate recovery: ~15 seconds Time to monitor and restore response time pre-crash: ~2 minutes ### **Events of 2021 September 2th \(UTC\)** \(15:37 UTC\) | Initial onset of the response time increase \(15:38 UTC\) | Disruption identified by Swapcard monitoring \(15:38 UTC\) | Swapcard status post activated \(15:39 UTC\) | Automatic monitoring system mitigated the issue \(15:40 UTC\) | Impacted event app began to recover \(15:41 UTC\) | Incident mitigated, pre-incident response time restored \(15:43 UTC\) | Status post resolved \(15:50 UTC\) | Swapcard Engineering identified the issue Affected customers may have been impacted by varying degrees and with a shorter duration than described above. ## **Forward Planning** Swapcard has deployed a permanent fix for this incident in accordance with our high standard in terms of deliverability.