Swapcard 장애 기록

1. 2026년 10월 9일 오후 12:30 UTCmonitoring

   우리의 데이터베이스 복제 중 하나에 대한 데이터 복제 지연은 인터페이스의 결과 또는 의도적 정보를 볼 수있는 일부 사용자가 발생할 수 있습니다. 일부 사용자는 가끔 오류로 실행할 수 있습니다. 데이터가 손실되지 않았습니다. 우리의 팀은 활동을 촬영하고, 우리는 복부를 붙잡는 동안 밀접하게 감시하고 있습니다. 문의 사항이 있으시면 페이지를 새로 고침하십시오. 우리는 한 번 또 다른 업데이트가 완전히 해결됩니다.

2. 2026년 10월 9일 오후 3:04 UTCresolved

   사건은 해결됩니다. 복제는 sync에서 완전히 뒤이고, 플랫폼은 일반적으로 작동됩니다.

공식 인시던트 업데이트를 자동 번역했습니다.

1. 2026년 10월 7일 오후 1:09 UTCresolved

   03:00 과 10:25 UTC 사이, 개발자 API는 일부 요청에 대한 간헐적 오류를 반환 할 수 있습니다. 이 창에서 실패한 요청은 안전하게 회복될 수 있습니다. 이 문제는 해결되고 Developer API는 정상적으로 작동됩니다.

2. 2026년 10월 7일 오후 1:09 UTCinvestigating

   개발자 API가 일부 요청에 대한 간헐적 오류를 반환할 수 있는 문제점을 조사하고 있습니다. 대부분의 요청은 일반적으로 성공합니다. 실패한 요구는 안전하게 retried 할 수 있습니다. 우리는 우리가 더 많은 정보를 가지고 있기 때문에 업데이트합니다.

공식 인시던트 업데이트를 자동 번역했습니다.

1. 2026년 10월 7일 오전 10:21 UTCinvestigating

   우리는 현재 푸시 알림에 영향을 미치는 문제점을 조사하고 있습니다. 일부 사용자는 푸시 알림을받지 못할 수 있습니다. 다른 모든 플랫폼 특징은 정상적으로 작동됩니다. 우리의 팀은 고침에 일하고, 우리는 우리가 더 많은 정보가 있기 때문에 갱신을 여기에서 포스트할 것입니다.

2. 2026년 10월 7일 오전 10:31 UTCmonitoring

   수정이 적용되고 푸시 알림은 일반적으로 다시 전달됩니다. 우리는 모든 것을 정상에 돌려보내는 감시입니다.

3. 2026년 10월 7일 오전 11:45 UTCresolved

   이 사건은 해결되었습니다.

공식 인시던트 업데이트를 자동 번역했습니다.

1. 2026년 9월 11일 오전 10:28 UTCinvestigating

   우리는 현재이 문제를 조사하고 있습니다.

2. 2026년 9월 11일 오전 10:29 UTCresolved

   이 사건은 해결되었습니다.

공식 인시던트 업데이트를 자동 번역했습니다.

1. 2026년 9월 8일 오전 9:01 UTCidentified

   문제는 식별되고 수정이 구현됩니다.

2. 2026년 9월 8일 오전 9:02 UTCmonitoring

   수정이 구현되었습니다. 우리는 분석 데이터 복원에 노력하고 있으며 공정 중에 데이터 손실이 없습니다.

3. 2026년 9월 8일 오전 9:04 UTCresolved

   이 사건은 해결되었습니다.

공식 인시던트 업데이트를 자동 번역했습니다.

1. 2026년 8월 31일 오후 6:52 UTCresolved

   8 월 31 일 월요일, 출품자 및 주최자 스캐닝 배지를 스캔 한 후 오류를보고, 스캔 한 다음 장치에서 오프라인 스캔으로 저장. 이 작업 인터넷 연결 장치에서 일어난, 이는 왜 그것은 처음 장소 네트워크 문제에 대 한 실수. iOS에서 일부 장치는 탭 할 때 열리지 않은 빨간색 배너 만 보여주었습니다. ** 리드 데이터가 손실되지 않았습니다. ** 두 메커니즘은 동시에 데이터를 보호. 서버 측, 각 지도는 검사의 순간에 우리의 1 차적인 데이타베이스에 저장되었습니다, 실패한 단계의 앞에, 그래서 app가 오류를 보고될 때 지도 이미 존재했습니다. 장치 측, 앱의 오프라인 모드는 스캔을 캡처하고 복구 한 번 동기화했습니다. 같은 스캔의 반복 제출이 새로운 것을 추가하는 것보다 기존의 리드를 업데이트하기 때문에, 이것은 중복 리드를 만들지 않았습니다. 원인은 배지 스캐닝 경로에 특정했다. 등록, 이벤트 페이지, 회의, 메시징, 수출 및 주최자 백 오피스는이 창에서 영향을받지 않았습니다.

공식 인시던트 업데이트를 자동 번역했습니다.

1. 2026년 7월 1일 오후 5:37 UTCidentified

   Sherlock is currently experiencing degraded performance due to an inference capacity issue. We are working with our inference providers to restore full capacity as quickly as possible.

2. 2026년 7월 1일 오후 7:08 UTCidentified

   A fix to increase capacity is currently under testing.

3. 2026년 7월 1일 오후 7:48 UTCresolved

   Inference capacity has been fully restored

1. 2026년 6월 19일 오후 2:59 UTCidentified

   The issue has been identified and a fix is being implemented.

2. 2026년 6월 19일 오후 3:00 UTCresolved

   This incident has been resolved.

3. 2026년 6월 19일 오후 3:35 UTCresolved

   We are providing a detailed post-mortem report regarding the service disruption that affected Swapcard Login on June 19th, 2026. This issue was caused by an upgrade of our cache cluster, which led to temporary login failures. The goal of this post-mortem is to share insights from our assessment and the steps taken to resolve the issue while providing transparency to our customers. **Summary** As part of a planned upgrade to increase the capacity of our cache cluster, we carried out a zero-maintenance-window upgrade, a rollout designed to require no scheduled downtime. During the rollout, a misconfiguration that was not identified in our pre-production environment affected the login service in production, resulting in a period during which users were unable to sign in. **Timeline (UTC)** [16:35] — Migration of production services to the upgraded cache cluster proceeds as part of the zero-downtime rollout. [16:38] — The login service begins failing to connect to the cache; sign-in errors start. [16:42] — Incident detected and investigation begins. [16:45] — Root cause identified: corrected configuration deployed to login. [16:48] — Login fully recovers; error rates return to normal. **Mitigation** We updated the login service to connect to the upgraded cache cluster using the correct configuration, restoring sign-in functionality. As a precaution, we reviewed all other production services and corrected one additional service that carried the same latent misconfiguration before it could cause further impact. **Root cause** During a planned upgrade to increase the capacity of our cache cluster, a misconfiguration was introduced that left the Login service unable to connect to the upgraded cluster. The issue was not identified in our pre-production environment and surfaced in production during the rollout, causing sign-in to fail until the configuration was corrected. **Next steps** We are adding automated checks to (1) detect any reference to a cache cluster scheduled for decommission and (2) verify configuration parity between our pre-production and production environments, so this class of issue is caught before reaching production in future upgrades.

1. 2026년 3월 19일 오후 1:07 UTCresolved

   This incident has been resolved.

2. 2026년 3월 19일 오후 1:07 UTCresolved

   ### **Summary** Approximately two weeks ago, users began experiencing disruption when accessing analytics data within App Studio. Initial investigation did not immediately identify the root cause; a deeper analysis ultimately traced the issue to a routine infrastructure consolidation that was capping database capacity below the threshold required to handle specific event analytics queries spanning long date ranges. Full service has since been restored following a targeted increase in database resources. ### **Timeline** * **2026-03-11 19:32 UTC** — Incident created. Initial investigation launched following a rise in user-reported loading issues on analytics, despite no prior performance degradation. The initial investigation did not surface a clear root cause at this stage. * **2026-03-18 10:45 UTC** — Automated monitoring triggered alerts for elevated database error rates, corroborated by incoming user reports of analytics failures. * **2026-03-19 09:30 UTC** — Deeper analysis by the Engineering team identified the root cause: high latency and query failures isolated to specific event analytics queries over long date ranges, linked to the earlier infrastructure consolidation. * **2026-03-19 09:40 UTC** — Remediation plan executed: database resources scaled up to meet query load demands. * **2026-03-19 10:05 UTC** — Database performance stabilized; analytics functionality verified as fully restored. ### **Resolution** To restore service, database resources were immediately scaled up to accommodate the throughput required by analytics queries. Once additional capacity was provisioned, query timeouts ceased and application stability was reestablished. The system remained under active monitoring until no further degradation was observed, at which point the incident was marked resolved. ### **Root Cause Analysis** The incident was caused by a capacity mismatch introduced during a routine infrastructure consolidation. Such consolidations are a standard part of our infrastructure lifecycle and are conducted with active monitoring in place to ensure continued performance across all services. In this instance, however, the capacity allocated to the Analytics Database was capped at a level that, while adequate for standard workloads, could not sustain the computational demands of specific event analytics queries executed over long date ranges. The initial investigation did not immediately surface this connection; it was only through deeper analysis that the consolidation was identified as the underlying cause. When these queries were executed, they exceeded available processing capacity, resulting in database timeouts and repeated failures of the analytics service; conditions that persisted until capacity was restored. The investigation also surfaced a broader pattern: a gradual uptick in user-reported loading issues over the preceding two weeks, despite analytics having previously performed reliably. This indicated the capacity cap had been quietly eroding performance well ahead of the full outage. ### **Conclusion** We are straightening our processes to prevent recurrence. Monitoring coverage applied during database consolidations will be strengthened to detect query latency degradation earlier, and capacity planning will be refined to better account for the demands of specific event analytics queries over long date ranges. Looking further ahead, the team is actively working on delivering a fully revamped analytics pipeline to production later this year. This is not an incremental improvement; the new pipeline is being built to deliver deeper, richer metrics that give event organizers a comprehensive view of their event performance.

1. 2026년 3월 13일 오전 2:05 UTCresolved

   This incident has been resolved.

2. 2026년 3월 13일 오전 3:05 UTCresolved

   ## Summary On March 11, 2026, we experienced a service disruption affecting the visibility of speaker and attendee profiles across multiple events. The incident was triggered by a timing issue during a scheduled infrastructure update — a data refresh was initiated before new field definitions had fully propagated to our Search Engine, causing the indexation system to be built from an outdated configuration. Our Search Engine relies on an indexation system to serve profile data quickly across events. When the refresh completed against the stale configuration, new fields were rejected due to strict schema enforcement, leaving profiles absent from search results and listings. The goal of this post-mortem is to share our assessment and the steps taken to resolve the issue, while providing full transparency to our customers. ## Timeline | Time \(UTC\) | Event | | --- | --- | | Mar 11 – 11:04 | Infrastructure update deployed to production. Changes confirmed as applied. | | Mar 11 – 13:21 | Data refresh initiated as part of the planned release process. Previous indexation remained fully functional during this period. | | Mar 11 – 13:42 | Release step complete. Data refresh running, expected duration ~3–4 hours. | | Mar 11 – 14:49 | Configuration mismatch errors first logged internally. | | Mar 11 – 15:33 | Data refresh reported as complete. Spot checks passed. Final release step initiated. | | Mar 11 – 16:52 | Reports of missing profiles across multiple events surfaced. Incident response team immediately activated. | | Mar 11 – 17:29 | Root cause identified in under few minutes: indexation system had been built before the new fields were fully available. Internal alerts corroborated the finding. | | Mar 11 – 18:08 | Analysis completed confirming the refresh did not produce an indexation with the expected new fields. | | Mar 11 – 18:39 | Corrective data refresh triggered with the correct configuration fully in place, within 30 minutes of initial reports. | | Mar 11 – 18:43 | New indexation confirmed to contain the correct fields. Full refresh in progress. | ## Resolution Once the root cause was identified, the team immediately initiated a new data refresh against the correct, fully propagated configuration. Profile visibility and search behavior were monitored throughout the process. Once the refresh completed, all profiles returned to normal search and listing behavior. ## Root Cause Analysis The root cause was a race condition between the deployment of new Search Engine field definitions and the triggering of a data refresh. The refresh started before the new configuration had fully propagated, causing the indexation system to be built from a stale schema. Due to strict schema enforcement, subsequent write operations referencing the new fields were rejected, leaving profiles absent from search results and listings. Several factors contributed to this incident: * **Ambiguous deployment signal:** A pipeline warning created uncertainty around whether the deployment had completed cleanly. * **Insufficient spot checks:** Post-refresh verification passed at a surface level, masking the underlying schema mismatch until live traffic exposed it. **Short-term improvements:** * Add a mandatory pre-refresh validation step to confirm the Search Engine schema matches the expected configuration before any refresh is initiated. * Update the release process to include explicit schema validation checkpoints for sub field, as a required step. ## Conclusion This incident stemmed from a timing edge case in our release process. The improvements listed above reflect our commitment to preventing similar incidents and maintaining a reliable platform for all our customers. For any questions or concerns regarding this incident, please reach out to our support team.

1. 2026년 3월 11일 오후 2:33 UTCinvestigating

   We are currently investigating this issue.

2. 2026년 3월 11일 오후 2:33 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

3. 2026년 3월 11일 오후 2:33 UTCresolved

   This incident has been resolved.

1. 2026년 3월 11일 오후 2:33 UTCresolved

   This incident has been resolved.

1. 2026년 3월 11일 오후 2:33 UTCidentified

   The issue has been identified and a fix is being implemented.

2. 2026년 3월 11일 오후 2:33 UTCresolved

   Please see our post-mortem below regarding the 404 error page from May 5h, 2022 at 13:11 UTC through to 13:20 UTC. It is our goal in this post-mortem to provide details on our initial assessment of the incident communicated on the Swapcard status page and to describe the remediation actions that we have taken to restore service. ## **Incident summary** On Tuesday, May 5th at 13:12 UTC, we experienced a 404 error page our studio due to a miss configuration on the web application \([studio.swapcard.com](http://studio.swapcard.com)\). On Tuesday, May 5th at 13:12 UTC, our infrastructure team has been automatically alerted of an un usual 404 error page on the web application \([studio.swapcard.com](http://studio.swapcard.com)\). Swapcard monitoring detected the start of disruption and activated the Swapcard Incident Response team. Swapcard’s team worked to triage and mitigate the incident by checking any recent changes on the web application \([studio.swapcard.com](http://studio.swapcard.com)\) configuration and revert to an older version. In parallel, the cause of the issue was investigated and short & mid term plans were put in place. ## **Mitigation deployment** At 13:13 UTC our infrastructure team has immediately revert the configuration changes for studio.swapcard.com. The switch process took around ~6min. The error reporting has stopped as the update propagated through our infrastructure. Swapcard engineering then monitored [studio.swapcard.com](http://studio.swapcard.com) to ensure full and proper recovery. As a result of the deployment of that change, customers would then see a reduction of the 404 error messages. At 13:20 UTC, Swapcard confirmed that the update was completed and no further error detected or reported. Swapcard’s Engineering team identified the root cause, and by ~14:31 UTC has redeploy the configuration change corrected. ## **Event Outline** ### **Duration Summary** Time alerted to the outage: 1 minutes Time to identify the source of disruption: ~2 minutes Time to initiate recovery: ~6 minutes Time to monitor and restore service pre-crash: ~1 minutes ### **Events of 2022 May 5th \(UTC\)** \(13:11 UTC\) | Initial onset of the error rate increase \(13:12 UTC\) | Disruption identified by Swapcard monitoring \(13:13 UTC\) | Configuration revert has been initiated \(13:12 UTC\) | Swapcard status post activated \(13:19 UTC\) | 404 error page decrease and recovered \(13:20 UTC\) | Incident mitigated \(13:20 UTC\) | Status post resolved \(14:31 UTC\) | Swapcard Engineering redeploy the configuration change corrected Affected customers may have been impacted by varying degrees and with a shorter duration than described above. ## **Forward Planning** Swapcard has deployed a permanent fix for this incident in accordance with our high standard in terms of deliverability. **We consider the likelihood of a recurrence of this issue to be extremely low.**

3. 2026년 3월 11일 오후 2:33 UTCresolved

   This incident has been resolved.

4. 2026년 3월 11일 오후 2:33 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

1. 2026년 3월 11일 오후 2:33 UTCresolved

   This incident has been resolved.

1. 2026년 3월 11일 오후 2:33 UTCresolved

   On Monday, January 17th at 15:30 UTC, we experienced an increase response time on our event app, the incident has lasted ~2min

1. 2026년 3월 11일 오후 2:33 UTCresolved

   Please see our post-mortem below regarding the increase error rate on community list for event planner accounts only from November 29th, 2021 at ~14:17 UTC. On Monday, November 29th at ~14:19 UTC, we experienced an increase error rate on the community list page on event planner accounts only, this is due to a data upgrade on organisation rights management in accordance to our zero downtime strategy. On Monday, November 29th at ~14:17 UTC, our infrastructure team has been automatically alerted of an un usual increase error rate on community list pages for event planner account. Swapcard monitoring detected the start of disruption and activated the Swapcard Incident Response team. Swapcard’s team worked to triage and mitigate the incident by reverting the data upgraded to their original value and reverted services configuration. In parallel, the cause of the issue was investigated and short & mid & long term plans were put in place. Swapcard has deployed a permanent fix for this incident in accordance with our high standard in terms of deliverability. Affected customers may have been impacted by varying degrees and with a shorter duration than described above, the engineering team confirmed that no disruption for attendees & exhibitors has been reported. We consider the likelihood of a recurrence of this issue to be extremely low.

1. 2026년 3월 11일 오후 2:33 UTCresolved

   Please see our post-mortem below regarding the increase error rate on video call & roundtable from November 3th, 2021 at 17:05 UTC through to 17:47 UTC. It is our goal in this post-mortem to provide details on our initial assessment of the incident communicated on the Swapcard status page and to describe the remediation actions that we have taken to restore service. ## **Incident summary** On Thursday, November 3th at 17:05 UTC, we experienced an increase error rate on video call & roundtable on our event app due to a authentification configuration issue between our platform and our video call partner. On Thursday, November 3th at 17:05 UTC, our infrastructure team has been automatically alerted of an un usual increase error rate on video call & roundtable feature. Swapcard monitoring detected the start of disruption and activated the Swapcard Incident Response team. Swapcard’s team worked to triage and mitigate the incident by switching from one to the other backup video call partner account to alleviate customer impact. In parallel, the cause of the issue was investigated and short & mid & long term plans were put in place. ## **Mitigation deployment** At 17:08 UTC our infrastructure team has immediately trigger the switch to our backup plan for video call outage in to alleviate the customer impact. The switch process took around ~15min. The error reporting has stoped as the update propagated through our different endpoints. Swapcard engineering then monitored all the endpoints to ensure full and proper recovery. As a result of the deployment of that change, customers would then see a reduction of the error messages. At 17:53 UTC, Swapcard confirmed that the update was completed and no further error detected or reported. Swapcard’s Engineering team identified the root cause, and by ~18:20 UTC has re-enabled the original authentification process with our video call partner. ## **Event Outline** ### **Duration Summary** Time alerted to the outage: 1 minutes Time to identify the source of disruption: ~10 minutes Time to initiate recovery: ~15 minutes Time to monitor and restore service pre-crash: ~10 minutes ### **Events of 2021 November 3th \(UTC\)** \(17:05 UTC\) | Initial onset of the error rate increase \(17:05 UTC\) | Disruption identified by Swapcard monitoring \(17:08 UTC\) | Switch from backup identical provider account has been initiated \(17:27 UTC\) | Swapcard status post activated \(17:40 UTC\) | Video & roundtable error rate decrease and recovered \(17:47 UTC\) | Incident mitigated \(17:53 UTC\) | Status post resolved \(18:20 UTC\) | Swapcard Engineering identified the original issue Affected customers may have been impacted by varying degrees and with a shorter duration than described above. ## **Forward Planning** Swapcard has deployed a permanent fix for this incident in accordance with our high standard in terms of deliverability. **We consider the likelihood of a recurrence of this issue to be extremely low.**

2. 2026년 3월 11일 오후 2:33 UTCresolved

   This incident has been resolved.

3. 2026년 3월 11일 오후 2:33 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

4. 2026년 3월 11일 오후 2:33 UTCidentified

   The issue has been identified and a fix is being implemented.

5. 2026년 3월 11일 오후 2:33 UTCinvestigating

   We are currently investigating this issue.

1. 2026년 3월 11일 오후 2:33 UTCresolved

   This incident has been resolved.

2. 2026년 3월 11일 오후 2:33 UTCidentified

   The issue has been identified and a fix is being implemented.

3. 2026년 3월 11일 오후 2:33 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

4. 2026년 3월 11일 오후 2:33 UTCresolved

   Please see our post-mortem below regarding the increase response time on our chat endpoint from September 2th, 2021 at 15:38 UTC through to 15:41 UTC. It is our goal in this post-mortem to provide details on our initial assessment of the incident communicated on the Swapcard status page and to describe the remediation actions that we have taken to restore service. ## **Incident summary** On Thursday, September 2th at 15:38 UTC, we experienced an increase response time on our event app due to one of our replicate database server being under pressure because of a specific query. On Thursday, September 2th at 15:37 UTC, our infrastructure team has been automatically alerted of an increase response time on our event app, requests were replying in a twice higher average time compare to the usual average response time. Swapcard monitoring detected the start of disruption at ~15:37 UTC and activated the Swapcard Incident Response team. Swapcard’s team worked to triage and mitigate the responsible database queries to alleviate customer impact. In parallel, the cause of the issue was investigated and short & mid & long term plans were put in place. ## **Mitigation deployment** At 15:39 UTC our monitoring system had immediately turned off the responsible database that was under pressure to mitigate the impact on our services and recover a proper event app response time. The turn off process took around ~15s. Event app increase response time stopped as the database turned off configuration propagated through our endpoints. Swapcard engineering team then monitored the event app endpoints to ensure full and proper recovery. As a result of the deployment of that change, customers would then see a reduction of the chat loading time and error messages. At 15:43 UTC, Swapcard confirmed that the update was completed and response time restored to pre-incident levels and ensuring that the traffic was back to the pre-incident rate. Swapcard’s Engineering team identified the cause, and by ~15:50 UTC has re-enabled the impacted database replication, the root cause has also been identified and solved. ## **Event Outline** ### **Duration Summary** Time alerted to the outage: 1 minutes Time to identify the source of disruption: ~2 minutes Time to initiate recovery: ~15 seconds Time to monitor and restore response time pre-crash: ~2 minutes ### **Events of 2021 September 2th \(UTC\)** \(15:37 UTC\) | Initial onset of the response time increase \(15:38 UTC\) | Disruption identified by Swapcard monitoring \(15:38 UTC\) | Swapcard status post activated \(15:39 UTC\) | Automatic monitoring system mitigated the issue \(15:40 UTC\) | Impacted event app began to recover \(15:41 UTC\) | Incident mitigated, pre-incident response time restored \(15:43 UTC\) | Status post resolved \(15:50 UTC\) | Swapcard Engineering identified the issue Affected customers may have been impacted by varying degrees and with a shorter duration than described above. ## **Forward Planning** Swapcard has deployed a permanent fix for this incident in accordance with our high standard in terms of deliverability.

1. 2026년 3월 11일 오후 2:33 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

2. 2026년 3월 11일 오후 2:33 UTCresolved

   This incident has been resolved.

3. 2026년 3월 11일 오후 2:33 UTCresolved

   Please see our post-mortem below regarding the increase response time on our chat endpoint from August 10th, 2021 at 16:48 UTC through to 17:15 UTC. It is our goal in this post-mortem to provide details on our initial assessment of the incident communicated on the Swapcard status page and to describe the remediation actions that we have taken to restore service. ## **Incident summary** On Tuesday, August 10th at 16:48 UTC, we experienced an increase response time on our chat endpoint due to the pressure of two database queries \(Count unread channel & Number of active one to one conversation\) on our chat database. On Tuesday, August 10th at 16:48 UTC, our infrastructure team has been automatically alerted of an increase response time on our chat endpoint, requests was replying in an average of 300ms compare to the usual average of 4ms response time. Swapcard monitoring detected the start of disruption at ~16:47 UTC and activated the Swapcard Incident Response team. Swapcard’s team worked to triage and mitigate the responsible database queries to alleviate customer impact. In parallel, the cause of the issue was investigated and short & mid & long term plan were put in place. ## **Mitigation deployment** At 16:52 UTC, Swapcard’s Engineering team identified the cause, and by 16:57 UTC had immediately turned off the responsible database queries to mitigate the impact on our services and recover a proper chat endpoint response time, the turn off process took us around ~8 min to ensure proper execution and deployment. Chat increase response time stopped as the update propagated through our chat endpoints. Swapcard engineering then monitored all the chat endpoints to ensure full and proper recovery. As a result of the deployment of that change, customers would then see a reduction of the chat loading time and error messages. At 17:30 UTC, Swapcard confirmed that the update was completed and response time restored to pre-incident levels and ensuring that the traffic was back to the pre-incident rate. ## **Event Outline** ### **Duration Summary** Time alerted to the outage: 01 minutes Time to identify the source of disruption: ~5 minutes Time to initiate recovery: ~5 minutes Time to monitor and restore response time pre-crash: ~12 minutes ### **Events of 2021 August 10 \(UTC\)** \(16:46 UTC\) | Initial onset of the response time increase \(16:47 UTC\) | Disruption identified by Swapcard monitoring \(16:48 UTC\) | Swapcard status post is activated \(16:52 UTC\) | Swapcard Engineering identified the issue \(17:05 UTC\) | Swapcard Engineering mitigate the issue \(17:13 UTC\) | Impacted chat endpoint began to recover \(17:15 UTC\) | Incident Mitigated, pre-incident response time restored \(17:38 UTC\) | Status post resolved Affected customers may have been impacted by varying degrees and with a shorter duration than as described above. ## **Forward Planning** Swapcard has deployed a permanent fix for this incident by optimising the responsible database queries and implemented technical measures to ensure proper covering of other databases queries on the chat endpoint. In accordance with our high standard in terms of deliverability, Swapcard conducted during the 5 following days post incident a series of optimisation to prevent further similar issues on the chat endpoint. **We consider the likelihood of a recurrence of this issue to be extremely low and will further reduce the risk as we make improvements.**

4. 2026년 3월 11일 오후 2:33 UTCinvestigating

   We are currently investigating this issue.

5. 2026년 3월 11일 오후 2:33 UTCidentified

   The issue has been identified and a fix is being implemented.

1. 2026년 3월 11일 오후 2:33 UTCresolved

   This incident has been resolved.

2. 2026년 3월 11일 오후 2:33 UTCresolved

   Please see our post-mortem below regarding a service delivery disruption that affected Swapcard customers from July 6th, 2021 at 05:58 UTC through to 6:18 UTC. It is our goal in this post-mortem to provide details on our initial assessment of the incident communicated on the Swapcard status page and to describe the remediation actions that we have taken to restore service. ## **Incident summary** On Tuesday, July 6th at 06:01 UTC, we experienced an increase error rate due to a rolling update \(capacity and security\) apply on our caching sever \(Redis\). On Tuesday, July 6th at 06:01 UTC, our private cloud provider has deployed an automatic update on our main caching server \(Redis\). This automatic change is applied in accordance with Swapcard standard infrastructure & security change and enhancement practices. Cluster are configured in High Availability/Failover mode to prevent any disruption of services and support our zero downtime strategy. An issue in our hot switch mechanism has prevent our services to take in consideration the primary/secondary switch has cause a significant disruption on the content delivery. Swapcard monitoring detected the disruption at 05:58 UTC and activated the Swapcard Incident Response team. Swapcard’s team worked to triage and restore services to alleviate customer impact. In parallel, the cause of the issue was investigated and mitigations were put in place. ## **Mitigation deployment** At 05:58 UTC, Swapcard’s Engineering team identified the cause, and by 06:08 UTC had deploy manually a configuration change to reflect the primary/secondary changes. Content delivery disruption stopped as the update propagated through our services. Swapcard engineering then monitored all the services to ensure full and proper recovery by 6:17 UTC. As a result of manually deploying this configuration change, customers would then see a reduction in the 5XX errors and error messages. At 06:23 UTC, Swapcard confirmed that the update was completed and capacity restored to pre-incident levels, ensuring that the traffic was back to the pre-incident rate. ## **Event Outline** ### **Duration Summary** Time alerted to the outage: 01 minutes Time to identify the source of disruption: 3 minutes Time to initiate recovery: 5 minutes Time to monitor and restore capacities pre-crash: 8 minutes ### **Events of 2021 June 22 \(UTC\)** \(05:58 UTC\) | Initial onset of content delivery disruption \(05:58 UTC\) | Disruption identified by Swapcard monitoring \(06:03 UTC\) | Swapcard status post is activated \(06:05 UTC\) | Swapcard Engineering identified the issue \(06:08 UTC\) | Swapcard Engineering initiate the manual configuration change \(06:13 UTC\) | Impacted services began to recover \(06:16 UTC\) | Majority of services recovered, additional mitigation measures taken \(06:17 UTC\) | Incident Mitigated, pre-incident capacity restored \(06:23 UTC\) | Status post resolved Affected customers may have been impacted by varying degrees and with a shorter duration than as described above. ## **Forward Planning** Swapcard has deployed a permanent fix for this incident and implemented technical measures to ensure proper automatic maintenance windows on this endpoint and improve the primary/secondary hot switch configuration. In accordance with our high standard in terms of deliverability, Swapcard will conduct an internal audit on the hot switch capacity and configuration. Control are already in place but today’s incident highlights the need for improvement on this part. **We consider the likelihood of a recurrence of this issue to be extremely low and will further reduce the risk as we make future interventions and improvements to our infrastructure and procedures.**

3. 2026년 3월 11일 오후 2:33 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

4. 2026년 3월 11일 오후 2:33 UTCidentified

   The issue has been identified and a fix is being implemented.