Störungsverlauf von Onfido

1. 7. September 2026 um 15:53 UTCresolved

   Zwischen 14:30 und 15:30 UTC erlebten Kunden in der CA-Region einen eingeschränkten Zugriff auf die Public API. Eine fehlerhafte Bereitstellung in der Authentifizierungsschicht führte dazu, dass gültige Token falsch abgelehnt wurden, was zu fehlgeschlagenen API-Anforderungen führte. Das Problem ist vollständig gelöst. Es ist keine Aktion an Ihrem Ende erforderlich. Wir entschuldigen uns für die Störung.

Automatisch aus der offiziellen Störungsmeldung übersetzt.

1. 1. September 2026 um 14:37 UTCresolved

   Von etwa 1:21 Uhr UTC bis 1:46 Uhr UTC gab es eine Störung bei der Annahme von Dokumentenanforderungen vom Web SDK sowie von Entrust IDV SDKs für Flüsse, die Webmodule enthalten. Onfido Android & iOS SDKs waren nicht betroffen. Wir entschuldigen uns für diese Störung. Ein detailliertes Postmortem wird folgen, sobald wir unsere Untersuchung abgeschlossen haben.

Automatisch aus der offiziellen Störungsmeldung übersetzt.

1. 27. Juli 2026 um 18:12 UTCinvestigating

   Wir untersuchen derzeit Verzögerungen in unserem US-Cluster für die Verarbeitung von Gesichtsähnlichkeit, Dokumenten und bekannten Gesichtern.

2. 27. Juli 2026 um 18:23 UTCinvestigating

   Wir sehen eine verminderte Leistung einiger Knoten in unserem Suchcluster für indizierte Gesichter. Wir untersuchen weiterhin die Ursache und mögliche Abhilfemaßnahmen.

3. 27. Juli 2026 um 18:51 UTCidentified

   Wir haben die Hauptursache für verminderte Leistung in unserem US-Suchcluster identifiziert und arbeiten aktiv daran, sie zu beheben. Wir werden nachgehen, wenn wir vollständig wiederhergestellt sind.

4. 27. Juli 2026 um 19:18 UTCmonitoring

   Wir haben einen Fix implementiert und überwachen weiterhin die Qualität der Verarbeitungszeit, die jetzt verbessert wird.

5. 27. Juli 2026 um 19:20 UTCmonitoring

   Die Verarbeitungszeit verbessert sich ständig, alle Systeme sind jetzt in Betrieb.

6. 27. Juli 2026 um 19:21 UTCresolved

   Die Bearbeitungszeiten sind wieder normal. Dieses Problem ist nun gelöst. Post-mortem folgt bald.

Automatisch aus der offiziellen Störungsmeldung übersetzt.

1. 24. Juli 2026 um 04:18 UTCinvestigating

   Identity Enhanced erlebt einen starken Rückgang der klaren Preise für Bewerber in der ganzen Welt außer Großbritannien

2. 24. Juli 2026 um 04:36 UTCidentified

   Wir arbeiten mit unserem Drittanbieter zusammen, um das Problem zu beheben, und werden Updates teilen, sobald sie verfügbar sind.

3. 24. Juli 2026 um 04:55 UTCidentified

   Wir arbeiten weiter an einer Lösung für dieses Problem.

4. 24. Juli 2026 um 05:09 UTCmonitoring

   Wir sehen, wie sich die Servicewiederherstellung unseres Anbieters verbessert und die Erfolgsraten verbessert werden. Wir verfolgen die Situation weiterhin aufmerksam.

5. 24. Juli 2026 um 07:08 UTCresolved

   Der Anbieter hat das Problem behoben und der Dienst funktioniert jetzt normal

Automatisch aus der offiziellen Störungsmeldung übersetzt.

1. 23. Juni 2026 um 02:38 UTCinvestigating

   We cannot complete QES tasks because our electronic signature provider is encountering issues.

2. 23. Juni 2026 um 02:57 UTCidentified

   Our provider is experiencing issues. All QES tasks cannot complete for now.

3. 23. Juni 2026 um 03:20 UTCidentified

   Our provider is still experiencing issues. All QES tasks cannot complete for now.

4. 23. Juni 2026 um 03:45 UTCmonitoring

   The issue has been fixed. We're monitoring and resuming blocked QES tasks, if they can be.

5. 23. Juni 2026 um 03:49 UTCresolved

   This incident has been resolved.

1. 22. Juni 2026 um 17:44 UTCmonitoring

   We're currently monitoring the EU cluster after an Amazon RDS issue.

2. 22. Juni 2026 um 17:56 UTCresolved

   We’re seeing recovery across our internal metrics, and processing has now returned to full capacity. At this time, the issue appears to be resolved. Our current leading hypothesis is resource contention on a shared Amazon RDS instance that several services depend on, potentially related to a VACUUM operation running alongside a long-running job deleting a large volume of accumulated historical data. We have not yet confirmed the root cause and will continue investigating as follow-up, but service has been restored for now. We'll be following up with a public post-mortem.

3. 2. Juli 2026 um 14:27 UTCpostmortem

   **Incident date:** 22 June 2026 **Region:** EU \(eu-west-1\) **Affected EU services**: API, Dashboard, Applicant Form, Document Verification, Facial Similarity, Watchlist, Identity Enhanced, Webhooks, Known Faces, Autofill, QES and Device Intelligence. **Customer impact:** ~16:50–17:00 UTC \(acute degradation\); ~17:00–17:20 UTC \(backlog recovery\) ## Summary On 22 June 2026, from approximately 16:50 UTC, a shared database cluster serving our EU region came under severe load and could not reliably serve queries for about 10 minutes. EU services returned elevated errors, and processing throughput briefly fell to ~20–35% of normal levels, with many subcomponents of our system \(e.g., Facial Similarity report processing\) being entirely disrupted, some others less heavily impacted \(e.g., Document report processing\). Service recovered by 17:01 UTC, the database fully stabilizing after an automatic failover \(~17:05–17:07 UTC\). A resultant report backlog was cleared by ~17:20 UTC. Requests in flight during the acute degradation window may have failed unless retried; queued background work was processed automatically once the database recovered. ## Root cause The incident was triggered by a routine database storage-reclamation task following standard scheduled data-deletion processing. This task normally completes without issue; why it failed on this occasion remains under investigation, although we observed that it was processing a larger-than-usual backlog. We have a support case open with our cloud provider to confirm a definitive root cause. The reclamation task began to compete with normal application queries, which slowed as the database struggled to keep up. Applications opened more and more connections, leading to connection saturation and causing queries across the affected services to fail. The database stabilized when an automatic failover to a healthy standby was triggered; the contention fully resolving with the failover to a new instance. ## Timeline \(UTC\) * **16:50** — Our monitoring detected errors and elevated latency across EU services caused by resource contention on a shared database cluster. * **16:53** — We start to see improvements, but system still not acting at normal levels of performance. * **~17:00** — Customer-facing errors subsided and processing resumed as the contention eased. * **17:01** — On-call engineers opened an incident and continued investigations. * **~17:05–17:07** — The database performed an automatic failover to a healthy instance, which reset the overloaded writer and fully stabilised the cluster. The failover was triggered because of resource contention \(out of memory\) caused by the heavy vacuuming in the preceding minutes of the incident. Once the impacting vacuum operations had finished freeing up resources, we had started to see signs of improvement \(16:53—17:01\), but added latency in the feedback loop and aggregation window at AWS still decided to trigger the failover, even though we were already in a recovering state. * **17:15–17:20** — Requests that had queued during the incident were worked through and the backlog returned to normal. * **17:21–17:56** — We monitored the recovery and confirmed processing remained at full capacity. ## Remedies * Reviewing connection limits and pooling so a single service cannot saturate a shared database, and evaluating dedicated database clusters per product to remove cross-service impact. * Changing large historical-data deletions to run in smaller, throttled batches, and tuning database maintenance to avoid large catch-up operations. * Adding earlier, proactive alerting on database memory, connections and load so we can intervene before customer impact. * Continue working with our cloud provider on a definitive root cause.

1. 17. Juni 2026 um 04:59 UTCidentified

   One of our providers is still experiencing issues, resulting in higher TaT for electronic signature tasks.

2. 17. Juni 2026 um 05:30 UTCidentified

   Our provider is actively working on fixing the issue.

3. 17. Juni 2026 um 06:19 UTCidentified

   Our provider is still working on fixing the issue. We're monitoring the impact and we're making sure to keep the TaT as low as possible, considering the situation.

4. 17. Juni 2026 um 06:50 UTCidentified

   Our provider's error rate dropped significantly. The TaT of QES tasks is now close to the usual value.

5. 17. Juni 2026 um 08:17 UTCidentified

   We're working closely with our provider to find the reason of the remaining errors. The error rate contacting our provider is stable and the impact on the QES task TaT is under control.

6. 17. Juni 2026 um 08:48 UTCmonitoring

   Our provider was able to fix the issue and we don't get any error when contacting them. The TaT of QES tasks is back to normal. Monitoring the situation to make sure the errors don't come back.

7. 17. Juni 2026 um 09:23 UTCresolved

   The QES tasks TaT is back to normal.

1. 16. Juni 2026 um 22:19 UTCidentified

   Our electronic signature tasks are taking more time to complete because one of our providers is doing some maintenance.

2. 16. Juni 2026 um 22:52 UTCidentified

   Our provider is still not able to answer to all requests, resulting in longer-than-usual electronic signature task processing time.

3. 16. Juni 2026 um 23:17 UTCresolved

   The service is operational again.

1. 13. Juli 2026 um 08:39 UTCresolved

   Incident was solved.

2. 13. Juli 2026 um 08:40 UTCpostmortem

   ## Summary Between May and June 2026, around 0.037% of the evidence files generated on the platform were incomplete and included in the corresponding evidence folders. There was no impact on workflow results, and no incorrect information was ever displayed — the affected files were simply empty rather than wrong. A small number of requests to generate timeline files were also affected \(around 0.073%\). ## Root Causes The issue was introduced during an initiative to improve how timeline and evidence PDFs are generated — making them significantly smaller, more efficient, and more resilient to produce. As part of these changes, in rare cases the new flow attempted to generate the PDF before its content had fully loaded, resulting in a blank or nearly empty file. Because the problem was intermittent and occurred only in rare timing conditions, it was not consistently reproduced during rollout testing. ## Timeline * 15 June 2026: Engineering investigation found an incomplete PDF and started the root cause investigation * 17 June 2026: Root cause investigation completed and a safeguard to prevent incomplete PDFs was deployed ## Remedies * We corrected the PDF generation flow so PDFs are only printed after content has fully loaded, and added a safeguard that stops generation when an incomplete PDF is detected. * We also improved our monitoring and test coverage for PDF generation, so similar issues are caught earlier in the future.

1. 6. Juni 2026 um 14:08 UTCinvestigating

   Some Watchlist search search profiles are not working correctly the impacted reports are not being processed. We are investigation the root cause.

2. 6. Juni 2026 um 14:36 UTCidentified

   We identified the issue and are implementing a fix.

3. 6. Juni 2026 um 15:05 UTCmonitoring

   The issue has been fixed, and the reports are being processed correctly. We are still monitoring the service.

4. 6. Juni 2026 um 15:49 UTCresolved

   This incident has been resolved.

1. 12. Mai 2026 um 12:25 UTCidentified

   We are facing issues with one of our providers and we see a slight decrease in clear rate for Identity Reports in UK jurisdiction

2. 12. Mai 2026 um 12:55 UTCmonitoring

   A fix has been implemented and we are monitoring the results. All clear rates should be back to normal

3. 12. Mai 2026 um 13:14 UTCresolved

   All reports are back to normal.

1. 9. Mai 2026 um 11:27 UTCidentified

   We cannot complete QES tasks because our electronic signature provider is encountering issues.

2. 9. Mai 2026 um 11:27 UTCidentified

   The provider is working on a fix.

3. 9. Mai 2026 um 11:32 UTCmonitoring

   Our provider fixed the issue. QES tasks are completing now. Tasks started during the incident were resumed.

4. 9. Mai 2026 um 12:03 UTCresolved

   The service is operational, no error where detected after the fix.

1. 5. Mai 2026 um 06:52 UTCinvestigating

   We noticed an issue with our QES tasks, we're investigating.

2. 5. Mai 2026 um 06:56 UTCidentified

   Our provider is experiencing issues. All QES tasks cannot complete for now.

3. 5. Mai 2026 um 07:35 UTCidentified

   Our provider is fixing the issue. QES tasks still cannot complete.

4. 5. Mai 2026 um 08:04 UTCmonitoring

   The issue has been fixed. We're monitoring and resuming blocked QES tasks, if they can be.

5. 5. Mai 2026 um 08:39 UTCresolved

   QES is operational again.

1. 21. April 2026 um 14:56 UTCidentified

   We identified an issue with one of our provider and some QES tasks may encounter some problems.

2. 21. April 2026 um 15:14 UTCidentified

   Our provider is still experiencing issues. All QES tasks cannot complete for now.

3. 21. April 2026 um 16:12 UTCidentified

   Our provider continues experiencing issues and is working on a fix.

4. 21. April 2026 um 17:02 UTCidentified

   Our provider is slowly recovering, we expect some QES tasks to complete.

5. 21. April 2026 um 17:12 UTCresolved

   QES is operational again.

1. 3. Februar 2026 um 10:42 UTCidentified

   One of our provider is experiencing issues. QES capture tasks will fail and QES verification tasks will have an increase in TaT.

2. 3. Februar 2026 um 11:13 UTCidentified

   The provider is working on a fix, QES is still suffering a partial outage.

3. 3. Februar 2026 um 11:44 UTCidentified

   The provider is still working on a fix.

4. 3. Februar 2026 um 12:16 UTCidentified

   The provider is still working on a fix. QES verification tasks may start to time out, depending on their configuration, as we passed 90 minutes of downtime.

5. 3. Februar 2026 um 12:46 UTCmonitoring

   The provider fixed the issue. We can confirm our QES tasks are now processed correctly. We'll monitor the situation to make sure the error is indeed fixed.

6. 3. Februar 2026 um 13:16 UTCresolved

   We can confirm the error stopped and services are stable and operational.

1. 2. Februar 2026 um 15:36 UTCidentified

   We identified an issue with one of our provider and some QES tasks may encounter some problems.

2. 2. Februar 2026 um 16:15 UTCidentified

   Our provider is still experiencing issues. All QES tasks cannot complete for now.

3. 2. Februar 2026 um 16:46 UTCmonitoring

   Our provider is slowly recovering, we expect some QES tasks to complete.

4. 2. Februar 2026 um 17:18 UTCmonitoring

   We're now seeing the error rate decrease. Most of the QES tasks should complete.

5. 2. Februar 2026 um 17:24 UTCresolved

   QES is operational again.

1. 28. Januar 2026 um 11:16 UTCinvestigating

   We are currently investigating this issue.

2. 28. Januar 2026 um 11:58 UTCmonitoring

   Issue found and fixed. Increased turn around time for manual reports. Estimated time to live manual processing is 4h.

3. 28. Januar 2026 um 15:22 UTCresolved

   Incident is fully resolved, manual processing is now working normally. Manual reports will keep having an increased turn around time for a few more hours while it works through the task backlog.

4. 2. Februar 2026 um 16:14 UTCpostmortem

   ### Summary Manual task assignment for all EU customers stopped working between 10h50 UTC and 11:50 UTC. This led to an increase in manual processing Turnaround Time \(TaT\) affecting approximately 20% of our document verification volumes with all customers recovering to TaT SLA by 18h00 UTC. During this period: * All checks that required **manual review** showed an increase in TaT. * **Fully automated reports were not affected** and continued to run as normal. The issue was caused by a **configuration error in our internal task management system**, which prevented it from correctly assigning tasks to our analysts. We fixed the configuration and **restored normal processing** by 28 Jan 2026 11h50 AM UTC, and cleared all manual task backlogs by 18h00 UTC. We have updated our validation and deployment checks to prevent similar issues in the future. ### Root Causes _Manual processing queue assignment was affected by an invalid manual configuration input. This single queue configuration parameter resulted in an error that affected assignments in all queues._ ### Timeline * all times in UTC: _10:50: Configuration manually updated and errors started, no more tasks assigned._ _11:02: On-call is notified of a spike in manual system assignment errors through our monitoring_ _11:07: The error responsible for the spike is identified \(invalid UUID\)_ _11:10: Incident declared_ _11:45: Origin of invalid UUID is found_ _11:50: Bad configuration parameter is deleted_ _11:56: Configuration reintroduced correctly_ _18:00: Recovered from manual task backlog_ ### Remedies * Adding appropriate input configuration value validation * Improve task assignment resilience to these types of errors * Review configuration guidance and post-release monitoring

1. 26. Januar 2026 um 12:46 UTCresolved

   Between 12:15 UTC and 12:40 UTC there was disruption to document report processing in EU. Autofill requests also saw a disruption between 12:16 UTC and 12:27 UTC. A postmortem will follow.

2. 30. Januar 2026 um 11:40 UTCpostmortem

   ## Summary On 26 January 2026 between 12.16 UTC and 12.35 UTC, our document reports processing was severely degraded with customers experiencing extended processing time to about 85% of their traffic. The incident also caused extended processing time on Biometric Authentications and Biometric Verifications for 20% of the traffic between 12.25 UTC and 12.35 UTC. ## Root Causes A core service for fraud prevention on documents came under elevated load, causing kubernetes pods to go down in quick sequence. Our upstream service retry policy proved too aggressive to let the service recover and required manual scaling-up. The retry policy also caused elevated load on a shared database which in turn also affected the biometrics service. ## Timeline * **12:16 UTC** – Our monitoring detected a sharp increase in errors when processing document reports. * **12:19** **UTC** – The on‑call team was alerted and began investigating the affected document‑processing service. * **12:25 UTC** – We identified that the incident was also affecting a small portion of biometric checks, leading to some failures and short delays. * **12:29** **UTC** – On-call engineers manually increased capacity for the impacted document fraud‑prevention service. * **12:35** **UTC** – Error rates for both document reports and biometric checks returned to normal and new requests were being processed successfully. * **12:45–13:06** **UTC** – We processed document reports that were impacted during the incident window. * **13:15** **UTC** – We confirmed that all affected document and biometric requests had completed successfully and marked the incident as resolved. ## Remedies * We have continued to fine-tune our auto-scaling parameters of the affected service to scale up with lower CPU targets. * We modified the retry policy of fraud services to avoid overloading an already struggling service so that it can auto-recover. * We made our load-tests on this service more representative of production traffic \(similar image sizes/document type distribution…\) and test for accelerated traffic spikes. * We lowered the total amount of shared database connections the document processing can take to avoid noisy neighbour impact on biometrics processing.

1. 21. Januar 2026 um 11:05 UTCinvestigating

   We have a high error rate in document report processing and autofill. Our team is investigating

2. 21. Januar 2026 um 11:49 UTCidentified

   We have identified the issue as a configuration issue between services, and are working on a fix.

3. 21. Januar 2026 um 12:07 UTCmonitoring

   We have deployed a fix. New document reports and autofill requests are being processed normally. We will rerun the earlier impacted reports.

4. 21. Januar 2026 um 12:34 UTCresolved

   This issue is now resolved: Between 10:30 and 12:00 UTC, document reports could not be processed and autofill requests were failing in the US region. Starting from 12:00, live traffic was unaffected. From 12:00 to 12:30 UTC, pending reports were processed. We take a lot of pride in running a robust, reliable service and we're working hard to make sure this does not happen again. A detailed postmortem will follow once we've concluded our investigation.

5. 30. Januar 2026 um 11:39 UTCpostmortem

   ### Summary Around 10:29 UTC, around 10 minutes after restoring service in the EU and CA from the 10:05-10:17 UTC outage, we go alerted for high error rate in the US region and we noticed the new ML model version was back online in that region. This impacted Turnaround Time \(TaT\) for all Document Report and Studio Autofill until 12:11 UTC. Autofill Classic had an average 100% error rate during the same time frame. All impacted Document Reports and Studio Autofill tasks were completed successfully with an average TaT of ~40 minutes. ### Root Causes A previously rolled back version of an ML model went back live automatically in the US region. The automated canary reversal did not succeed in that region, leaving the deployment in an inconsistent state. As a result, the model routing labels were not updated as expected during the next forced deployment, and traffic continued to be served by pods running the incorrect model version until we manually rolled back the model in the cluster. This was the result of a bug in our version of Helm. ### Timeline _10:29 UTC: The new model release went back live in the US region only, after the previous automated canary rollback_ _11:44 UTC: We manually re-deployed a previous version via CI/CD pipeline. This was not successful and errors continued._ 1_2:00 UTC: We manually rolled back the version of the model directly in the cluster._ 1_2:11 UTC: All services went back to normal_ ### Remedies * We will fix our deployment tooling to reliably apply model-routing label changes to these resources by upgrading Helm.

1. 21. Januar 2026 um 14:28 UTCresolved

   Between 10:30 UTC and 10:47 UTC there was disruption to document report processing in EU, as a side effect of reports delayed from the earlier incident being re-processed. Further details will follow in a postmortem.

2. 30. Januar 2026 um 11:37 UTCpostmortem

   ### Summary For the EU region, one critical service struggled to reprocess the traffic affected by a previous faulty release which led to higher Turnaround Time \(TaT\) for all Document reports created between 10:32 and 10:50 UTC. All impacted Document Reports were completed successfully with an average TaT of ~6 minutes. ### Root Causes The re-processing batch caused a spike in traffic and the auto-scaling didn’t work as expected for one critical service. The service entered a crash loop state and had to be manually up-scaled to recover. An unbounded number of in-flight requests were accepted leading to memory exhaustion and I/O event loop non-responsiveness, while waiting for a downstream ML inference service to scale up. ### Timeline _10:32 UTC: The critical service went up to a 100% error rate_ _10:33 UTC: Engineers who initiated the report backlog reprocessing become aware of the issue through our monitoring and start investigating_ _10:48 UTC: We manually scaled up the critical service_ 1_0:50 UTC: The service went back to normal and errors stopped_ ### Remedies * Investigate how to reduce memory footprint on this service, allowing bigger request queues while it waits for downstream ML model serving to scale up * Change auto-scaling parameters to be more aggressive \(i.e. scale with lower CPU targets\) * Add concurrent requests monitoring * Reduce ML model serving image sizes for faster scaling of inference services * Improve back-pressure mechanisms to be able to sustain minimum traffic levels independent of spikes while auto-scaling kicks-in * Change our weekly load testing scripts to specifically test for accelerated traffic spikes