Historique des pannes de Onfido

1. 7 septembre 2026 à 15:53 UTCresolved

   Entre 14h30 et 15h30 UTC, les clients de la région CA ont connu un accès dégradé à l'API publique. Un déploiement défectueux dans la couche d'authentification a causé le rejet incorrect des jetons valides, ce qui a entraîné une erreur dans les requêtes d'API. La question a été entièrement réglée. Aucune action n'est nécessaire à votre fin. Nous nous excusons pour la perturbation.

Traduit automatiquement depuis la mise à jour officielle de l'incident.

1. 1 septembre 2026 à 14:37 UTCresolved

   De 13h21 à 13h46 UTC, il y a eu une perturbation à accepter les demandes de documents du SDK Web ainsi que les SDK IDV Entrust pour les flux contenant des modules Web. Onfido Android et iOS SDKs n'ont pas été touchés. Nous nous excusons pour cette perturbation. Un post mortem détaillé suivra une fois notre enquête terminée.

Traduit automatiquement depuis la mise à jour officielle de l'incident.

1. 27 juillet 2026 à 18:12 UTCinvestigating

   Nous enquêtons actuellement sur les retards dans notre grappe américaine pour le traitement des rapports de similitude faciale, de documents et de visages connus.

2. 27 juillet 2026 à 18:23 UTCinvestigating

   Nous voyons des performances dégradées de certains nœuds dans notre grappe de recherche de visages indexés. Nous continuons d'étudier la cause fondamentale et les mesures correctives possibles.

3. 27 juillet 2026 à 18:51 UTCidentified

   Nous avons identifié la cause profonde de la dégradation des performances dans notre grappe de recherche américaine et nous travaillons activement à la résoudre. Nous ferons le suivi une fois entièrement restauré.

4. 27 juillet 2026 à 19:18 UTCmonitoring

   Nous avons mis en place un correctif et continuons de surveiller la qualité du temps de traitement, qui est maintenant amélioré.

5. 27 juillet 2026 à 19:20 UTCmonitoring

   Le temps de traitement continue à s'améliorer, tous les systèmes sont maintenant opérationnels.

6. 27 juillet 2026 à 19:21 UTCresolved

   Les temps de traitement sont de retour à la normale. Cette question est maintenant résolue. Après mortem à suivre bientôt.

Traduit automatiquement depuis la mise à jour officielle de l'incident.

1. 24 juillet 2026 à 04:18 UTCinvestigating

   Identity Enhanced connaît une baisse importante des taux nets pour les candidats dans le monde entier, sauf au Royaume-Uni

2. 24 juillet 2026 à 04:36 UTCidentified

   Nous travaillons avec notre fournisseur tiers pour résoudre le problème et partagerons les mises à jour dès qu'elles seront disponibles.

3. 24 juillet 2026 à 04:55 UTCidentified

   Nous continuons de travailler sur une solution à ce problème.

4. 24 juillet 2026 à 05:09 UTCmonitoring

   Nous constatons une amélioration du taux de récupération du service de notre fournisseur et des taux de réussite des demandes. Nous continuons de suivre de près la situation.

5. 24 juillet 2026 à 07:08 UTCresolved

   Le fournisseur a résolu le problème et le service fonctionne maintenant normalement

Traduit automatiquement depuis la mise à jour officielle de l'incident.

1. 23 juin 2026 à 02:38 UTCinvestigating

   We cannot complete QES tasks because our electronic signature provider is encountering issues.

2. 23 juin 2026 à 02:57 UTCidentified

   Our provider is experiencing issues. All QES tasks cannot complete for now.

3. 23 juin 2026 à 03:20 UTCidentified

   Our provider is still experiencing issues. All QES tasks cannot complete for now.

4. 23 juin 2026 à 03:45 UTCmonitoring

   The issue has been fixed. We're monitoring and resuming blocked QES tasks, if they can be.

5. 23 juin 2026 à 03:49 UTCresolved

   This incident has been resolved.

1. 22 juin 2026 à 17:44 UTCmonitoring

   We're currently monitoring the EU cluster after an Amazon RDS issue.

2. 22 juin 2026 à 17:56 UTCresolved

   We’re seeing recovery across our internal metrics, and processing has now returned to full capacity. At this time, the issue appears to be resolved. Our current leading hypothesis is resource contention on a shared Amazon RDS instance that several services depend on, potentially related to a VACUUM operation running alongside a long-running job deleting a large volume of accumulated historical data. We have not yet confirmed the root cause and will continue investigating as follow-up, but service has been restored for now. We'll be following up with a public post-mortem.

3. 2 juillet 2026 à 14:27 UTCpostmortem

   **Incident date:** 22 June 2026 **Region:** EU \(eu-west-1\) **Affected EU services**: API, Dashboard, Applicant Form, Document Verification, Facial Similarity, Watchlist, Identity Enhanced, Webhooks, Known Faces, Autofill, QES and Device Intelligence. **Customer impact:** ~16:50–17:00 UTC \(acute degradation\); ~17:00–17:20 UTC \(backlog recovery\) ## Summary On 22 June 2026, from approximately 16:50 UTC, a shared database cluster serving our EU region came under severe load and could not reliably serve queries for about 10 minutes. EU services returned elevated errors, and processing throughput briefly fell to ~20–35% of normal levels, with many subcomponents of our system \(e.g., Facial Similarity report processing\) being entirely disrupted, some others less heavily impacted \(e.g., Document report processing\). Service recovered by 17:01 UTC, the database fully stabilizing after an automatic failover \(~17:05–17:07 UTC\). A resultant report backlog was cleared by ~17:20 UTC. Requests in flight during the acute degradation window may have failed unless retried; queued background work was processed automatically once the database recovered. ## Root cause The incident was triggered by a routine database storage-reclamation task following standard scheduled data-deletion processing. This task normally completes without issue; why it failed on this occasion remains under investigation, although we observed that it was processing a larger-than-usual backlog. We have a support case open with our cloud provider to confirm a definitive root cause. The reclamation task began to compete with normal application queries, which slowed as the database struggled to keep up. Applications opened more and more connections, leading to connection saturation and causing queries across the affected services to fail. The database stabilized when an automatic failover to a healthy standby was triggered; the contention fully resolving with the failover to a new instance. ## Timeline \(UTC\) * **16:50** — Our monitoring detected errors and elevated latency across EU services caused by resource contention on a shared database cluster. * **16:53** — We start to see improvements, but system still not acting at normal levels of performance. * **~17:00** — Customer-facing errors subsided and processing resumed as the contention eased. * **17:01** — On-call engineers opened an incident and continued investigations. * **~17:05–17:07** — The database performed an automatic failover to a healthy instance, which reset the overloaded writer and fully stabilised the cluster. The failover was triggered because of resource contention \(out of memory\) caused by the heavy vacuuming in the preceding minutes of the incident. Once the impacting vacuum operations had finished freeing up resources, we had started to see signs of improvement \(16:53—17:01\), but added latency in the feedback loop and aggregation window at AWS still decided to trigger the failover, even though we were already in a recovering state. * **17:15–17:20** — Requests that had queued during the incident were worked through and the backlog returned to normal. * **17:21–17:56** — We monitored the recovery and confirmed processing remained at full capacity. ## Remedies * Reviewing connection limits and pooling so a single service cannot saturate a shared database, and evaluating dedicated database clusters per product to remove cross-service impact. * Changing large historical-data deletions to run in smaller, throttled batches, and tuning database maintenance to avoid large catch-up operations. * Adding earlier, proactive alerting on database memory, connections and load so we can intervene before customer impact. * Continue working with our cloud provider on a definitive root cause.

1. 17 juin 2026 à 04:59 UTCidentified

   One of our providers is still experiencing issues, resulting in higher TaT for electronic signature tasks.

2. 17 juin 2026 à 05:30 UTCidentified

   Our provider is actively working on fixing the issue.

3. 17 juin 2026 à 06:19 UTCidentified

   Our provider is still working on fixing the issue. We're monitoring the impact and we're making sure to keep the TaT as low as possible, considering the situation.

4. 17 juin 2026 à 06:50 UTCidentified

   Our provider's error rate dropped significantly. The TaT of QES tasks is now close to the usual value.

5. 17 juin 2026 à 08:17 UTCidentified

   We're working closely with our provider to find the reason of the remaining errors. The error rate contacting our provider is stable and the impact on the QES task TaT is under control.

6. 17 juin 2026 à 08:48 UTCmonitoring

   Our provider was able to fix the issue and we don't get any error when contacting them. The TaT of QES tasks is back to normal. Monitoring the situation to make sure the errors don't come back.

7. 17 juin 2026 à 09:23 UTCresolved

   The QES tasks TaT is back to normal.

1. 16 juin 2026 à 22:19 UTCidentified

   Our electronic signature tasks are taking more time to complete because one of our providers is doing some maintenance.

2. 16 juin 2026 à 22:52 UTCidentified

   Our provider is still not able to answer to all requests, resulting in longer-than-usual electronic signature task processing time.

3. 16 juin 2026 à 23:17 UTCresolved

   The service is operational again.

1. 13 juillet 2026 à 08:39 UTCresolved

   Incident was solved.

2. 13 juillet 2026 à 08:40 UTCpostmortem

   ## Summary Between May and June 2026, around 0.037% of the evidence files generated on the platform were incomplete and included in the corresponding evidence folders. There was no impact on workflow results, and no incorrect information was ever displayed — the affected files were simply empty rather than wrong. A small number of requests to generate timeline files were also affected \(around 0.073%\). ## Root Causes The issue was introduced during an initiative to improve how timeline and evidence PDFs are generated — making them significantly smaller, more efficient, and more resilient to produce. As part of these changes, in rare cases the new flow attempted to generate the PDF before its content had fully loaded, resulting in a blank or nearly empty file. Because the problem was intermittent and occurred only in rare timing conditions, it was not consistently reproduced during rollout testing. ## Timeline * 15 June 2026: Engineering investigation found an incomplete PDF and started the root cause investigation * 17 June 2026: Root cause investigation completed and a safeguard to prevent incomplete PDFs was deployed ## Remedies * We corrected the PDF generation flow so PDFs are only printed after content has fully loaded, and added a safeguard that stops generation when an incomplete PDF is detected. * We also improved our monitoring and test coverage for PDF generation, so similar issues are caught earlier in the future.

1. 6 juin 2026 à 14:08 UTCinvestigating

   Some Watchlist search search profiles are not working correctly the impacted reports are not being processed. We are investigation the root cause.

2. 6 juin 2026 à 14:36 UTCidentified

   We identified the issue and are implementing a fix.

3. 6 juin 2026 à 15:05 UTCmonitoring

   The issue has been fixed, and the reports are being processed correctly. We are still monitoring the service.

4. 6 juin 2026 à 15:49 UTCresolved

   This incident has been resolved.

1. 12 mai 2026 à 12:25 UTCidentified

   We are facing issues with one of our providers and we see a slight decrease in clear rate for Identity Reports in UK jurisdiction

2. 12 mai 2026 à 12:55 UTCmonitoring

   A fix has been implemented and we are monitoring the results. All clear rates should be back to normal

3. 12 mai 2026 à 13:14 UTCresolved

   All reports are back to normal.

1. 9 mai 2026 à 11:27 UTCidentified

   We cannot complete QES tasks because our electronic signature provider is encountering issues.

2. 9 mai 2026 à 11:27 UTCidentified

   The provider is working on a fix.

3. 9 mai 2026 à 11:32 UTCmonitoring

   Our provider fixed the issue. QES tasks are completing now. Tasks started during the incident were resumed.

4. 9 mai 2026 à 12:03 UTCresolved

   The service is operational, no error where detected after the fix.

1. 5 mai 2026 à 06:52 UTCinvestigating

   We noticed an issue with our QES tasks, we're investigating.

2. 5 mai 2026 à 06:56 UTCidentified

   Our provider is experiencing issues. All QES tasks cannot complete for now.

3. 5 mai 2026 à 07:35 UTCidentified

   Our provider is fixing the issue. QES tasks still cannot complete.

4. 5 mai 2026 à 08:04 UTCmonitoring

   The issue has been fixed. We're monitoring and resuming blocked QES tasks, if they can be.

5. 5 mai 2026 à 08:39 UTCresolved

   QES is operational again.

1. 21 avril 2026 à 14:56 UTCidentified

   We identified an issue with one of our provider and some QES tasks may encounter some problems.

2. 21 avril 2026 à 15:14 UTCidentified

   Our provider is still experiencing issues. All QES tasks cannot complete for now.

3. 21 avril 2026 à 16:12 UTCidentified

   Our provider continues experiencing issues and is working on a fix.

4. 21 avril 2026 à 17:02 UTCidentified

   Our provider is slowly recovering, we expect some QES tasks to complete.

5. 21 avril 2026 à 17:12 UTCresolved

   QES is operational again.

1. 3 février 2026 à 10:42 UTCidentified

   One of our provider is experiencing issues. QES capture tasks will fail and QES verification tasks will have an increase in TaT.

2. 3 février 2026 à 11:13 UTCidentified

   The provider is working on a fix, QES is still suffering a partial outage.

3. 3 février 2026 à 11:44 UTCidentified

   The provider is still working on a fix.

4. 3 février 2026 à 12:16 UTCidentified

   The provider is still working on a fix. QES verification tasks may start to time out, depending on their configuration, as we passed 90 minutes of downtime.

5. 3 février 2026 à 12:46 UTCmonitoring

   The provider fixed the issue. We can confirm our QES tasks are now processed correctly. We'll monitor the situation to make sure the error is indeed fixed.

6. 3 février 2026 à 13:16 UTCresolved

   We can confirm the error stopped and services are stable and operational.

1. 2 février 2026 à 15:36 UTCidentified

   We identified an issue with one of our provider and some QES tasks may encounter some problems.

2. 2 février 2026 à 16:15 UTCidentified

   Our provider is still experiencing issues. All QES tasks cannot complete for now.

3. 2 février 2026 à 16:46 UTCmonitoring

   Our provider is slowly recovering, we expect some QES tasks to complete.

4. 2 février 2026 à 17:18 UTCmonitoring

   We're now seeing the error rate decrease. Most of the QES tasks should complete.

5. 2 février 2026 à 17:24 UTCresolved

   QES is operational again.

1. 28 janvier 2026 à 11:16 UTCinvestigating

   We are currently investigating this issue.

2. 28 janvier 2026 à 11:58 UTCmonitoring

   Issue found and fixed. Increased turn around time for manual reports. Estimated time to live manual processing is 4h.

3. 28 janvier 2026 à 15:22 UTCresolved

   Incident is fully resolved, manual processing is now working normally. Manual reports will keep having an increased turn around time for a few more hours while it works through the task backlog.

4. 2 février 2026 à 16:14 UTCpostmortem

   ### Summary Manual task assignment for all EU customers stopped working between 10h50 UTC and 11:50 UTC. This led to an increase in manual processing Turnaround Time \(TaT\) affecting approximately 20% of our document verification volumes with all customers recovering to TaT SLA by 18h00 UTC. During this period: * All checks that required **manual review** showed an increase in TaT. * **Fully automated reports were not affected** and continued to run as normal. The issue was caused by a **configuration error in our internal task management system**, which prevented it from correctly assigning tasks to our analysts. We fixed the configuration and **restored normal processing** by 28 Jan 2026 11h50 AM UTC, and cleared all manual task backlogs by 18h00 UTC. We have updated our validation and deployment checks to prevent similar issues in the future. ### Root Causes _Manual processing queue assignment was affected by an invalid manual configuration input. This single queue configuration parameter resulted in an error that affected assignments in all queues._ ### Timeline * all times in UTC: _10:50: Configuration manually updated and errors started, no more tasks assigned._ _11:02: On-call is notified of a spike in manual system assignment errors through our monitoring_ _11:07: The error responsible for the spike is identified \(invalid UUID\)_ _11:10: Incident declared_ _11:45: Origin of invalid UUID is found_ _11:50: Bad configuration parameter is deleted_ _11:56: Configuration reintroduced correctly_ _18:00: Recovered from manual task backlog_ ### Remedies * Adding appropriate input configuration value validation * Improve task assignment resilience to these types of errors * Review configuration guidance and post-release monitoring

1. 26 janvier 2026 à 12:46 UTCresolved

   Between 12:15 UTC and 12:40 UTC there was disruption to document report processing in EU. Autofill requests also saw a disruption between 12:16 UTC and 12:27 UTC. A postmortem will follow.

2. 30 janvier 2026 à 11:40 UTCpostmortem

   ## Summary On 26 January 2026 between 12.16 UTC and 12.35 UTC, our document reports processing was severely degraded with customers experiencing extended processing time to about 85% of their traffic. The incident also caused extended processing time on Biometric Authentications and Biometric Verifications for 20% of the traffic between 12.25 UTC and 12.35 UTC. ## Root Causes A core service for fraud prevention on documents came under elevated load, causing kubernetes pods to go down in quick sequence. Our upstream service retry policy proved too aggressive to let the service recover and required manual scaling-up. The retry policy also caused elevated load on a shared database which in turn also affected the biometrics service. ## Timeline * **12:16 UTC** – Our monitoring detected a sharp increase in errors when processing document reports. * **12:19** **UTC** – The on‑call team was alerted and began investigating the affected document‑processing service. * **12:25 UTC** – We identified that the incident was also affecting a small portion of biometric checks, leading to some failures and short delays. * **12:29** **UTC** – On-call engineers manually increased capacity for the impacted document fraud‑prevention service. * **12:35** **UTC** – Error rates for both document reports and biometric checks returned to normal and new requests were being processed successfully. * **12:45–13:06** **UTC** – We processed document reports that were impacted during the incident window. * **13:15** **UTC** – We confirmed that all affected document and biometric requests had completed successfully and marked the incident as resolved. ## Remedies * We have continued to fine-tune our auto-scaling parameters of the affected service to scale up with lower CPU targets. * We modified the retry policy of fraud services to avoid overloading an already struggling service so that it can auto-recover. * We made our load-tests on this service more representative of production traffic \(similar image sizes/document type distribution…\) and test for accelerated traffic spikes. * We lowered the total amount of shared database connections the document processing can take to avoid noisy neighbour impact on biometrics processing.

1. 21 janvier 2026 à 11:05 UTCinvestigating

   We have a high error rate in document report processing and autofill. Our team is investigating

2. 21 janvier 2026 à 11:49 UTCidentified

   We have identified the issue as a configuration issue between services, and are working on a fix.

3. 21 janvier 2026 à 12:07 UTCmonitoring

   We have deployed a fix. New document reports and autofill requests are being processed normally. We will rerun the earlier impacted reports.

4. 21 janvier 2026 à 12:34 UTCresolved

   This issue is now resolved: Between 10:30 and 12:00 UTC, document reports could not be processed and autofill requests were failing in the US region. Starting from 12:00, live traffic was unaffected. From 12:00 to 12:30 UTC, pending reports were processed. We take a lot of pride in running a robust, reliable service and we're working hard to make sure this does not happen again. A detailed postmortem will follow once we've concluded our investigation.

5. 30 janvier 2026 à 11:39 UTCpostmortem

   ### Summary Around 10:29 UTC, around 10 minutes after restoring service in the EU and CA from the 10:05-10:17 UTC outage, we go alerted for high error rate in the US region and we noticed the new ML model version was back online in that region. This impacted Turnaround Time \(TaT\) for all Document Report and Studio Autofill until 12:11 UTC. Autofill Classic had an average 100% error rate during the same time frame. All impacted Document Reports and Studio Autofill tasks were completed successfully with an average TaT of ~40 minutes. ### Root Causes A previously rolled back version of an ML model went back live automatically in the US region. The automated canary reversal did not succeed in that region, leaving the deployment in an inconsistent state. As a result, the model routing labels were not updated as expected during the next forced deployment, and traffic continued to be served by pods running the incorrect model version until we manually rolled back the model in the cluster. This was the result of a bug in our version of Helm. ### Timeline _10:29 UTC: The new model release went back live in the US region only, after the previous automated canary rollback_ _11:44 UTC: We manually re-deployed a previous version via CI/CD pipeline. This was not successful and errors continued._ 1_2:00 UTC: We manually rolled back the version of the model directly in the cluster._ 1_2:11 UTC: All services went back to normal_ ### Remedies * We will fix our deployment tooling to reliably apply model-routing label changes to these resources by upgrading Helm.

1. 21 janvier 2026 à 14:28 UTCresolved

   Between 10:30 UTC and 10:47 UTC there was disruption to document report processing in EU, as a side effect of reports delayed from the earlier incident being re-processed. Further details will follow in a postmortem.

2. 30 janvier 2026 à 11:37 UTCpostmortem

   ### Summary For the EU region, one critical service struggled to reprocess the traffic affected by a previous faulty release which led to higher Turnaround Time \(TaT\) for all Document reports created between 10:32 and 10:50 UTC. All impacted Document Reports were completed successfully with an average TaT of ~6 minutes. ### Root Causes The re-processing batch caused a spike in traffic and the auto-scaling didn’t work as expected for one critical service. The service entered a crash loop state and had to be manually up-scaled to recover. An unbounded number of in-flight requests were accepted leading to memory exhaustion and I/O event loop non-responsiveness, while waiting for a downstream ML inference service to scale up. ### Timeline _10:32 UTC: The critical service went up to a 100% error rate_ _10:33 UTC: Engineers who initiated the report backlog reprocessing become aware of the issue through our monitoring and start investigating_ _10:48 UTC: We manually scaled up the critical service_ 1_0:50 UTC: The service went back to normal and errors stopped_ ### Remedies * Investigate how to reduce memory footprint on this service, allowing bigger request queues while it waits for downstream ML model serving to scale up * Change auto-scaling parameters to be more aggressive \(i.e. scale with lower CPU targets\) * Add concurrent requests monitoring * Reduce ML model serving image sizes for faster scaling of inference services * Improve back-pressure mechanisms to be able to sustain minimum traffic levels independent of spikes while auto-scaling kicks-in * Change our weekly load testing scripts to specifically test for accelerated traffic spikes