Socure 장애 기록

We are currently experiencing 5xx errors affecting multiple modules. Our team is actively investigating the issue and will provide updates as more information becomes available.

Socure services have now fully recovered and are operating normally. Our engineering and operations teams will continue to work to conduct a comprehensive root cause analysis (RCA) of the incident. We will share the RCA publicly within the next few days once this review is complete.

Between Thursday, June 25, 2026, at 2:06 PM EDT and Friday, June 26, 2026, at 6:03 AM EDT, customers subscribed to webhook events experienced delayed webhook deliveries due to an issue affecting our webhook processing pipeline. During this period, approximately half of the webhook deliveries were delayed. No webhook events were lost. The processing backlog did not surface within the webhook workers until 5:06 AM EDT on June 26. Once it became visible, the queued webhook events were processed rapidly without manual intervention, and webhook delivery latency returned to normal by 6:03 AM EDT. We are continuing to investigate why a subset of webhook events became blocked from processing and why the backlog later surfaced without intervention. We will publish a detailed root cause analysis (RCA), including corrective actions, once our investigation is complete.

## **Root Cause Analysis** **Incident:** RiskOS Delayed Webhook Deliveries **Date Range:** June 25, 2026 2:06 PM ET to June 26, 2026 6:03 AM ET **Impact:** Customers subscribed to webhook events experienced delayed webhook deliveries during the incident window. Approximately half of webhook events were delayed; no webhook events were lost. The webhook processing pipeline auto recovered without manual intervention, and no customer action was required. **Status:** Resolved **1.Summary** Between approximately 2:06 PM ET on June 25, 2026 and 6:03 AM ET on June 26, 2026, customers subscribed to RiskOS webhook events experienced delayed webhook deliveries due to an unexpected degradation in the webhook processing pipeline. During the impact window, approximately half of webhook events were delayed. No webhook events were lost. The processing backlog did not surface within the webhook workers until 5:06 AM ET on June26. Once it became available, the queued webhook events were processed, and webhook delivery latency returned to normal at approximately 6:03 AM ET. The pipeline auto-recovered without any manual intervention. Subsequent log and metric analysis identified an issue in the upstream message consumption layer as the primary contributor: following an infrastructure scale-up, the Kafka consumer group that feeds the webhook processing pipeline did not rebalance correctly, leaving a subset of partitions effectively unassigned for consumption. As a result, webhook events landing on those partitions were not picked up by the webhook workers in real time. The condition self-resolved when consumer group membership stabilized, at which point the queued events were processed quickly and deliveries returned to normal. Standard Kafka topic-lag monitoring did not surface the condition in a timely manner because aggregate topic lag remained within thresholds while the affected partitions accumulated work in the background. Socure engineering has identified a set of corrective and preventive actions — described below — to harden the webhook processing pipeline against partition rebalance failures and to improve observability of partition-level consumption health. As an immediate preventive action, an additional alert that detects this class of failure has already been deployed. **2. Timeline** | **Time \(ET\)** | **Event** | | --- | --- | | **Thu, Jun 25 — 2:06 PM**  | Webhook delivery for a subset of events starts getting impacted. This was following an infrastructure scale-up performed, the Kafka consumer group that feeds the webhook processing pipeline did not rebalance correctly, leaving a subset of partitions effectively unassigned for consumption. Webhook events on those partitions begin to accumulate without being delivered. Standard Kafka topic-lag monitoring does not surface because aggregate topic lag remains within thresholds. | | **Thu, Jun 25 2:06 PM – Fri, Jun 26 – 5:06 AM** | Webhook delivery latency remains elevated for approximately half of webhook events. Events on unaffected partitions continue to be delivered normally. No webhook events are lost. | | **Fri, Jun 26 — 5:06 AM**  | The processing backlog surfaces within the webhook workers. Consumer group membership stabilizes, the affected partitions begin to be consumed, and the webhook workers report a steep increase in lag as they begin processing the queued events. | | **Fri, Jun 26 — 5:06 AM –6:03 AM** | Queued webhook events are processed rapidly. Webhook delivery latency returns to normal at approximately 6:03 AM ET. No manual mitigation is required. | | **Fri, Jun 26 — 10:00 AM** | On-call engineering reviews telemetry from the overnight processing spike and correlates it with customer reports of delayed webhook deliveries. | | **Fri, Jun 26 — 11:57 AM**  | Incident formally declared in Socure’s incident management system as a retrospective incident. Incident response team assembled and investigation bridge convened. Telemetry for the webhook processing pipeline reviewed in parallel. | | **Post Incident**  | Detailed log and metric analysis of the webhook processing pipeline. Primary contributor identified as a Confluent Kafka | **3. Root Cause** **Primary Root Cause:** Following an infrastructure scale-up performed Thu, Jun 25 02:06 PM, the Kafka consumer group that feeds Socure’s webhook processing pipeline did not rebalance correctly. As a result, a subset of Kafka partitions was effectively unassigned for consumption, and webhook events landing on those partitions were not picked up by the webhook workers in real time. The condition was transient: when consumer group membership stabilized at approximately 5:06 AM ET on June 26, the affected partitions resumed consumption, the queued events were processed rapidly, and webhook delivery latency returned to normal at approximately 6:03 AM ET. No webhook events were lost. **Contributing Factors:** * **Consumer group rebalance failure:** The configured Kafka consumer rebalance protocol did not produce a complete partition assignment for the consumer group following the scale-up.Investigation revealed an edge-case consumer group rebalance failure during the infrastructure scale-up. We have since implemented application-level validation to actively monitor partition health.. * **Observability gap on partition-level consumption:** Existing Kafka topic-lag monitoring did not surface the condition in a timely manner. Topic-aggregate lag remained within thresholds during the impact window because lag on actively consumed partitions stayed at zero, masking the accumulating lag on the affected partitions until consumption resumed and the workers reported a step increase. **Non-contributing Factors:** * **Webhook worker fleet capacity:** The webhook worker fleet was operating within nominal capacity throughout the incident window. No worker-level resource saturation, restart, or scaling failure was associated with the incident. * **Customer-side delivery endpoints:** Customer-owned webhook endpoints were not associated with the incident. Once the backlog began processing, deliveries to customer endpoints completed at expected rates. * **Recent application code changes:** No Socure webhook application code release during the incident window was associated with the incident. **4. Resolution** **Service restoration:** The condition was transient and self-resolved when the affected Confluent Kafka consumer group’s partition assignment stabilized at approximately 5:06 AM ET on June 26, 2026. At that point, the previously unassigned partitions resumed consumption, and the webhook workers processed the queued events rapidly. Webhook delivery latency returned to normal at approximately 6:03 AM ET. No manual mitigation was required. Socure engineering verified recovery through telemetry and continued monitoring through and beyond the incident window. **Validation and verification:** A post-incident review was conducted to confirm the scope and duration of impact. Detailed log and metric analysis identified the Confluent Kafka consumer group rebalance failure as the primary contributor, occurring in the context of an infrastructure scale-up performed the prior day. As an immediate preventive action, Socure engineering deployed an additional alert that monitors divergence between message production rate and message polling \(consumption\) rate over a time period — designed specifically to detect scenarios where partitions are not being consumed even when topic-aggregate lag does not surface the delay. Additionally, we have now added a safety net for watching partition updates for ensuring the partition assignments. **5. Corrective and Preventive Actions** | **Action** | **Description** | **ETA / Status** | | --- | --- | --- | | **Partition-level consumption monitoring** | Deploy an additional alert that monitors divergence between message production rate and message polling \(consumption\) rate over a time period, complementing existing Kafka topic-lag monitoring to detect partition-level consumption gaps that aggregate lag alone does not surface. | **Completed** | | **Consumer group handling partitional updates** | We have implemented application-level event listeners to dynamically monitor Kafka partition state changes. In the event of a recurring Kafka broker or partition failure, the application will now automatically detect the disruption and execute a self-healing recovery sequence. | **June 26th, 2026** | | **Scale-up runbook hardening** | Update the infrastructure scale-up runbook for the webhook processing pipeline to include explicit verification of Kafka consumer group rebalance completion and partition assignment health before the scale-up is declared complete. | **mid-Q3, 2026** | **6. Lessons Learned** This incident exposed two specific gaps in the webhook processing pipeline. First, infrastructure scale-up operations can leave the Confluent Kafka consumer group in an incomplete rebalance state, with a subset of partitions effectively unassigned, without that condition surfacing to operators in real time. Second, Kafka topic-aggregate lag is not a sufficient indicator of partition-level consumption health when only a subset of partitions is affected, because lag on actively consumed partitions can mask accumulating lag on unassigned ones. Follow-up work is focused on detecting and preventing partition-level consumption gaps, hardening the scale-up runbook for the webhook processing pipeline, and strengthening observability so that rebalance anomalies are surfaced as they occur rather than after they have already produced customer-visible impact. **7. Next Steps & Ongoing Commitment** Socure takes accountability for this incident and the impact it had on the affected customers. We remain committed to improving the resilience and maintaining a reliable experience for our customers. The corrective actions underway focus on three areas: strengthening partition-level consumption monitoring for the webhook processing pipeline so that rebalance-related consumption gaps are detected as they occur, identifying and remediating the underlying cause of the Confluent Kafka consumer group rebalance failure, and hardening the infrastructure scale-up runbook to verify consumer group health and partition assignment before any scale-upis declared complete. We remain committed to improving the resilience and maintaining a reliable experience for our customers.

We are currently experiencing errors affecting our RiskOS APIs. Our team is actively investigating the issue and will provide updates as more information becomes available.

Our team has mitigated the issue that was causing errors with the RiskOS APIs. We are seeing error rates return to normal levels and are continuing to monitor the system closely. We will provide additional updates if there are any further changes.

We are continuing to monitor our services and are not observing any errors at this time. We will provide additional updates if there are any further changes.

The issue impacting RiskOS has been resolved, and service has returned to normal operation. Customers should no longer experience errors, degraded performance, or delays when accessing RiskOS services and workflows. Our teams have confirmed system stability and will continue to monitor performance closely. We appreciate your patience throughout this incident and apologize for the impact this outage may have had on your operations.

**Root Cause Analysis** **Incident:** RiskOS Evaluation API Elevated 502 Errors **Date Range:** June 22, 2026, 10:23 PM EDT – 10:59 PM EDT **Impact:** Customers calling the Socure RiskOS Evaluation APIs in production experienced an elevated rate of HTTP 502 \(Bad Gateway\) responses for approximately 36 minutes. **Status:** Resolved ## **1. Summary** On June 22, 2026, between approximately 10:23 PM and 10:59 PM EDT, Socure’s RiskOS Evaluation APIs in the production environment experienced an elevated rate of HTTP 502 errors. The disruption was caused by a misconfiguration in the aggregations engine — a component that enriches Evaluation API processing by computing velocity based signals across entity data. The misconfiguration was introduced during a configuration update as part of Local Graph GA readiness, and was not caught by existing validation controls. When the service attempted to use this configuration during evaluation processing an unexpected runtime exception occurred that triggered automated service restarts. As the affected services cycled, backend capacity became intermittently unavailable, causing requests during those periods to receive HTTP 502 \(Bad Gateway\) responses. Socure's automated monitoring detected the elevated error rate at **10:24 PM EDT** and the Engineering team began investigation immediately. The team mitigated the issue by disabling aggregations processing to stabilize traffic, remediated the invalid configuration, and validated recovery before re-enabling aggregations. Evaluation APIs returned to normal operation by **10:59 PM EDT**. Aggregations were re-enabled after validation at approximately **11:13 PM EDT**. Requests that received HTTP 502 responses during the incident window were not processed and no corresponding evaluation was created. Customers who experienced HTTP 502 responses between 10:23 PM and 10:59 PM EDT on June 22, 2026 should verify any intended evaluations and resubmit them if necessary. ## **2. Timeline** | **June 22, 2026 / Time** | **Event** | | --- | --- | | **10:22 PM EDT** | A configuration change in Entity Aggregations introduced an invalid value. | | **10:23 PM EDT** | Elevated HTTP 502 error rates begin on RiskOS Evaluation APIs in the production environment. | | **10:24 PM EDT** | Automated monitoring detected the elevated error rate and the on-call Engineering team was notified. Investigation began immediately. | | **10:25 PM to 10:40 PM EDT** | Engineering observed multiple servers failing intermittently. The team reviewed logs and traced the failures across affected servers to isolate the root cause. | | **~10:40 PM EDT** | Engineering identified that the issue was related to an aggregations configuration affecting Evaluation API processing. | | **~10:45 PM EDT** | Engineering began targeted mitigation, initially isolating specific aggregations-dependent components. Instability persisted, and the team proceeded to disable aggregations processing entirely to stabilize traffic. | | **10:55 PM EDT** | Aggregations processing was fully disabled. Evaluation API traffic stabilized. | | **10:59 PM EDT** | Evaluation APIs returned to normal operation. Elevated 502 errors were resolved.  | | **10:59 PM to 11:13 PM EDT** | Engineering identified and remediated the specific invalid aggregation configuration. Recovery was validated against the corrected configuration. | | **11:13 PM EDT** | Aggregations re-enabled following successful validation of the corrected configuration. | ## **3. Root Cause** #### **Primary Root Cause:** An invalid configuration was applied to the aggregations engine. When the service attempted to use this configuration during Evaluation processing, it triggered an unhandled failure path inside the processing pipeline. Rather than gracefully rejecting the invalid configuration, the failure path caused service-level crashes and the affected servers entered a restart cycle. While the affected services were cycling, the RiskOS Evaluation APIs' available capacity to serve traffic dropped, and a portion of incoming requests reached the gateway without a healthy backend service available to respond. The gateway returned HTTP 502 \(Bad Gateway\) responses for those requests. #### **Contributing Factors:** * **Insufficient configuration validation:** The server did not gracefully reject the invalid Aggregation configuration at the load time. Instead of failing safely, the invalid spec propagated into a runtime failure path that crashed the affected pod, producing the customer-visible 502 responses. * **Shared processing impact:** Because the issue affected shared Evaluation API processing, a configuration-specific issue resulted in elevated API failures for customers during the incident window. #### **Non-contributing Factors:** * **Infrastructure health:** No underlying cluster, network, storage, or platform infrastructure issue contributed to the incident. Compute, network, and storage layers supporting RiskOS remained healthy throughout the incident window. Once the offending configuration was mitigated, the backend service stabilized with Evaluation processing restored back to normal. ## **4. Resolution** Socure restored service by disabling aggregations processing entirely, which stabilized Evaluation API traffic and returned normal operation by 10:59 PM EDT. The team then identified and removed the specific invalid aggregation configuration, validated the correction, and re-enabled aggregations at 11:13 PM EDT. Runtime safeguards have been implemented to handle invalid aggregation configurations gracefully without affecting service availability. Pre-activation validation checks currently in development will prevent invalid configurations from reaching production. ## **5. Corrective and Preventive Actions** | **Action** | **Description** | **ETA / Status** | | --- | --- | --- | | **Runtime safeguards for Aggregations Configurations** | Updated Evaluation API processing to handle incomplete or missing aggregation configuration values gracefully across all similar configuration processing paths, preventing service crashes. The specific condition that triggered this incident is now handled safely without impacting service stability. | **Completed** | | **Aggregations configuration validation** | Added stricter validation so invalid, incomplete, or unsupported configurations - are rejected before they can become active in production. | **In Progress** | | **Pre-activation configuration checks** | Added checks before aggregation configurations are used in Evaluation API processing to identify invalid or conflicting states earlier in the pipeline, before they can impact live traffic. | **In Progress** | | **Blast radius control**  | Strengthen isolation so an issue in a specific workflow path does not affect broader shared Evaluation API processing. | **Q3, 2026** | ## **6. Lessons Learned** This incident highlighted a specific gap between Aggregations configuration validation and Evaluation API runtime handling. Follow-up work is focused on preventing invalid configurations from becoming active, improving runtime safeguards, and strengthening isolation for workflow-specific issues. ## **7. Next Steps & Ongoing Commitment** Socure recognizes that customers rely on RiskOS Evaluation APIs for reliable, real-time decisioning. We take ownership of the disruption caused by this incident and are implementing changes to reduce the likelihood and impact of similar issues. The corrective actions underway focus on three areas: strengthening aggregations configuration validation, improving Evaluation API runtime safeguards, and limiting the blast radius of any future configuration-related issues.  We remain committed to improving the resilience and maintaining a reliable experience for our customers.

We are investigating an issue where some users may be unable to generate Document Request links from the dashboard. Affected users may see the link generation process continue loading instead of returning a generated document request link. Initial investigation indicates this may be related to role provisioning or permission configuration impacting a subset of accounts. Our team is working to confirm the required permissions and restore functionality for impacted users. We will provide another update as soon as more information is available.

We have identified the issue causing Document Request link generation to remain in a loading state for some users. Our investigation determined that the degradation was caused by a recent change that introduced an additional permission requirement for this action. We are currently reverting the related change to restore normal functionality. We expect recovery within approximately 15 minutes and will provide another update once the functionality has been fully restored.

We are continuing to work on a fix for the issue affecting Document Request link generation. The revert process is taking longer than initially anticipated. Our team remains actively engaged and is working through the revert. Once it is complete, we will immediately begin validation to confirm that functionality has been fully restored. We apologize for the continued inconvenience and appreciate your patience while we work toward resolution. We will provide another update as soon as recovery has been confirmed.

The issue affecting Document Request link generation has been resolved, and functionality has been fully restored. Our engineering team has completed the necessary changes and validated the fix. Users should now be able to generate Document Request links normally. We apologize for the inconvenience and appreciate your patience while we worked to resolve the issue. We are currently completing our post-incident review and will publish a Root Cause Analysis (RCA) within the next few days once that review is complete.

**Root Cause Analysis** ### Incident: Unable to generate DocV links via dashboard **Date Range:** Tue, Jun 16, 2026 8:52 am – 10:12 am ET **Impact:** Document Verification link generation via Admin Dashboard **Status:** Resolved ### **1. Summary** On Jun 16, 2026, a scheduled deployment introducing stricterOn June 16, 2026, a scheduled deployment introducing stricter access control logic inadvertently broke the Document Verification \(DocV\) link generation workflow for a subset of client configurations. The issue persisted for approximately 1 hour and 20 minutes after being reported by a client. The incident was mitigated via a full rollback. We are actively overhauling our release validation protocols and implementing shadow-mode deployment capabilities to ensure enterprise permission configurations are thoroughly tested before enforcing hard access restrictions in the future. ### **2. Timeline** | **Time \(ET\)**  | **Event** | | --- | --- | | **June 16, 5:16 AM ET**  | A production deployment was completed, introducing updates to the system's permission logic. This logic contained a defect rendering the DocV request links feature unusable. | | **8:52 am ET** | Clients started reporting Admin Dashboard DocV link generation stuck in loading for some users. | | **9:04 am–9:24 am ET**  | Upon determining that the problem stemmed from recent updates to role and permission provisioning within the admin dashboard, the team resolved to execute a revert. | | **10:00 am–10:08 am ET**  | The rollback was verified in a staging environment before being deployed to production, followed by thorough regression and sanity testing. | | **10:10 am ET**  | The client confirmed that the Document Verification link generation workflow was fully restored. | | **10:11 am ET** | The incident was resolved. | ### **3. Root Cause** * **Primary Root Cause:** * Stricter permission validation logic was deployed in the morning that was meant to make our permission model more consistent across pages. However, this update caused unintended disruptions for specific role setups, preventing authorized users from accessing the document request link generation process. ### **4. Resolution** * To restore functionality, the team implemented a rollback of the stricter permission enforcement logic. Following the deployment, we performed validation of the admin dashboard's link generation flow and verified that the service was operating correctly. ### **5. Corrective and Preventive Actions** | **Action**  | **Description** | **ETA / Status** | | --- | --- | --- | | Revisit stricter access control with a safer rollout approach | Reintroduce the permission enforcement in shadow mode first using a safer rollout approach \(e.g., staged rollout and additional validation\) to prevent unintended impact to existing customer role configurations. Communicate with impacted customers to guide them in updating their permission settings | **07/03** | | Add monitoring for errors/issues in document link generation | Add monitoring for errors in document link generation to be notified of and address issues in the future proactively. | **07/03** | | Expand validation of access control changes to cover all user configurations | Update our release validation process so access-control and permission changes are tested against a broader range of role setups \(including environment-scoped scenarios\) prior to production release. | **07/03** | ### **6. Lessons Learned** * Permission/environment enforcement needs testing across representative customer role setups. Dark-launch/shadow-mode logging to measure who would be blocked, communicating to customers, and establishing a rollout plan is a safer path than hard enforcement.  ### **7. Next Steps & Ongoing Commitment** We are committed to _Corrective and Preventive Actions_ noted above.

On June 16, 2026, from 03:45 AM to 04:03 AM EDT, RiskOS APIs experienced elevated error rates. The issue is fully resolved and all systems are now operating normally. The degradation occurred during remediation work tied to an networking error alert. A subset of API requests returned 5XX errors during this window, though the service remained partially available throughout. No customer action is required. A full Root Cause Analysis (RCA) will be shared once the investigation concludes.

**Root Cause Analysis** **Incident:** RiskOS API Services Degradation During NAT Gateway Remediation **Date Range:** 06-16-2026 **Impact:** Intermittent degradation for a subset of API requests **Status:** Recovered ## **1. Summary** On June 16th, 2026, RiskOS API services experienced intermittent degradation for a subset of API requests between **3:46 AM EDT and 4:02 AM EDT**. During this period, some requests failed while network connectivity was being transitioned as part of a controlled remediation. Earlier that day, Socure observed elevated connection error alerts and identified critically high Cloud NAT port utilization in the RiskOS commercial environment. The condition was not impacting existing customer transactions at the time of detection, but it created a significant capacity risk for outbound connectivity if additional workloads restarted, scaled, or deployed. To reduce the risk of broader impact, Socure performed a controlled network remediation during a lower-workload window. The remediation improved capacity handling and reduced shared NAT resource contention across origin IP blocks. During the transition, some existing connections were briefly interrupted and had to be re-established, resulting in intermittent request failures. The remediation activity completed in approximately 5 minutes, and overall recovery and stabilization completed within approximately 15 minutes. No customer action was required after connectivity was restored. ## **2. Timeline** | **June 16th,** **2026 / Time** | **Event** | | --- | --- | | 12:00 AM EDT | The Socure Engineering team observed elevated connection error alerts and began an investigation. No customer transaction impact was identified at this time. | | 12:30 AM EDT | Investigation identified critically high Cloud NAT port utilization, creating a capacity risk for outbound connectivity. SRE confirmed that existing customer transactions were not being impacted at that time. | | 12:30–3:45 AM EDT | The Socure Engineering team evaluated remediation options, assessed the risk of leaving the configuration unchanged, and planned remediation during a lower-workload window to reduce potential customer impact. | | 3:45 AM EDT | The engineering team proceeded with the planned network capacity remediation. | | 3:46 AM EDT | Intermittent request failures began as some existing network connections were briefly interrupted during the transition to the updated configuration. | | 3:52–4:00 AM EDT | Network configuration updates were applied to improve capacity handling and reduce shared NAT resource contention across origin IP blocks. | | 4:02 AM EDT | Intermittent request failures ended after traffic completed transition to the updated configuration and connections were re-established. | | ~4:02 AM EDT | Platform connectivity stabilized. | | ~4:15 AM EDT | Incident considered resolved after validation and monitoring confirmed recovery. | ## **3. Root Cause** **Primary Root Cause:**The incident was caused by critically high Cloud NAT port utilization in the RiskOS commercial environment. The NAT configuration had insufficient available port capacity to provide safe headroom for normal connection churn, workload restarts, scaling activity, or deployments. **Contributing Factors:** * The residual NAT capacity risk was a consequence of emergency recovery activities undertaken after the June 9 RiskOS service outage, where the primary focus was restoring service availability. * NAT port utilization reached approximately 99–100%, creating a significant risk of outbound connectivity degradation. * NAT capacity was shared across origin IP blocks, which increased the likelihood of port exhaustion under workload pressure. * During remediation, some existing connections were reset while traffic transitioned to the updated network configuration, resulting in temporary intermittent request failures. ## **4. Resolution** Socure updated the NAT configuration to improve capacity handling and reduce shared-resource contention across origin IP blocks. The remediation was performed because the existing NAT utilization level presented a significant reliability risk, and delaying the change could have increased the chance of broader service degradation. During the transition, some existing connections were reset and had to be re-established. Once the updated configuration took effect, intermittent request failures stopped and platform connectivity stabilized. No customer action was required after connectivity was restored. Requests retried after recovery were expected to process normally. ## **5. Corrective and Preventive Actions** | **Action** | **Description** | **ETA / Status** | | --- | --- | --- | | Reduce NAT capacity contention | NAT usage was separated to reduce shared capacity risk and limit the blast radius of future NAT capacity issues. | Completed | | Enable dynamic port allocation | Dynamic port allocation was enabled to improve NAT port capacity management and reduce exhaustion risk. | Completed | | Improve NAT utilization monitoring | Add or tune monitoring for NAT port utilization, dropped packets, and connection errors with actionable alert thresholds before utilization reaches critical levels. | Completed | | Move network configuration to Infrastructure as Code | Continue moving RiskOS network configuration into version-controlled Infrastructure as Code to improve reviewability, repeatability, and recovery as a part of AWS Migration. | Planned | ## **6. Lessons Learned** * NAT port utilization should be treated as a critical reliability signal for workloads that depend on outbound connectivity. * NAT capacity should have sufficient headroom for normal connection churn, workload restarts, scaling events, and deployments. * Remediation of high-risk network conditions can still create temporary impact when existing connections are reset, so change planning should include clear validation and communication steps. * Network changes should include pre-change checks for current utilization, available NAT capacity, expected connection reset behavior, and rollback options. * Continued investment in environment isolation, Infrastructure as Code, and stronger monitoring will reduce the likelihood and impact of similar incidents. ## **7. Next Steps & Ongoing Commitment** Socure recognizes that customers rely on RiskOS to be available and reliable. We take accountability for the disruption caused during this remediation and are continuing to improve our network architecture and operational controls. To reduce the likelihood of recurrence, Socure is pursuing the following: 1. **NAT Capacity Controls:** We are improving monitoring and alerting for NAT utilization, connection drops, and port exhaustion risk so that capacity issues can be identified and remediated before they impact customer traffic. 2. **Infrastructure as Code:** We are continuing to move RiskOS network configuration to a version-controlled model so that every change is reviewed, repeatable, and can be recovered quickly. 3. **Environment Isolation:** We are separating network resources and reducing shared capacity dependencies so that the impact of any capacity issue or network change is contained. 4. **Safer Network Operations:** We are strengthening pre-change validation, rollback planning, and maintenance window selection for network changes that may affect customer traffic. 5. **Operational Runbooks:** We are updating operational runbooks for NAT capacity exhaustion and Cloud NAT changes so that response and recovery steps are repeatable and clearly documented.

On June 15, 2026, between 12:15 AM and 1:00 AM EDT, the OTP Verification endpoint experienced elevated error rates. The system recovered automatically at 1:00 AM EDT. No action is required from customers. Our investigation confirmed the issue was transient. All systems are operating normally. A Root Cause Analysis (RCA) will be published once our investigation is complete.

## **Root Cause Analysis \(RCA\)** **Incident:** Elevated 5XX errors on the OTP Verification endpoint **Date Range:** June 15, 2026 — 12:15 AM ET to 1:00 AM ET **Impact:** Customers calling the Socure OTP Verification endpoint \(otp/send\) on the ID\+ production platform or RiskOS workflows experienced HTTP 5XX errors during the incident window, resulting in failed OTP verifications. The endpoint auto-recovered without manual intervention, and no customer action was required. **Status:** Resolved ### **1. Summary** On June 15, 2026, between approximately 12:15 AM and 1:00 AM ET, Socure’s OTP Verification endpoint \(otp/send\) on the ID\+ production platform experienced an elevated rate of HTTP 5XX errors. OTP requests failed or were degraded during this 45-minute window. The endpoint auto-recovered at 1:00 AM ET with no manual intervention required. A concentrated surge in OTP verification volume placed sustained load on Socure’s OTP service, which coordinates OTP generation, delivery, and verification. The service is configured to scale horizontally, but the additional replicas did not successfully come up during the surge window. The existing instances became saturated and restarted repeatedly under load, and resulted in HTTP 5XX errors for the OTP Verification endpoint during this period. Initial investigation considered upstream messaging-provider behavior as a potential contributor; subsequent log and metric analysis confirmed that vendor-side rate limiting was not a material factor during the window, and identified the internal OTP service scaling failure as the primary cause. The service self-recovered as upstream demand returned to baseline. Socure engineering has identified a set of corrective and preventive actions — described below — to address the underlying scaling behavior, strengthen capacity baselines for elevated traffic events, and improve observability around scaling-related saturation conditions. ### **2. Timeline** | Time \(ET\) | Event | | --- | --- | | 12:15 AM | Elevated HTTP 5XX error rates begin on the OTP Verification endpoint \(otp/send\) in the ID\+ production environment. | | 12:17 AM – 12:28 AM | Initial high-volume burst observed on the OTP Verification endpoint; OTP verification request volume rose sharply, placing concentrated load on the MFA Orchestrator service that coordinates OTP delivery and verification. | | 12:41 AM | Incident formally declared and opened within Socure’s incident management system. On-call engineering paged; incident response team assembled. An investigation bridge was convened and telemetry for the OTP service was reviewed in parallel. | | ~12:45 AM | Initial investigation considered upstream messaging-provider behavior as a potential contributor; no broad vendor outage reported on the public status page. Investigation continued in parallel into the health of Socure-owned services in the OTP path. | | 1:00 AM | Upstream OTP verification request volume subsided; service load on the OTP service returned to capacity, and error rates on the OTP Verification endpoint auto-recovered to normal levels. No manual mitigation was required. | | Post Incident | Incident moved to post-incident review; engineering continued investigation and impact assessment to confirm scope of affected accounts and validate the upstream cause. | | June 16 | Detailed log and metric analysis completed. Primary contributor identified as a horizontal scaling failure in Socure’s OTP service: the configured autoscaling did not produce additional replicas during the surge, the existing instances became saturated, and the majority of the observed errors traced to OTP service response failures. Upstream messaging-provider rate limiting was ruled out as a material factor. Investigation into the underlying autoscaling behavior is ongoing with Site Reliability Engineering. | ### **3. Root Cause** **Primary Root Cause:** A concentrated surge in OTP verification request volume placed a sustained load on Socure’s OTP service — the service responsible for coordinating one-time passcode generation, delivery, and verification. The service is configured to scale horizontally under load. During the surge window, the additional instances did not successfully come up. The existing replicas became saturated and restarted repeatedly under sustained load, and the majority of the HTTP 5XX errors observed traced to response failures during this saturation period. Subsequent log and metric analysis confirmed that upstream messaging-provider rate limiting was not a material factor during the window. The dominant error signature in OTP logs corresponded to internal response-handling failures under load, not to vendor-side throttling. The condition was transient: as upstream OTP request volume returned to baseline, pod load returned to capacity and the OTP Verification endpoint auto-recovered without any change to Socure-owned services or vendor configuration. **Contributing Factors:** * **Horizontal scaling failure:** The service’s configured horizontal autoscaling policy did not produce additional instances during the surge window. Without the headroom of additional instances, the existing instances were unable to sustain throughput, became saturated, and restarted multiple times. Investigation into the underlying cause of the autoscaling failure is ongoing with Site Reliability Engineering. * **Baseline capacity headroom:** The baseline instance count for the OTP service, combined with the time required to bring additional instances online, did not provide sufficient headroom to absorb the observed surge prior to additional capacity becoming available. **Non-contributing Factors:** * **Upstream messaging provider:** Service-level rate limiting from the upstream messaging provider was not a material factor during the incident window. Genuine vendor-side 4XX responses observed in logs were limited in volume — far below the dominant OTP error — and did not drive the customer-visible impact. * **Recent changes or deployments:** No Socure application code, configuration, or release change was associated with the incident window. No broad outage was reported on any third-party dependency’s public status page during this period. ### **4. Resolution** * **Service restoration:** The condition was transient and self-resolved as upstream OTP verification request volume returned to baseline. Load on the OTP service returned to capacity, and error rates on the OTP Verification endpoint returned to normal at approximately 1:00 AM ET on June 15, 2026. No manual mitigation was required. Socure engineering verified recovery through telemetry and continued monitoring through and beyond the incident window. * **Validation and verification:** A post-incident review was conducted to confirm the scope and duration of impact. Detailed log and metric analysis identified the OTP service horizontal scaling failure as the primary contributor and ruled out upstream messaging-provider rate limiting as a material factor. Investigation into the underlying cause of the autoscaling failure is ongoing with Socure’s Site Reliability Engineering team. ### **5. Corrective and Preventive Actions** | # | Action | Priority | Status | | --- | --- | --- | --- | | 1 | Vendor engagement — Confirm with the upstream messaging provider the applicable API rate-limit model and validate the effective limits in force for Socure’s account. | High | In Progress | | 2 | Capacity baseline review — Re-baseline the OTP capacity envelope; revisit the baseline \(pre-scale\) instances count and autoscaling thresholds to ensure sufficient headroom to absorb peak surge traffic prior to additional replicas becoming available. | High | ETA: June 25th | | 3 | OTP  autoscaling investigation — Investigate with Site Reliability Engineering the root cause of why the configured horizontal autoscaling did not produce additional replicas during the surge, covering pod scheduling, resource constraints, and node availability. | High | In Progress | | 4 | Capacity planning and forecasting — Establish a recurring process to forecast peak OTP send volume from high-throughput customer programs and pre-coordinate vendor capacity ahead of anticipated traffic events. | Medium | Planned | ### **6. Conclusion** This incident was caused by a horizontal scaling failure in Socure’s OTP service during a brief, concentrated surge in OTP verification traffic. The service’s configured autoscaling policy did not produce additional instances during the surge window, the existing instances became saturated, and the majority of the errors observed traced to OTP service response failures. The condition was transient and self-resolved as upstream traffic subsided. Subsequent investigation ruled out upstream messaging-provider rate limiting as a material factor, and engineering is actively investigating the underlying cause of the autoscaling failure to prevent recurrence. Socure takes accountability for this incident and the impact it had on the affected customers. We remain committed to improving the resilience and maintaining a reliable experience for our customers.

Between 1:22 PM and 1:31 PM ET, DocV experienced a latency spike due to an internal node upgrade. During this 9-minute window, some customers may have experienced increased latency or timeout errors. The change was reverted right after the latency spike and DocV performance has returned to normal. We apologize for any inconvenience this may have caused.

We are currently investigating an issue causing increased latency and intermittent timeouts on the Transaction Search page within the Admin Dashboard. Our engineering team is actively working to resolve the issue and restore normal performance. API requests and transaction processing are working as expected and are not impacted. We apologize for the inconvenience and appreciate your patience while we work to resolve this issue.

API requests and transaction processing remain unaffected. We have identified the root cause of the latency and timeout issues affecting the Transaction Search page and have implemented mitigation measures. Some users may still experience delays in transaction visibility while we continue working toward a full resolution. We apologize for the inconvenience and appreciate your patience.

The issue causing increased latency and intermittent timeouts on the Transaction Search page within the Admin Dashboard has been resolved, and functionality has returned to normal. API requests and transaction processing were not impacted during this incident. We apologize for any inconvenience and appreciate your patience.

We are currently investigating a system-wide issue impacting RiskOS. Customers may experience errors, degraded performance, or delays when accessing RiskOS services and workflows. Our engineering teams are actively working to identify the root cause and restore full functionality as quickly as possible. We will provide additional updates as more information becomes available.

RiskOS API and real-time processing services have been fully operational for all customers since 07:17 PM ET. - Dashboard functionality, including case listing and search, has been restored for customers. Some customers may experience intermittent delays in seeing new case updates or decision changes reflected in the dashboard. - Webhook deliveries remain impacted for a small subset of customers, and our team is actively working toward a full resolution. We will continue to monitor system health and provide additional updates as we confirm complete service restoration. Thank you for your patience and understanding.

The issue impacting RiskOS has been resolved, and service has returned to normal operation. Customers should no longer experience errors, degraded performance, or delays when accessing RiskOS services and workflows. Our teams have confirmed system stability and will continue to monitor performance closely. We appreciate your patience throughout this incident and apologize for the impact this outage may have had on your operations.

**Root Cause Analysis** ### **Incident: RiskOS Network Connectivity Loss** **Date Range:** **06-09-2026** **Impact: Sev1** **Status:** Recovered ### **1. Summary** On June 9th, 2026 between 17:59 EST and 20:00 EST, network connectivity for RiskOS was disrupted in our commercial environment, resulting in widespread 5xx errors, limited webhook connection problems, and platform unavailability for multiple customers. The issue was linked to a recent GCP network change, which was reverted after discovery. As network connectivity was impacted, API calls never reached Socure, and so could not be recovered after the incident. Customer retries that happened after connectivity was restored worked as intended. API functionality began recovering around 18:41 EST, with dashboard recovery beginning at 19:10 EST, with full resolution at 20:00 EST. ‌ ### **2. Timeline** ‌ | **June 9th, 2026 / Time \(ET\)** | **Event** | | --- | --- | | 17:59 EST | Cloud Router resource deleted, cascades deletion to RiskOS Sandbox and Production Cloud NAT gateway resources. **IMPACT STARTS** | | 18:06 EST | Alerting and customer escalations are correlated, and the rollback to the prior configuration begins. | | 18:15 EST | First customers report API and dashboard impacts. | | 18:24 EST | First notification about incident posted via the Socure Status page | | 18:26 EST | Cloud NAT resource re-created with existing known-to-customer elastic IPs re-attached to Cloud NAT. Evaluation API functionality restored.**PRIMARY IMPACT RESOLVED** | | 18:41 EST | Identified continuing errors linked to customers who use IP whitelisting, as well as dashboard errors. Continued investigation. | | 19:10 EST | RiskOS production platform is now largely recovered, with API and real-time processing services operating normally for all customers. Troubleshooting continues for services supporting dashboard functionality. | | 19:29 EST | Dashboard functionality, including case listing and search, restored for most users. | | 19:49 EST | It is identified there was a missing rule in NAT configuration segregating traffic to come from specific elastic IPs. Rules configured, last 403 errors begin resolving. **ALL IMPACT RESOLVED** | | 19:52 EST | Updated Status page about progress on the case | | 20:00 EST | RiskOS connectivity issues declared resolved. | | 20:11 EST | Notification sent to customers that issue was fully resolved via Status page | ### **3. Root Cause** * **Primary Root Cause:** Unintentional deletion of Cloud Router in production Google Cloud Platform account caused deletion of Cloud NAT resources for RiskOS Sandbox and Production networks, thereby severing outbound connectivity. * **Contributing Factors:** * Google Cloud configurations are currently manually managed, contributing to higher accidental error rate. * In our existing configuration, the central networking resources are defined with production and sandbox resources, increasing the blast radius of core network modifications. In this case, the change unintentionally impacted production resources. ### **4. Resolution** Socure acted to restore the deleted Cloud NAT resources, as well as re-attaching known elastic IP addresses to network egress to prevent customer action. Additionally, select services \(the backends to our dashboard case listing and search\) were intolerant of network interruption, and required manual restart. ‌ ### **5. Corrective and Preventive Actions** ‌ | **Action**  | **Description** | **ETA / Status** | | --- | --- | --- | | Restore Cloud NAT resources. | The Cloud NAT and accompanying cloud router were restored. | Completed | | Refactor network resource configuration | Enforce separation of definitions for production and sandbox resources | Planning. Tentative ETA: July 2026 | | Detect connectivity failure faster | Increase the frequency of our external synthetic probes to detect connectivity loss faster. | In-Progress. ETA: Jun 30, 2026 | | Enable deletion prevention policies in Google Cloud Console | Move to add deletion protection mechanisms on critical resources to prevent unintentional resource modifications. | In-Progress. ETA: Jun 19, 2026 | ### **6. Lessons Learned** * The Cloud NATs which were deleted were unrelated to the scope of the network changes planned in the maintenance. Separating the production configuration and implementing our standard high-scrutiny change process is the highest priority. * Configure resource protection for critical infrastructure components to prevent unintentional changes. * Conduct network operations during non US business hours for more availability of support resources in the event of service impact. * Leverage infrastructure as code for absolute configuration and to ensure resource repeatability, as well as facilitate human review for dangerous operations. ### **7. Next Steps & Ongoing Commitment** Socure knows customers rely on it to be available and accurate, and takes full accountability for this configuration error and the disruption it caused. To ensure this problem does not recur, we are pursuing the following steps: 1. **Resource Protection:** We are adding deletion and change-protection controls to critical network infrastructure so that an accidental change cannot disrupt the platform. 2. **Infrastructure as Code:** We are already moving RiskOS network configuration to a version-controlled model so that every change is reviewed, repeatable, and can be recovered quickly within Socure’s AWS environment. 3. **Environment Isolation:** We are separating workload and core platform network resources so that the impact of any single change is contained. 4. **Safer maintenance windows:** We are scheduling network maintenance during periods with the strongest engineering and support coverage.

On June 3, 2026, at 12:50 AM ET, a subset of RiskOS customers experienced degraded functionality affecting evaluation processing, API responses, and portions of the RiskOS user interface. The issue was caused by an internal configuration error that impacted the availability of certain application capabilities. Evaluation processing and API functionality were restored by 5:00 AM ET on June 3, 2026, and case list consistency was fully restored by 5:00 AM ET on June 4, 2026. We are actively implementing additional validation controls and monitoring enhancements to help prevent similar incidents in the future. We sincerely apologize for the disruption and appreciate your patience while we worked to resolve the issue.

On May 23nd, 2026 between 12:20 AM ET and 1:12 AM ET, we observed 5xx errors returned by the EmailAuthScore endpoint for the DocV module. Minimal transaction failures occurred during this time. Errors have completely subsided and our team will share an RCA after analysis.

We are currently experiencing temporary SMS delivery delays for a small subset of users, which may require retrying verification attempts. The team is actively working to mitigate impact while monitoring the issue closely. No customer data or systems have been impacted.

The issue affecting DocV SMS delivery has been resolved. End users should now receive DocV SMS messages as expected.

# **Root Cause Analysis** Report on Socure High SMS Handoff Failure Rate in DocV V5 May 20, 2026 ## **Summary** On May 20, 2026, starting at approximately 9:05 AM ET, Socure's Document Verification \(DocV\) V5 service experienced a significant increase in SMS handoff failures in the commercial production environment. End users were unable to receive SMS verification links. The issue was caused by the exhaustion of Socure's AWS End User Messaging monthly SMS sending budget, triggered by an abnormal volume of international SMS traffic. Socure applied account-level rate limiting, restricted international SMS delivery, and engaged AWS Support to expedite a budget increase. The incident was fully resolved at 12:15 PM ET on May 20, 2026. ## **Timeline** First Alert: May 20, 2026, 9:05 AM ET Issue Initially Contained: May 20, 2026, 9:20 AM ET Issue Resurfaced: May 20, 2026, 10:44 AM ET Issue Fully Resolved: May 20, 2026, 12:15 PM ET **Customer Impact Duration:** ~1 hour 46 minutes **Total Incident Lifecycle:** ~3 hours 10 minutes ## **Description of Incident** On May 20, 2026, at 9:05 AM ET, Socure's monitoring detected an elevated SMS handoff failure rate in DocV V5 for the commercial production environment. Investigation revealed that the AWS End User Messaging monthly SMS sending budget had been exhausted, driven by an abnormal volume of international SMS traffic originating from a single account.  Once the monthly budget limit was reached, AWS End User Messaging blocked all outbound SMS delivery from the account. Failures were permanent — messages were not retried once the limit was hit. ## **Impact** * 100% of SMS handoff attempts failed during the peak impact window * Affected customers' end users were unable to receive SMS verification links via DocV Socure immediately applied account-level rate limiting to reduce ongoing spend and requested an emergency budget increase from AWS. International SMS delivery to the top destination countries was also blocked to halt the spend burn while the root cause was addressed. AWS Support was formally engaged with details on traffic patterns and user opt-in flows. Socure published a status page communication to notify customers of the SMS delivery delays and ongoing mitigation efforts. The source of the abnormal traffic ceased sending, and the incident was fully resolved at 12:15 PM ET on May 20, 2026. ## **Resolution** Socure applied account-level rate limiting and blocked international SMS to some destination countries where we noticed unusual delivery pattern to immediately reduce the rate of budget consumption. AWS Support was engaged to expedite an increase to the monthly SMS sending budget. Once the abnormal traffic was stopped and the budget constraints were addressed, SMS delivery was restored. The incident was confirmed fully resolved at 12:15 PM ET on May 21, 2026. ## **Mitigation Plan** These measures are aimed at improving SMS delivery resilience and preventing similar incidents from affecting production environments in the future. _We are committed to transparency, timely remediation, and making the necessary improvements to prevent recurrence._

We are aware of an issue that caused latency and intermittent 5xx errors that affected a small subset of DocV transactions that call Image Alert List. The team is looking into solutions to prevent this from recurring.

Our team has applied a fix to prevent any recurrence of latency and intermittent 5xx errors for DocV transactions calling Image Alert List. We appreciate your patience as our team worked through this issue.

We are currently investigating reports of intermittent errors and performance degradation across several platform modules. Users may experience 5xx error messages. Our engineering team is actively diagnosing the root cause and is working to restore full functionality as quickly as possible.

We are actively working in close partnership with AWS to address the underlying issues currently impacting system performance. Users may still experience 5xx error messages as we work towards a solution to restore full functionality.

We are pleased to inform that all our services are operating normally. We are currently monitoring our services in partnership with AWS.

Socure services have now fully recovered and are operating normally. System latency and performance metrics have returned to expected levels. Our engineering and operations teams will continue to work closely with AWS to conduct a comprehensive root cause analysis (RCA) of the incident. We will share the RCA publicly within the next few days once this review is complete.

DocV mobile app users may experience an issue where the app becomes stuck on a loading spinner during launch within the web view, preventing access. This only impacts certain users, specifically those using app configurations that rely on a web view. Our team is actively investigating and will provide updates as more information becomes available.

Our team has implemented a fix and is actively monitoring its performance to ensure the issue is fully resolved.

The issue has been resolved and services are now operating normally. Thank you for your patience while our team worked to address this.

On April 14th, 2026 we observed issues with displaying transactions calling the First Party Fraud module in the ID+ platform. No transactions failed during this time but some users would have seen a blank screen when reviewing the transaction details in the dashboard. Our team quickly identified the root cause and worked out a fix to mitigate this problem, restoring normal functionality as of 4:00PM ET.

We’re currently seeing that RiskOS dashboards for some customer accounts are not loading. This is due to maintenance on the vendor side taking longer than expected. We are actively working with the vendor to restore the visualizations as soon as possible. We apologize for the inconvenience and appreciate your patience while we work to resolve this.

We apologize for the continued interruption to the dashboard visualizations and are actively working with vendor to restore full service as quickly as possible. There is no impact to evaluation APIs or other core platform functionalities, which continue to operate normally. We’ll continue to share updates as we make progress. Thank you for your patience.

The issue has now been resolved, and services have been fully restored. We appreciate your patience and understanding while we worked through this. If you notice anything unexpected or have any further questions, please don’t hesitate to reach out.

We are currently observing intermittent timeouts impacting eCBSV requests. Our team is actively investigating. We sincerely apologize for the inconvenience and will share further updates as they become available.

We are still observing intermittent timeouts impacting eCBSV requests, and our team is actively investigating the issue. We sincerely apologize for the inconvenience and will continue to share updates as they become available. In the meantime, we truly appreciate your patience.

We continue to observe intermittent timeouts affecting eCBSV requests. The root cause has been identified, and remediation efforts are currently in progress. We’ll share further updates as we make progress toward resolution.

We’re pleased to inform you that the required deployments have been successfully completed on our end, and the issue is now resolved. We are currently working closely with our vendor to compile a detailed root cause analysis, which will be shared with you in the coming days.

We are currently observing errors impacting eCBSV following SSA’s recent maintenance window from 9:00 PM ET on Saturday, March 21, 2026, through 1:00 AM ET on Sunday, March 22, 2026. We continue to see the usual maintenance-window error indicating that the SSA service is unavailable due to planned maintenance, and we are actively coordinating with SSA to better understand the situation. We will provide additional updates as soon as they are available.

We have not seen the eCBSV error for the past 30 minutes, and the unavailability issue appears to have auto-resolved on SSA’s end. We will follow up with SSA to better understand what led to this unavailability extending beyond the planned maintenance window. We sincerely apologize for any inconvenience caused.

On March 21st, 2026 between 1:30 PM ET and 5:50 PM ET, we observed latency degradation in the ID+ platform. No transaction failures occurred during this time. Mitigation has been applied, and latency has since returned to normal levels. Investigation is ongoing to prevent recurrence.