Secureauth 장애 기록

Our team of engineers are investigating. Firing - The component IAM SaaS AU Region is down and we're working on it.

The problem has been resolved. Resolved - The component IAM SaaS AU Region is down and we're working on it.

We are currently investigating an issue affecting mfa.acceptto.com. We will provide updates as soon as we have more information. If you have any questions in the meantime, please log a case at https://support.secureauth.com.

We have identified the issue and have implemented a fix. We will continue to monitor closely. An RCA will be added to the post mortem once the investigation is complete. If you have any questions regarding this matter or are experiencing any problems, please let us know by logging a ticket at https://support.secureauth.com. Thank you.

This incident has been resolved. An RCA will be added to the post mortem once the investigation is complete. If you have any questions regarding this matter or are experiencing any problems, please let us know by logging a ticket at https://support.secureauth.com. Thank you.

We are investigating reports of intermittent SSL Issues for Workforce Cloud IdP applications. We will provide updates as soon as we have more information. If you have any questions or concerns, please reach out to support at https://support.secureauth.com.

We have identified the issue and have implemented a solution. SSL errors have all been resolved. An RCA will be provided as soon as the investigation has been complete. If you have any questions, please reach out to support at https://support.secureauth.com.

Our team of engineers are investigating. Firing - The component IAM SaaS US Region is down and we're working on it.

The problem has been resolved. Resolved - The component IAM SaaS US Region is down and we're working on it.

We are currently experiencing issues with processing some of the incoming requests. Our team is investigating the cause and working to restore full service as quickly as possible.

The issue has been identified and resolved. The system is now operating normally.

We observed a ~1 minute disruption within our CIAM product in the US region. The disruption was quickly fixed, with no risk for further issues. If you have any questions regarding this matter, please reach out to our Support Team at https://support.secureauth.com.

Our SMS and Telephony provider has indicated that they are currently experiencing quality degradation toward US Multiple Operators. Customers that send SMS messages to Multiple Operators users may notice a difficulties in message delivery. Verizon has reported a major incident affecting SMS and Telephony Services. If you have any questions regarding this matter, please reach out to our Support Team at https://support.secureauth.com.

Our SMS and Telephony provider has informed us that the network issues previously reported by Multiple Operators from the US have been resolved. Operator end users should no longer experience delays receiving SMS messages as all services are now functioning properly.

We are currently investigating a Cloud Services issue, and will provide updates as we have more information. If you have any questions in the meantime, please log a ticket at https://support.secureauth.com. Thank you

The issue has been identified and our operations team is working on implementing a fix.

We are continuing to work on a fix for this issue.

A fix has been implemented and we are monitoring the results. If you are still having issues, please let us know by logging a ticket at https://support.secureauth.com

The incident has been resolved. We will continue to monitor and will provide an RCA once the post-mortem investigation is complete. If you have any questions or require assistance in the meantime, please contact support at https://support.secureauth.com

**RCA - Cascading Service Failure – 12-15-2025**  **Incident date:** December 15, 2025 **Duration:** ~90 minutes **Severity:** SEV-1 **Services affected:** Polaris platform services  **Summary**  On December 15, 2025, we experienced a significant service disruption caused by a cascading failure across multiple internal systems.  A brief loss of network connectivity \(leadership\) in our Vault cluster triggered restart loops in dependent services. Automated scaling amplified those restarts, resulting in a surge of database connections that overloaded our CockroachDB cluster. This caused elevated latency and service unavailability across a majority of tenants.  No customer data was lost.  Fortunately, improvements made with a new EKS cluster with GitOps methodology, engineering was able to quickly gain details on the various impacted services and formulate a deep understanding of what was occurring. This lead to a faster recovery time, while also leading to more informed action items as an outcome.  **What happened**  At approximately 1:25 PM PST, Vault lost its active leader following a transient internal connectivity issue. Vault itself was not under resource pressure, but several application services depended on Vault being continuously available.  When Vault became unavailable:  * Twilight and other dependent services failed health checks  * Those services entered restart loops  * Kubernetes autoscaling rapidly increased pod counts, overloading cluster  * The resulting connection storm overloaded CockroachDB  Once CockroachDB was saturated, Vault could not recover cleanly, prolonging the incident until manual intervention reduced system load.  The combination of the overloaded cluster and CockroachDB led to the new cluster's improved self healing and auto recovery being blocked.  **Impact**  During the incident window:  * Requests across all tenants experienced increased latency or failure  * Services depending on CockroachDB were degraded  * **New workloads were unable to schedule due to cluster pressure**  Customer-facing services were restored once database load and service mass restarts returned to normal levels.  **Timeline**  * **All times PST.**  * **1:25 PM –** Vault loses leader following internal connectivity disruption  * **1:27 PM –** Twilight health checks begin failing; pods restart repeatedly  * **1:28 PM –** Autoscaling increases pod counts to maximum limits  * **1:28 PM –** CockroachDB CPU and connection counts spike sharply  * **1:37 PM –** Operations team begins investigation into alert spikes  * **1:50 PM –** Support updates slack with incoming support tickets  * **1:55 PM –** Support/Operations declare official incident, Zoom meeting occurs, StatusPage updated  * **2:10 PM –** CockroachDB scaled to stabilize database performance  * **2:30 PM –** Vault recovers after database stabilization  * **2:35 PM –** Affected services manually scaled down to alleviate cluster pressure due to HPA  * **2:45 PM – Platform fully recovered, services return to typical state**    **Vault leadership loss**  Vault lost its leader early in the incident window. While brief, this event disrupted secret retrieval for dependent services. Vault was not CPU or memory constrained; the failure mode was related leadership and internal connectivity.  **Dependency failure**  Twilight services required continuous Vault connectivity to operate. When Vault became unavailable, those services were unable to start successfully. This hard dependency prevented graceful degradation and contributed directly to restart loops.  **Service restart amplification**  Twilight services repeatedly restarted after losing access to Vault. Autoscaling responded to these restarts by rapidly increasing replica counts. Rather than restoring availability, this behavior amplified load across the cluster.  **CockroachDB saturation**  The surge in restart activity and autoscaled pods caused a sustained spike in database CPU usage and connection counts. This saturation increased query latency and prevented dependent systems from recovering automatically.  **Root cause**  This incident resulted from multiple conditions interacting:  * **Hard dependency on continuous Vault availability -** Services failed rather than degrading when secrets could not be retrieved.  * **Overly permissive autoscaling limits** - Default HPA maximums allowed rapid scaling beyond operational needs.  * **Shared database cluster infrastructure overload** - The resulting connection storm overwhelmed CockroachDB, blocking recovery.  Individually, none of these issues would have caused a platform-wide outage. Together, they resulted in a cascading failure.  ‌ **What we’re changing**  We are making the following _improvements_:  _Application architecture_  * Remove hard dependencies on continuous Vault availability \(Github PR created, in review\)  * Ensure secrets are retrieved and held securely at startup \(Github PR created, in review\)  * Prevent expensive startup operations from running on every restart, no longer running on every pod restart, but once per major maintenance \(Github PR created, in review\)  _Infrastructure configuration_  * Reduce autoscaling maximums to realistic service-specific limits \(To be updated 12/16 overnight\)  * Isolate Vault onto a dedicated database backend \(In Progress\)  * Review usage of Vault, reduce dependency by moving to alternative solution \(K8s secrets with SOPs\)  _Resilience and monitoring_  * Update health checks so non-critical failures do not trigger restarts \(PR created, in review\)  * Add safeguards to prevent database overload during restart storms  * Expand failure-mode testing in lower environments  * Multi-Region Hot Standby \(In Progress, Available Q2 2026\)  ‌ **Closing**  A brief disruption in a single component should not cascade into a platform-wide incident. This event highlighted areas where coupling and automated responses amplified failure instead of containing it.  While we have made vast improvements to our overall infrastructure over the last few months \(adopting GitOps, creating a new cluster with improved monitoring and core services, increased metric and log gathering and retention, eliminating any manual changes, etc\) we are addressing those gaps and strengthening the platform to improve resilience going forward.  We will be scheduling a maintenance window within the next two weeks to implement immediate remediation action items, while hot standby will lead to further improvements and failover within minutes once available. These immediate changes \(due to deep RCA provided by improved cluster telemetry\) coupled with the multi-region hot standby live in Q2 will prevent the underlying causes of the recent incidents.

We are investigating reports of intermittent issues with the Mobile Service potentially affecting Push and TOTP MFA methods. We will post updates as we gain understanding to the issue. SMS OTP and Telephony OTP services are operational and can be used as an alternative MFA option. If you have questions or require additional assistance, please log a ticket with support at https://support.secureauth.com.

We've identified the issue and have implemented a fix. We are continuing to monitor results and will provide a root cause once post mortem investigation is complete.

We have monitored the results of the fix for an extended period and have not observed any issues since the fix was implemented. The issue has been fully resolved. An RCA will be be posted once the post mortem investigation is complete.

**RCA - Intermittent Mobile Service issues - 12.05.2025** **Issue Description:** On December 5, 2025, a subset of customers experienced intermittent disruptions affecting the Push-2-Accept and TOTP multi-factor authentication \(MFA\) methods for end users. **Cause:** At approximately 5:00PM PST on December 5, 2025, an update was applied to the nginx ingress routing configuration for a specific tenant. Due to the complexity of the tenant’s ingress rules, the change exceeded nginx’s processing capacity, resulting in performance degradation and intermittent routing failures. Consequently, a small portion of network traffic could not be serviced, causing intermittent MFA issues for a small percentage of customers. **Recovery:** The ingress configuration changes were rolled back, restoring nginx to a stable state and resolving the affected MFA functionality. **Corrective Actions:** * The tenant with highly complex ingress routing requirements will be migrated to a dedicated custom deployment to prevent similar issues in the future.

Cloudflare is currently aware of and investigating a widespread issue affecting multiple customers. Our engineering team is actively working on this issue and coordinating closely with Cloudflare to fully assess the impact and expedite remediation. Further updates will be provided as more information becomes available.

We have adjusted network routing for Passwordless to bypass Cloudflare infrastructure. This change should begin to take effect shortly, depending on DNS propagation and your ISP’s DNS settings. We continue to actively work on this issue and are coordinating closely with Cloudflare as they work toward full resolution. We will provide further updates as they become available.

Our internal monitoring systems indicate that service functionality has returned to normal. However, Cloudflare has not yet confirmed full remediation of the underlying issue. We continue to closely monitor the situation. We remain in active communication with Cloudflare and will provide additional updates as soon as we have more information.

Cloudflare reports that they are seeing services recover; however, you may continue to experience higher-than-normal error rates as remediation efforts continue. We will provide further updates as additional information becomes available.

We are continuing to work on a fix for this issue.

Cloudflare identified the issue and started implementing a fix. In the meantime, we are continuing our efforts to restore full functionality as quickly as possible.

Cloudflare reports a fix has been implemented and we believe the incident is now resolved. We are continuing to monitor for errors to ensure all services are back to normal.

All services remain operational. We will continue to monitor. Any workaround actions that were taken for SAIdP and Acceptto will be reverted later this evening at 1am PST. No impact to customers. Once completed we will close this incident.

All workaround actions for SAIdP and Acceptto have been successfully reverted, and our systems have fully returned to their normal operating state. We have confirmed stable service across all components and no customer impact during the reversion process.

We are investigating an issue with our backend cloud services that may impact customer workflows. If you have any questions regarding this issue, please log a ticket at https://support.secureauth.com. Our teams are on standby and are ready to assist.

We see heavy database utilization leading to service outages. We are in the process of failing over to secondary database to alleviate pressure. If you have any questions regarding this issue, please log a ticket at https://support.secureauth.com. Our teams are on standby and are ready to assist.

We are starting to execute the implementation of a fix. This should take up to 30 minutes to be completely rolled out. If you have any questions regarding this issue, please log a ticket at https://support.secureauth.com. Our teams are on standby and are ready to assist.

All services are operational again and we are continuing to monitor. If you have any questions regarding this issue, please log a ticket at https://support.secureauth.com. Our teams are on standby and are ready to assist.

All services remain fully recovered. If you have any questions regarding this issue, please log a ticket at https://support.secureauth.com. Our teams are on standby and are ready to assist.

**RCA – Cloud Services Outage/RDS Failure - October 23, 2025** **Problem Description:** On October 23, 2025, at 12:30 PM PDT, SecureAuth’s Cloud infrastructure experienced degradation of the IP risk evaluation service, which by around 2:00 PM PDT, escalated into widespread database connection issues, affecting multiple cloud services and resulted in authentication failures for impacted customers. A gradual rolling restart of all single tenant services in batches restored services, leading to a full recovery at 4:00 PM PDT. **Cause:** The incident originated with an unbounded increase in the IP risk evaluation service traffic \(ipintelsvc\) to the production database. The resulting proliferation of concurrent, heavy-query sessions overwhelmed the shared Aurora RDS cluster, including the segment serving the Vault secret-storage service. Loss of database connectivity led to a Vault crash and downstream failures in dependent services. **Recovery:** The DevOps and Engineering teams initiated resolution efforts by scaling down the IP risk evaluation service traffic \(ipintelsvc\), which led to RDS CPU and connection metrics gradually decreasing, allowing auxiliary vault services to come back up. A rolling restart was then performed for all customer service deployments, in a gradual batched fashion. **Timeline:** October 23, 2025 * 12:45 PM PDT – Alerts triggered for spikes for two customers * 1:08 PM PDT – DevOps and Engineering teams begin investigation * 1:13 PM PDT – First Support ticket received regarding outage * 1:30 PM PDT – ipintelsvc logs confirm spike * 1:45 PM PDT – ipintelsvc scaled down * 2:00 PM PDT – Cleanup of lingering sessions in progress * 2:05 PM PDT – Vault Polaris service recovered * 2:15 PM PDT – Rolling restart of all customer services begins * 3:15 PM PDT – Near full recovery confirmed, with a small batch requiring manual intervention * 4:00 PM PDT – Full recovery confirmed **Corrective Actions:** * Software updates to the infrastructure and backend services to improve performance and security will be scheduled during a change maintenance window for customers on a weekly cadence. * Further enhancements leading to the separation and resilience of single tenant services will be made, eliminating the risk of the cascading restart requirement, and decoupling from the shared services RDS. This infrastructure has already been implemented and applied in Dev and Test SecureAuth Cloud environments, and we are starting to roll it out in Production. If there is a preference to be placed on a priority list to have the updates performed to your tenant, please reach out to your CSM or Support and we will work on prioritizing a change window for your organization.

SecureAuth is experiencing major impact to our cloud services due to an ongoing AWS incident within the us-east-1 region. This includes authentication-related functions and 2FA methods such as PUSH notifications. Our teams are actively monitoring AWS recovery efforts and working to restore normal service levels as quickly as possible. If you have any questions regarding this issue or require assistance with applying the fix, please log a ticket at https://support.secureauth.com. Our teams are on standby and are ready to assist.

All affected service issues are operational. We will continue to work closely with AWS and monitor their recovery efforts. Performance degradation may continue as recovery efforts on AWS services are performed. We will be closely monitoring our systems throughout this effort. If you have any questions regarding this issue or require assistance with applying the fix, please log a ticket at https://support.secureauth.com. Our teams are on standby and are ready to assist.

The ongoing AWS us-east-1 Incident (https://health.aws.amazon.com/health/status) is leading to limited functionality specifically regarding Secureauth PUSH services. If affected, customers are advised to use alternative MFA methods until the issue is resolved. We will keep the Secureauth status page updated with any changes, including when service issues are resolved. If you have any questions regarding this issue or require assistance with using alternative MFA, please log a ticket at https://support.secureauth.com. Our teams are on standby and are ready to assist.

The ongoing AWS us-east-1 Incident (https://health.aws.amazon.com/health/status) is leading to limited functionality specifically regarding Secureauth PUSH services. Cloud customers will also see limited functionality with EMAIL services. If affected, customers are advised to use alternative MFA methods until the issue is resolved. We will keep the Secureauth status page updated with any changes, including when service issues are resolved. If you have any questions regarding this issue or require assistance with using alternative MFA, please log a ticket at https://support.secureauth.com. Our teams are on standby and are ready to assist.

PUSH services to Android devices may experience degraded performance. All affected service issues are now operational. We will keep the Secureauth status page updated with any changes, including when service issues are resolved. If you have any questions regarding this issue or require assistance with using alternative MFA, please log a ticket at https://support.secureauth.com. Our teams are on standby and are ready to assist.

All affected service issues are now operational. We will continue to work closely with AWS and monitor their recovery efforts. Performance degradation may continue as recovery efforts on AWS services are performed. We will be closely monitoring our systems throughout this effort. If you have any questions regarding this issue or require assistance with using alternative MFA, please log a ticket at https://support.secureauth.com. Our teams are on standby and are ready to assist.

All affected service issues continue to be operational. AWS has marked their incident as resolved (https://health.aws.amazon.com/health/status). We are updating this incident as Resolved. If you have any questions regarding this issue please log a ticket at https://support.secureauth.com. Our teams are on standby and are ready to assist.

IAM SaaS US Region is experiencing performance degradation, which is leading to an increase in 500 errors. We have identified the issue and are implementing a resolution. We will provide an update once available, please reach out to support@secureauth.com with any questions

We have applied the resolution and are no longer seeing 500 error spikes. We will continue to monitor. We will provide an update once available, please reach out to support@secureauth.com with any questions

This incident has been resolved.

Our team of engineers are investigating. Firing - The component IAM SaaS US Region is down and we're working on it.

The problem has been resolved. Resolved - The component IAM SaaS US Region is down and we're working on it.

Our team of engineers are investigating. Firing - The component IAM SaaS US Region is down and we're working on it.

The problem has been resolved. Resolved - The component IAM SaaS US Region is down and we're working on it.

Our team of engineers are investigating. Firing - The component IAM SaaS US Region is down and we're working on it.

The problem has been resolved. Resolved - The component IAM SaaS US Region is down and we're working on it.

Our team of engineers are investigating. Firing - The component IAM SaaS US Region is down and we're working on it.

The problem has been resolved. Resolved - The component IAM SaaS US Region is down and we're working on it.

Our team of engineers are investigating. Firing - The component IAM SaaS US Region is down and we're working on it.

The problem has been resolved. Resolved - The component IAM SaaS US Region is down and we're working on it.

Our team of engineers are investigating. Firing - The component IAM SaaS US Region is down and we're working on it.

The problem has been resolved. Resolved - The component IAM SaaS US Region is down and we're working on it.

Our team of engineers are investigating. Firing - The component IAM SaaS US Region is down and we're working on it.

The problem has been resolved. Resolved - The component IAM SaaS US Region is down and we're working on it.