Strongdm 中断历史

We’ve identified that yesterday’s incident also impacted Slack Integrations. Specifically, token refresh requests were timing out, which caused some Slack connections to disconnect. Latency has improved following the rollback, and we have implemented a server-side fix (version 123.37.0) to address the Slack Integration disconnections. This update restores affected integrations and prevents future disconnections in the event of a token refresh timeout. Customers who are still experiencing issues may reconnect the Slack Integration manually via the StrongDM Admin UI. We will continue to monitor performance and provide additional updates as needed.

We are currently investigating elevated latency and intermittent timeout errors affecting the US control plane (app.strongdm.com). Customers may experience slow load times in the Admin UI and delayed or failed/slow API requests. Our engineering team is actively working to identify the root cause and mitigate the impact. We will provide an update as soon as more information is available.

Status Update – Performance Investigation Engineering has rolled back the recent change, and we’ve observed improvement in latency metrics. However, other performance indicators have not shown the same level of improvement. We are continuing to actively investigate the issue to determine the underlying cause. Some incremental improvements may still be observed as we proceed with analysis and mitigation efforts. Look for further updates here. A full Root Cause Analysis (RCA) will be provided once we have identified the cause and implemented a confirmed fix.

We have identified the root cause for the elevated latency and intermittent timeout errors which were affecting the US control plane (app.strongdm.com). There should be no further service-impacts as a result of this event. Engineering is taking steps to prevent such occurrences moving forward, and will provide an RCA within the next 7 days.

On February 17, 2026, some customers experienced elevated latency in the US Control Plane due to a relatively small number of specific query log requests. These specific requests were searching for a very small subset of queries within a very large dataset, which created significant server load for such a small number of requests.  We implemented safeguards to improve log retrieval behavior and restored system performance. These safeguards should not affect normal query retrievals, but will reduce the load of specific query log requests by spreading out the request over a longer time period. Longer term, we are prioritizing other improvements to our query log architecture.

StrongDM is currently investigating an issue that may impact connectivity to AWS services, specifically AWS Secret Stores, for some customers. Our team has confirmed a regression and is actively working on a fix. We will be rolling back the affected change to restore service for impacted environments. We will continue monitoring closely and will provide additional updates as more information becomes available.

The rollback has been completed. The issue will also be fixed in code.

TIMELINE * On January  14, 2026 at 4:59pm PDT, a pull request was merged which changed the way the relays obtain credentials from AWS. We migrated to Go AWS SDK v2 and our migration was missing container credential support which worked out of the box with v1. In EKS environments our code skipped the container endpoint and went directly to IMDS, which could return the wrong IAM role * On January 28 2:20am PDT first ticket came into support * On January 29 9:53am PDT, the cli release was identified and rolled back. * By 9:59am PDT all control planes were rolled back. CUSTOMER IMPACT * Relays running in EKS failed to assume correct IAM roles and resources turned unhealthy CAUSES * Incomplete SDK migration: The migration from AWS SDK v1 to v2 did not account for differences in the default credential provider chain. SDK v1  automatically supported container credentials, while v2 required explicit configuration. * In EKS environments, IMDS returns the EC2 node's IAM role rather than the pod's IAM role \(configured via IRSA or EKS Pod Identity\), causing relays to authenticate with incorrect/insufficient permissions POSSIBLE REMEDIATIONS * Include more testing for AWS credential providers * Add monitoring for relay credential failures: Create alerts for spikes in IAM authentication failures or resources transitioning to unhealthy state

Investigation has started. Updates to follow. At this time the only StrongDM component impacted is our Slack Integration.

The issue has been identified and a fix is being implemented.

We have rolled back to a previous version and the Slack Integration should now be working. If you are still having problems related to our Slack integration please reach out to Support: https://help.strongdm.com/hc/en-us/requests/new We are monitoring the situation.

This issue has been resolved. The cause: The SKU of certain customers was miscalculated by the Slack integration, resulting in affected customers losing functionality. A code change will be pushed out to make fix this issue.

We are currently investigating the issue. We will update here as we know more.

We have identified an issue causing performance issues in our product DB. We are monitoring and a more detailed update will follow.

The incident has been resolved and we will provide an RCA within the next 7 days.

**Summary** On December 11th 2025, StrongDM experienced a service degradation affecting the US Control Plane. The issue was caused by resource exhaustion from an internal monitoring component and lasted approximately 73 minutes before service was fully restored. **What Happened** An internal monitoring component configured to collect operational telemetry was unable to process data efficiently under US production load. This led to resource exhaustion on the Control Plane. UK and EU Control Planes were not affected due to lower total throughput volumes. **Resolution** Infrastructure rolled back the monitoring component's configuration, immediately relieving resource pressure and restoring normal service. **Prevention & Remediation** To prevent recurrence, StrongDM is updating testing processes to better catch issues like this and revising configuration review processes for internal tooling updates.

Our orchestration platform has been experiencing intermittent errors and we are actively working to fix the issues as quickly as possible. At this time the issue has been resolved and reporting is working correctly.

StrongDM is aware of an issue with the Admin UI not loading. The issue has been identified and a fix is being implemented.

We are continuing to work on a fix for this issue.

The Admin UI's for both the US and EU control planes have been restored. We are still working on restoring service to our UK control plane. Updates to follow.

At this time Admin Access to the UK control plane has been restored. If you are still having issues connecting to the StrongDM Admin UI reach out to StrongDM Support https://help.strongdm.com/hc/en-us/requests/new

**Summary** On October 21st, 2025, StrongDM experienced an outage affecting access to the Admin UI across multiple Control Planes. The issue began shortly after a planned update and lasted approximately 30 minutes before service was fully restored. **What Happened** Prior to the incident, Control Planes were operating on Admin UI version 116.27.0. Around 12:00 PM PDT, a code change was deployed that unintentionally disrupted replication between internal systems responsible for propagating updates across Control Planes. At 12:45 PM PDT, Admin UI version 116.28.0 was released as part of a routine update. Due to the earlier replication issue, this new version was not consistently distributed to the Control Planes. This inconsistency caused components of the Admin UI to become unavailable. The outage lasted for approximately 30 minutes while the team identified the replication issue, restored consistency, and validated recovery. **Resolution** Engineering corrected the replication pathway and ensured all Control Planes received the appropriate version updates. Normal service was restored once consistency was re-established. **Prevention & Remediation** To prevent this issue from occurring again, StrongDM engineering has implemented additional validation and safety checks in the release process to ensure that updates are properly replicated before becoming active.

We’re aware of an ongoing issue with AWS that may impact some StrongDM US Control Plane customers. While this is not a StrongDM issue, we are actively monitoring the situation and will provide updates as needed. For more information, please refer to AWS’s status page: https://health.aws.amazon.com/health/status

As of 3:53 PM PDT this issue was resolve according to AWS. More information can be found here: https://health.aws.amazon.com/health/status this issue appears to be resolved.

The problem has been identified and a rollback is underway. More details will follow.

A fix has been implemented and we are monitoring the results.

This incident has been resolved. An RCA will be provided within a week. If you logged in to the StrongDM service during this short outage, the authentication won't be able to be verified with our control plane. Please log out and log back in to refresh the authentication with a valid token.

**Summary:** During a routine system update on October 2, a server release was pushed to production that changed how tokens were validated, which led to failed connections to resources. Our internal canary deployment caught the issue and alerted the team. They started to rollback the change when we started to learn about customer impact. There were two compounding issues that caused this outage.  **Cause:** It took some time for the issue to manifest in our Canary environment. By the time the system alerted, the build was deemed OK and pushed to production. Which was the ultimate root cause of the failure. **Impact:** * Disruption to resource connections. * Users who logged in during the window needed to re-authenticate. * No unauthorized access or security exposure. **Remediation & Prevention:** * Extending the delay between staging and production deployments for better early detection. * Updating infrastructure to manage environment

We are currently investigating this issue. Updates to follow.

We are continuing to investigate this issue.

The issue has been identified and a fix is being implemented.

A fix has been implemented and we are monitoring the results. If you are still seeing issues at this point you should be able to restart your relay/gateways to get back into a good state. Let us know if you are still having authorization failures via our ticket portal: https://help.strongdm.com/hc/en-us/requests/new

This incident has been resolved. If you are still seeing issues please reach out to StrongDM Support via our Help Center: https://help.strongdm.com/hc/en-us/requests/new An RCA will be posted within the next 7 business days.

**Cause:** * A change to the control plane was rolled out which altered the default policy used to authorize connections when the policy editor is not enabled for an organization. This policy was synced down to customer infrastructure for local authorization but, critically, new entities which were referenced by that policy were not always synchronized.  * This was caused by an incomplete implementation of a database synchronization function on the control plane, which, in some cases, informed clients about the policy but not about the new entities. * The policy began to deny authorizations on customer infrastructure due to missing entity references which caused the policy to not apply. This issue affected customers who had policy _disabled_ \(either by SKU or by toggle in the Admin UI\) because the globalpolicy is intended to stand in for hand-written policy when policy is enabled. **Resolution:** * To fix the issue, the control plane was rolled back to a version that did not have the new globalpolicy plugin. However, the bad policy could have synchronized to customer nodes and, in order to make sure that it was purged, we later merged a change to bump the authsync version which forced all customers' nodes to fully resynchronize their policies and entities with the control plane. **Downtime:** * Total downtime was just over an hour. Restarting nodes helped some customers due to forcing a resynchronization of the full set of policies and entities.  **Prevention:** * Going forward StrongDM now automatically forces a full resynchronization whenever code that is involved in synchronizing policies and entities to the nodes is modified.

The US Control Plane experienced a degraded state affecting the ability for users to log in. The situation has stabilized and we are currently monitoring the Platform's stability. Customers who experience issues accessing the US Control Plane should reach out to the Support Team https://help.strongdm.com/hc/en-us/requests/new

SDM is currently working to resolve the issue. We will update again once we have additional information. Standard functionality for those already invited to SDM remains normal.

StrongDM has identified the issue blocking email delivery for our UK Control Plane. Currently users will not receive emails for the following reasons: *New user invitations *Access request workflow notifications (these will still work within the Admin Console and Slack/Teams) Please note, that Customers using StrongDM's native authentication (username/password) are currently unable to invite new users, but Customers using an external authentication provider are still able to onboard new users.

Email notifications are once again being sent on our UK control plane. We are monitoring the issue. If you continue to see issues with email notifications please reach out to StrongDM Support: https://help.strongdm.com/hc/en-us/requests/new

This issue has been resolved. No additional incidents were reported or logged over the weekend.

We're currently investigating an issue that is causing degraded performance for some users. We will provide further updates as they become available.

We have found and resolved the issue affecting the US control plane. Estimated impact on performance was approximately 6 minutes. We are continuing to monitor affected systems.

This incident has been resolved.

We are currently investigating an authentication issue affecting some users logging in via OneLogin SSO at app.strongdm.com and the StrongDM GUI client. SAML: A recent change from OneLogin appears to impact SAML logins. As a workaround, remove the trailing slash from the Login URL in your StrongDM application settings within OneLogin. This restores login functionality for most SAML users. OIDC: A recent change from OneLogin appears to impact OIDC logins. OneLogin has reverted the change. No further action is required.

We are continuing to investigate this issue.

This issue has now been resolved. SAML: The app login URL should not include a trailing slash. OIDC: OneLogin reverted the changes to OIDC authentication behavior, no further changes are required.

StrongDM is experiencing intermittent connection issues with https://app.strongdm.com a rollback is in process. We will update with more information.

StrongDM is currently investigating an issue where some customers are experiencing issues logging into the SDM client with the error "This email doesn't have a StrongDM account" Information will be added here as it's discovered.

Engineering is actively working to return functionality for impacted customers.

The problem has been found and a fix implemented. StrongDM is monitoring the situation until resolution is confirmed.

This incident has been resolved. If customers are still seeing individual clients not able to connect please reach out to support for help: https://help.strongdm.com/hc/en-us/requests/new

StrongDM Engineering was alerted to and responded to an issue affecting background task processing within our platform. Core application functionality remained operational if slow. The issue was resolved as of 10am Pacific.

A small portion of organizations using Google SSO were being incorrectly redirected when attempting to login through the Desktop client. We have rolled back affected clients. If users are still experiencing issues, close the Desktop client and reopen it for the new update to apply.

On March 4th at approximately 2:30pm PST, we experienced an unusual load event that caused degraded service for 30 minutes. We experienced a follow up event later that day at 11pm PST for 30 minutes again. During this time, the service was slower than usual to respond to requests and some session connections would fail. We have implemented mitigations and are currently monitoring.

Timeline: * At 10:38 PM UTC 2025-03-04 we received a first notification from our alerting system about problems with our messaging infrastructure being overwhelmed by the number of requests * At 11:06 PM UTC 2025-03-04 the system recovered.  * Engineers identified the issue and created a ticket to address it. * At 6:56 AM UTC 2025-03-05 we received another notification from our alerting system with the same problem as before * At 7:28 AM UTC 2025-03-05 the system recovered again. * At 2:27pm UTC 2025-03-05, we began working on mitigations and further investigation as the US team started their day. * At 7:23pm UTC 2025-03-05, mitigations were released to production and we saw significant performance improvements in our metrics during the next spike overnight. ‌ Root cause: * One of our message passing systems was being overloaded by messages due to a period of heavy use. * The specific messages that caused the issue were broadcasts for the creation of connections to local clients. * These messages were not being produced or consumed efficiently. * Once the messaging system was overloaded, it stopped accepting new messages for a brief period of time. * As consumers caught up, performance would improve. * All managed secret actions \(validation, rotation, retrieval\) depend on this messaging system. Once the system was overloaded, the managed secret actions would either time out or be dropped. ‌ Mitigation: * We dramatically improved the efficiency of both the production and consumption of these messages. We experienced a similar high load spike after the mitigation was released, and we were able to observe the mitigations were very effective.

The cause of the outage is due to a change in our production db. Engineering has identified the cause and is working to remediate the issue. Updates to follow.

A fix has been implemented and we are monitoring the results. This should be resolved. If you are still seeing issues please reach out to support: https://help.strongdm.com/hc/en-us/requests/new

This issue does not appear to have impacted customers using our EU and UK Control Planes. This outage impacted the US Control Plane only.

This incident was resolved at 13:49 UTC. An RCA will be posted here within a week.

On December 3rd, SDM released a server build that added a new index to the table tracking latency between nodes. While the build succeeded on the EU and UK Control Planes, it failed on the US Control Plane. Retrying the build on the US Control Plane left the index in an invalid state, which went undetected by our migration tools. The following day, December 4th, SDM released a server build that relied on the new index. Without the index, node-to-node latency was no longer stored, and approximately three minutes later, the routing system stopped attempting multi-hop routes. This issue was limited to the US Control Plane, as the index was successfully created in the EU and UK Control Planes. StrongDM has already taken steps with internal processes to ensure that this kind of issue does not repeat in the future. Thank you for your patience and understanding.