Twingate 中断历史

Our upstream provider for Linux packages is having an issue impacting Twingate's Linux package downloads.

We are continuing to investigate this issue.

This incident has been resolved.

We are currently investigating this issue.

We are continuing to investigate this issue.

We are continuing to investigate this issue. We have narrowed the scope of the issue to Control Plane - Authorization.

We are continuing to investigate this issue. We are seeing recovery and have updated the Control Plane - Authorization status to partial outage.

The issue has been fixed. We are still working with cloud provider on root cause.

We're continuing to work with our cloud provider to identify the root cause.

We are currently awaiting the completion of our cloud provider’s investigation into the issue. In the meantime, we continue to closely monitor the system, and all service metrics have returned to normal operating levels. At this time, we are closing this incident. We will publish a comprehensive incident report as soon as it becomes available.

# Incident Report – Authorization Service Degradation ## Components Impacted Control Plane – Authorization Service ## Summary On May 28, 2026, between 10:00 UTC and 12:30 UTC, Twingate experienced a degradation of its Authorization service that affected approximately 15% of active connections at peak impact. During the incident, authorization requests experienced elevated network latency. As request processing times increased, the Authorization service's effective capacity to handle incoming traffic was reduced, resulting in elevated error rates and intermittent authorization failures for a subset of customers. Twingate operates the Authorization service across multiple cloud regions in an active-active configuration. While the platform remained available throughout the event, the combination of increased request latency and reduced service capacity led to customer impact until additional capacity was provisioned and service performance stabilized. ## Root Cause The incident was triggered by elevated network latency affecting communication paths used by the Authorization service. As requests took longer to complete, individual service instances were able to process fewer requests than normal. This reduction in throughput exposed a limitation in our auto-scaling configuration, which primarily relied on CPU utilization to determine service capacity requirements. As request-processing workers spent more time waiting on network operations, CPU utilization declined even as request latency increased. As a result, the service scaled down during a period of elevated request latency, reducing available capacity and amplifying customer impact. Recovery efforts were further complicated by an unusually high rate of spot instance preemptions in two regions, which reduced available compute capacity during stabilization. ## Resolution Engineering teams mitigated the incident by manually increasing Authorization service capacity and expanding available cluster resources. As additional capacity came online, request latency and error rates returned to normal levels and service performance fully recovered. ## Corrective Actions ### Completed * Increased the baseline capacity of the Authorization service by raising the minimum number of service instances. * Increased baseline node capacity across affected Kubernetes clusters. * Implemented additional rate controls for unusually large authorization requests to better protect overall service availability during periods of elevated load. ### In Progress * Rebalance spot and on-demand node capacity to reduce sensitivity to spot instance interruptions. * Enhance auto-scaling policies to incorporate latency and service performance metrics in addition to CPU utilization. * Expand monitoring and alerting to better detect conditions where request latency increases while resource utilization decreases.

We are seeing an issue with relay443. We are investigating. Note: These are special relays that offer connectivity over port 443 and are used by a subset of customers. So majority of customers shouldn't be seeing an issue,

The issue has been identified and a fix is being implemented.

The fix has been rolled out and relays443 are now fully functional.

**Components impacted** * Data Plane - Relay 443 flows only **Summary** On May 9, 2026, a subset of Twingate customers experienced an interruption to network access affecting Relay 443 connectivity. The disruption lasted approximately 2 hours and 10 minutes and was caused by a configuration error introduced during a routine software update. We have resolved the issue and are taking concrete steps to prevent a recurrence. **Root Cause** Twingate regularly releases software updates to our relay infrastructure, deploying them in stages across groups to minimize risk. As part of this update cycle, we also migrated our deployment tooling to a newer version of our infrastructure management system. The newer tooling applies stricter configuration validation rules than its predecessor. While preparing the update, our team identified and corrected a configuration conflict this stricter validation had surfaced. However, the fix was shipped in the same release bundle as a second change that was designed to serve as a prerequisite — specifically, a change that prevents the stricter validation from modifying existing, live deployments. The result was that a critical network configuration — the definition that tells our relay nodes to accept connections on port 443 — was silently removed from existing cluster deployments. **Why wasn't the issue detected earlier with the initial rollout?** Our continuous smoke testing did not include end-to-end coverage for Relay 443 specifically, so no automated alert fired when previous groups were deployed. Because those early upgrade groups carried relatively low Relay 443 traffic, no customer-visible impact surfaced either, leaving the issue undetected for approximately 48 hours until the higher-traffic groups were rolled out. **Why wasn't the issue caught in lower environments?** The issue was identified in a lower environment and a fix was prepared. However, the fix was deployed in the same release as the problematic change rather than as a prerequisite, which prevented it from taking full effect on existing cluster upgrades. Subsequent deployments confirmed the issue was fully resolved. **Remediation** Once we confirmed the root cause, we re-deployed the corrected relay configuration — explicitly restoring the port 443 host mapping — across all cluster groups in sequence. We validated each group before proceeding to the next. Full recovery was confirmed at 11:55 UTC on May 9. **Corrective actions** Short-term: * **Completed:** Redeployed to production to confirm the fix is stable and the issue will not recur. * We are adding regional smoke tests for Relay 443 to match the coverage already in place for other relay deployments, ensuring failures like this are caught before impacting customers.

The AWS outage is affecting GemFury, a service we use to host our APT/RPM packages.

Our testing indicates that APT/RPM packages are now operational, though Gemfury continues to report degraded performance. We will continue to monitor.

This incident has been resolved.

We are currently investigating service issues related to our cloud provider. The disruption is affecting the Twingate service, including the Admin Console, Report Downloads, and Network Dashboards. We will continue to monitor the situation and provide updates as we learn more.

Our cloud provider has acknowledged a service issue. We are continuing to monitor the situation and will provide updates as we learn more.

Our cloud provider has indicated that their service is recovering. Impacted Twingate services are now back online, but we are currently processing a backlog of Events. We expect this to be fully resolved within the next few hours and will provide an update when processing is complete.

Our cloud provider has confirmed full recovery of their services, and our backlog of Events has now been fully processed. We consider this incident fully resolved.

The issue has been identified and a fix is being implemented.

A fix has been implemented and we are monitoring the results.

This incident is resolved now. We'll provide a post-mortem as soon as we have details.

### **Incident Duration** **UTC Time**: May 28, 2025, from **20:45 to 21:14** \(29 minutes\) ### **Components Impacted** **Control Plane Services** * Authentication \(Enterprise & Social\) * Multi-Factor Authentication \(MFA\) * Authorization * Connector Heartbeat **Management Plane Services** * Identity Provider Sync * Public API * Admin Console * Real-Time Updates * Client Log Upload * 3rd Party Integrations * Network Dashboards * DNS Filtering Dashboards ### **Summary** On May 28, 2025, Twingate experienced a service disruption that impacted multiple core services for approximately 29 minutes. The issue stemmed from a misconfiguration introduced during a rollout that updated routing rules in our global load balancer. A new routing rule was unintentionally prioritized, resulting in most API traffic being directed to a single backend. This overwhelmed the target service, leading to elevated error rates. Lower environments did not expose this issue due to their reduced traffic levels. The incident was promptly detected and the rollback was initiated quickly. However, recovery was delayed due to infrastructure dependencies that also relied on affected components. A manual intervention using local tooling ultimately restored service. ### **Root Cause** A routing misconfiguration caused a traffic imbalance that overwhelmed a backend service, leading to widespread API failures. ### **Resolution Timeline** * **20:45 UTC** – Incident begins. API traffic misrouted. * **20:48 UTC** – Incident detected and triage begins. * **20:52 UTC** – Rollback initiated. * **21:05 UTC** – Automated rollback process encounters delays. * **21:09 UTC** – Engineers execute manual fix. * **21:14 UTC** – Services fully recovered. ### **Corrective and Preventative Actions** ### **Short-Term \(Completed\)** * ✅ Routing rule corrected. * ✅ Alerts added to detect backend traffic anomalies in lower environments. * ✅ Improved resource scaling thresholds. ### **Mid-Term \(In Progress\)** * ⚙️ Decouple infrastructure tooling from service dependencies to ensure faster recovery options. ### **Long-Term \(Planned\)** * 🔄 Introduce backend sharding to isolate customer workloads and reduce incident impact. We sincerely apologize for the disruption and are taking actions to improve our systems' resilience. We appreciate your continued support and trust.

We are seeing higher latency than usual with our cloud provider's APIs to process data used for our dashboards and reports. They confirmed the issue with their service. We hope this should be resolved soon.

The Big Query service latencies have dropped and we see improvements with our internal dashboards.

All of internal alerts are cleared. You should not see delays/errors with dashboards and reports any longer.

We are currently investigating this issue.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

**Twingate Public RCA: March 21, 2025 Authentication/Authorization Incident** **Summary** On March 21, 2025, between 21:10 and 21:41 UTC, a subset of authentication and authorization requests to Twingate services experienced elevated error rates. The issue was mitigated by 21:41 UTC and services returned to normal operation. **What Happened** Twingate runs an active-active architecture across multiple Google Kubernetes Engine \(GKE\) clusters, balanced by a Global Load Balancer \(GLB\). At the onset of the incident, we observed anomalies in one of our clusters, and to protect overall system health, we proactively scaled down deployments in that cluster. This action shifted traffic to other clusters in the topology, primarily one that typically handles a lighter load and had been scaled accordingly. The target cluster began autoscaling as expected, but the increased traffic caused elevated error rates, which triggered our retry mechanisms across services. While these retries helped many requests succeed, they also increased the overall system load. Simultaneously, the cluster underwent a cloud provider-initiated update operation that caused pod restarts and reduced capacity. To stabilize the system, we reintroduced capacity in the previously affected cluster, rebalancing the traffic across regions. Once this occurred, error rates subsided and retries diminished. **Root Cause** Anomalous behavior from our cloud provider, including unexpected request timeouts at the load balancer level and instability during a cluster update, led to a cascade of retry traffic that temporarily overwhelmed parts of the system. We are actively investigating both the unexpected timeout configuration and the behavior of the cluster during the update with our cloud provider. **Corrective actions** Short-Term: * Continue collaboration with our cloud provider to understand the root cause of the unexpected timeouts and cluster update impact. * COMPLETED: Keep deployments on all regions with same number of replicas in HPA configuration * COMPLETED: Increase max node counts on cluster node auto-scalers to give us more room to scale up. * COMPLETED: Configure HPA to scale up faster

We're currently experiencing a temporary disruption with the DNS Activity Reports. The team is working on restoring access, and we'll have them back up shortly.

Recent DNS Activity is now restored and we are monitoring the system.

This incident has been resolved.

We're seeing increased errors when syncing trusted devices from Jamf. We're currently investigating.

This incident is still being investigated. It started after a recent upgrade performed by Jamf. We have reached out to Jamf on the issue and working with them on resolution.

Issue with syncing trusted devices from Jamf has been fixed.

We've detected an issue with DNS events ingestion affecting some of our customers who are not able to see DNS events when exporting to S3. We're currently looking for root cause.

We've identified the issue and are working on a fix.

DNS events are being ingested again and we're monitoring the system.

This incident has been resolved.

GitHub is currently experiencing an incident affecting its authentication - https://www.githubstatus.com/incidents/kz4khcgdsfdv Social login via Github is therefore temporarily not working until GitHub resolves their issues.

This incident has been resolved.

We are seeing issues with Secure DNS (Internet Security) Dashboards loading. We are investigating.

We are continuing to investigate this issue.

Issue has been identified and we're working on a fix.

Dashboards are gradually coming back online.

Dashboard are back to normal.

We are having an issue with S3 Sync of Internet Security events. Team is engaged and issue is being investigated.

There was an ingestion delay that's fixed . We see logs flowing again.

This incident has been resolved.

We are seeing issues with AWS S3 Sync of Internet Security events. We are currently investigating.

We are continuing to investigate this issue.

We have identified the issue and working on a fix. Will update once fix is deployed.

The fix has been pushed to production and we confirmed Internet Security Events are now syncing to S3 buckets again.

During the incident , some of our customers MFA tokens became invalid. Less than 15 of our customers were impacted by this incident. We identified the issue and will make this key rotation safer going forward so it doesn't impact any of our customers. Impact Duration: 12:47 - 14:40 UTC

On June 26, 2024, between 8:16 pm UTC and 8:28 pm UTC, Twingate experienced several database connectivity alerts due to a failed rollout of one of its components. The rollout was promptly reversed, and our existing reliability measures prevented any major disruption to customer traffic.

**Components impacted** Management: Public API Management: Admin Console ‌ **Summary** On June 26, 2024 between 20:16 and 20:24 UTC, Twingate’s SQL proxies restarted, causing a brief failure to a small percentage of calls made to our Public API \(Terraform, Pulumi, k8s Operator, etc.\) and to the Admin console. There was no impact to Clients or Connectors. A change to our SQL proxy deployments that was targeting staging and development environments was pushed to production due to a misconfiguration, causing our SQL proxy instances to restart. ‌ **Root cause** Due to a misconfiguration, a change to our SQL proxy deployments intended for staging and development environments was pushed to production, causing them to restart. ‌ **Corrective actions** Short Term: * Ensure that SQL proxy deployments are only pushed in a controlled manner by resuming the GitOps workflow manually. * Fix the misconfiguration in our GitOps deployment mechanism for our SQL proxy deployments and set the Helm chart version to a static value so that all upgrades are done in a controlled manner. * Enhance our SQL proxy Helm chart to reduce the impact to services during updates and upgrades.

Recent DNS Activity screen on the Admin console isn't available.

Recent DNS Activity screen on the Admin console is back up.

This incident has been resolved.

We are currently investigating this issue.

We are continuing to investigate this issue.

A software rollout has broken the logins with Github. We are working on rolling out a hotfix.

We have successfully rollout the HotFix for the issue with Github logins.

**Components impacted** Control Plane: Authentication **Summary** On April 30, 2024, between 15:44 and 18:38 UTC, users were unable to login to Twingate through GitHub. The Twingate Engineering team investigated the issue upon receiving support communications and found that logins with other identity providers were functioning normally, but GitHub logins were not working as expected. The problem was traced back to a software rollout that had inadvertently impacted GitHub logins. Engineering was able to create a fix and roll it out at 18:38 UTC, which restored the GitHub logins to its normal functionality. The team is investigating better ways to identify and prevent these types of issues from reaching production and address them as quickly as possible if they ever arise again. **Root cause** A recent package upgrade introduced a bug that impacted Twingate logins via GitHub. **Corrective actions** Short-term: * Increase our testing coverage to allow early detection of login issues for all supported identity providers in all environments. This will aid in early detection when software is deployed to lower, non-production environments. * Improve alerting for login issues.

Recent DNS Activity on admin is unavailable. We have identified the issue and working on a fix.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

**Components impacted** Management: Admin Console **Summary** On April 20, 2024, between 5:32 GMT and 6:57 GMT, Recent DNS Activity on the Admin Console became unavailable. Shortly after the incident began, the Twingate on-call team received alerts regarding abnormal database activity. Workers on the clusters that manages DNS filtering logs starting seeing errors from the logs API, leading to excessive retries and database writes. To mitigate the issue, the DNS Log Streaming workers were temporarily disabled. The root cause was identified as a malfunction in the DNS Filtering Log API caused by a problematic dependency upgrade. Consequently, viewing DNS filtering logs and analytics in the Admin Console was temporarily unavailable. A rollback of the update was issued, and normal operations were restored at 6:57 GMT after which DNS filtering logs and analytics were available in the Admin Console. **Root cause** The DNS Filtering Log API went down due to a bad dependency upgrade. **Corrective actions** Already completed: * Rectified Admin Console's infinite retry logic by enhancing the retrieval of DNS activity logs during error states. * Optimized DNS Log Streaming retry and database write procedures to reduce unnecessary operations when no events are returned from the DNS Filtering API Short-term: * Improve the dependency upgrade process for the DNS FIltering API