Azure East US Capacity Constraints Impacting Virtual Desktop Availability
Début 21 mai 2026 à 16:05 UTC · 11d 5h
Pending
identified
Dear Customer,
Workspot has identified an issue affecting the availability of Virtual Desktops hosted in the Azure East US region.
Microsoft has confirmed ongoing Azure Compute capacity constraints in this region. Customers attempting to create, update, resume, scale, start, or restart Virtual Machines may experience allocation failures.
Current Impact:
- VM lifecycle operations may intermittently fail
- Running desktops are not impacted unless restarted or redeployed
How to Check if You Are Impacted:
If your subscription is affected, details will be visible under:
-->Azure Portal → Service Health → Active / Maintenance Events
Workaround:
- Redeploy the Virtual Machine from Workspot Control to help bring it online
Recommendations:
- Try a different Availability Zone within East US
- Use an alternate Azure region with lower demand
Mitigation Options:
- Disable Pause/Resume (VM runs continuously; may increase cost)
- On-demand Capacity Reservation:
https://learn.microsoft.com/azure/virtual-machines/capacity-reservation-overview
- Reserved Instances (cost benefit + prioritized capacity):
https://learn.microsoft.com/azure/cost-management-billing/reservations/save-compute-costs-reservations
Ongoing Actions:
Workspot is actively engaged with Microsoft and monitoring the situation closely.
identified
Update from Microsoft:
Current Status:
Due to an increase in demand for compute resources in the region, there are capacity constraints on available compute infrastructure. This in turn resulted in service management operation and allocation failures for VM resources.
We are currently working on platform levers to restore healthy buffers in relation to capacity. We will continue to monitor the situation and supply updates accordingly.
An update will be provided on 28 May 2026, or as events warrant.
resolved
Update from Microsoft:
Current Status: Resolved
What went wrong and why?
Due to an increase in demand for compute resources in a particular zone in the region, there were capacity constraints on available compute infrastructure. This in turn resulted in service management operation and allocation failures for VM resources.
How did we respond?
We’ve load balanced the capacity demands and restored the capacity buffers to a healthy state. With the congestion resolved, services should now function as usual.
What happens next?
• Ensuring capacity for our customers is a top priority for Microsoft. We sincerely apologize for any impact this may have caused, but be assured, we are continuously improving tools around monitoring usage, forecasting stock out, maintaining healthy buffers and improving cycle times to reduce this type of issue from happening.
Intermittent Restarts of Control Dynos
Début 5 mai 2026 à 12:08 UTC · 16h 24m
Pending
Composants affectés
Workspot Control
investigating
We are observing intermittent dynos restarts reported by our platform provider, Heroku. At this time, we have not observed any application downtime. This activity is not related to any security incident.
Our engineering team is actively investigating the issue. We will continue to provide updates as we learn more.
investigating
We have not observed any dyno restarts in the last 30 minutes. The engineering team is closely monitoring the environment and continues to investigate the underlying cause of the earlier dyno restarts.
We will provide further updates as more information becomes available.
investigating
We have not observed any dyno restarts since the incident was reported, nor any application downtime.
The engineering team continues to monitor the environment closely and is investigating the underlying cause of the earlier dyno restarts.
We will provide further updates as more information becomes available.
investigating
No new issues have been observed. As a mitigation measure, the engineering team has implemented changes to improve platform stability.
The team is closely monitoring the environment and continues to investigate the underlying cause of the earlier instability.
The next update will be shared in one hour. We will provide further updates sooner if there are any changes.
investigating
No new update at this time.
The platform remains stable, and the engineering team continues to actively monitor the environment.
If there are no changes, the next update will be shared in the next two hours. We will provide an update sooner if there is any change in status.
investigating
There are no new updates at this time.
The platform remains stable, and the engineering team continues to monitor the environment.
If there are no changes, the next update will be shared in the next 2 hours. We will provide further updates sooner if needed.
resolved
Workspot investigated the restart of the Control Dynos reported by our platform provider. The system is stable and we are closing the incident on status, but continuing to actively monitor and investigate the root cause.
We will share the RCA and updates as soon as they become available. Thank you.
VM Connectivity Issues – Azure East US
Début 24 avril 2026 à 13:31 UTC · 3d 17h
IssuesIncident mineur
Composants affectés
Workspot Control
investigating
Workspot has identified an issue affecting accessibility of virtual desktops hosted in the Azure East US region.
Microsoft has confirmed an ongoing Azure platform incident impacting a subset of Virtual Machines and Virtual Machine Scale Sets in this region. As a result, some customers may experience loss of network connectivity or inability to access their VMs.
This Azure incident aligns with the connectivity issues currently being observed.
Workspot is monitoring the situation closely and is in communication with Microsoft regarding the incident status. We will provide updates as they become available.
investigating
Impact Statement: Starting at 11:39 UTC on 24 Apr 2026, a subset of customers using Virtual Machines and Virtual Machine Scale Sets in East US may experience issues with resources hosted in this region.
Current Status: We are aware of this issue and are actively investigating. The next update will be provided as events warrant.
IMPACTED SERVICE(S) AND REGION(S):
Service Name Region
Virtual Machine Scale Sets -->East US
Virtual Machines --> East Asia
investigating
Update from the Microsoft:
Service: Virtual Machine Scale Sets, Virtual Machines
Region: East US, East Asia
Impact Statement: Starting at 11:39 UTC on 24 Apr 2026, a subset of customers using Virtual Machines and Virtual Machine Scale Sets in East US may experience issues with resources hosted in this region.
Current Status: We are aware of this issue and are actively investigating. The next update will be provided as events warrant.
IMPACTED SERVICE(S) AND REGION(S):
Service Name Region
Virtual Machine Scale Sets -->East US
Virtual Machines --> East Asia
investigating
Microsoft has shared an update on the ongoing Azure platform incident impacting Virtual Machines and Virtual Machine Scale Sets in the East US region.
Microsoft identified the issue through automated monitoring, which detected an unusual increase in failures within the service supporting virtual machine operations in East US. Microsoft is currently investigating the underlying cause and is actively working toward mitigation.
Current Status:
The issue remains in progress and under investigation by Microsoft. Customers may continue to experience VM connectivity or access issues in East US, and related impact may also be seen in East Asia.
Microsoft indicated that the next update will be shared within approximately 60 minutes, or sooner if there are meaningful developments.
Workspot continues to monitor the situation closely and will provide further updates as new information becomes available from Microsoft.
investigating
Current Status:
At this time, no new update has been received from Microsoft beyond their previous communication. The incident remains under investigation on their side.
We are continuing to monitor the situation closely and are tracking customer impact, which currently includes multiple reported cases of virtual desktop connectivity issues and VM accessibility failures.
Next Update:
We will provide an update as soon as Microsoft shares further information or once there is any material change in status.
investigating
Microsoft has shared below update on the ongoing Azure platform incident impacting Virtual Machines and Virtual Machine Scale Sets in the East US region.
CURRENT STATUS:
We detected this issue through automated monitoring after identifying an unusual drop in service success rates. The issue was caused by a recent change, and rollback actions are in progress across availability zones (AZs). Customers may begin to see recovery if using resources in AZ03, with Availability Zone 01 currently in progress. Please refer to this documentation to understand the logical to physical availability zone mapping for your subscription.
However, there is no confirmed estimated time for full resolution at this moment.
The next update will be shared within the next 60 minutes, or as events warrant.
investigating
Microsoft has shared below update on the ongoing Azure platform incident impacting Virtual Machines and Virtual Machine Scale Sets in the East US region.
Current Status: Rollback is complete for Availability Zone (AZ) 01 and AZ03, and is still in progress for AZ02. Mitigation is expected to complete within the next 1 hour.
Customers should observe improvements to their services, and downstream services are also starting to recover.
NEXT STEPS:
The next update will be shared within 60 minutes, or as events warrant.
monitoring
Update: Microsoft has shared below update on the ongoing Azure platform incident impacting Virtual Machines and Virtual Machine Scale Sets in the East US region.
Current Status: Resolved
What happened?
Between 11:39 UTC on 24th April and 00:15 UTC on 25 April 2026, a platform issue resulted in an impact to a subset of Azure services in the East US region. Impacted customers experienced failures or delays when provisioning, scaling, or updating Azure resources. Intermittent connectivity issues may also be seen on existing running workloads in Virtual Machines and Azure Virtual Desktop sessions.
This issue is now mitigated. An update with more information will be provided shortly.
Next Steps:
The next update will be shared once Microsoft provides the RCA. In the meantime, Workspot will continue to monitor impacted pools and virtual desktops to ensure stability and identify any residual issues.
resolved
Update:
We have received the Preliminary Post Incident Review(PIR) from Microsoft for the Azure East US control plane issue that impacted your environment on April 24.
Here is a summary of what happened and why:
STATUS: RCA
COMMUNICATION:
This is our Preliminary PIR to share what we know so far. After our internal retrospective is completed (generally within 14 days) we will publish a Final PIR with additional details.
What happened?
Between 11:30 UTC on 24 April and 00:15 UTC on 25 April 2026, customers may have experienced failures or delays when attempting to provision, scale, or update resources in East US. Beyond this, a smaller subset of impacted customers may have experienced intermittent connectivity issues on existing workloads (including Virtual Machines and Azure Virtual Desktop sessions) for scenarios dependent on unhealthy internal service dependencies.
The issue initially began with impact to a subset of customers in a single Availability Zone (physical AZ-01) but as demand shifted, similar symptoms were observed impacting a subset of customers in AZ-02 and AZ-03. While none of these zones were impacted for the full duration of the incident, customers experienced periods of impact in each zone for portions of the incident.
The following services were among those affected: Azure Application Gateway, Azure App Service, Azure Batch, Azure Cache for Redis, Azure Data Explorer, Azure Data Factory, Azure Databricks, Azure Health Data Services, Azure Kubernetes Service (AKS), Azure Red Hat OpenShift, Azure Service Fabric, Azure Synapse Analytics, Azure Virtual Desktop, Azure Virtual Machines, Azure Virtual Network Manager, Azure VMware Solution, Oracle Database@Azure, Virtual Machine Scale Sets – and potentially additional services that were dependent on new compute allocations in the region.
Note: Logical availability zones assigned to customer subscriptions may map to different physical availability zones. Customers can use the Locations API to understand this mapping: https://learn.microsoft.com/rest/api/resources/subscriptions/list-locations?HTTP#availabilityzonemappings.
What went wrong and why?
The Azure PubSub service is a key component of the networking control plane, acting as an intermediary between resource providers and networking agents on Azure hosts. Resource providers, such as the Network Resource Provider, publish customer configurations during Virtual Machine or networking create, update, or delete operations. Networking agents (subscribers) on the hosts retrieve these configurations to program the hosts networking stack. Additionally, the service functions as a cache, ensuring efficient retrieval of configurations during VM reboots or restarts. This capability is essential for deployments, resource allocation, and traffic management in Azure Virtual Network (VNet) environments.
During normal platform operations, one partition of this PubSub service in AZ-01 became unhealthy and automatically attempted to fail over to a secondary replica. The failover did not complete successfully, resulting in a partial loss of control plane availability within AZ-01. We intervened to investigate and attempted a manual failover of the primary partition, but this attempt was also unsuccessful.
Shortly afterward, we observed a similar condition in AZ-03, which led to a partial loss of control plane availability in AZ-03 as well. As the investigation progressed, we suspected that a previously deployed update to a regional control plane dependency had introduced a latent regression. This issue did not surface during earlier validation and only manifested when failover conditions were triggered under sustained production load.
As part of our mitigation efforts, we identified a version from the prior week that represented a Last Known Good (LKG) state. We first applied this rollback in AZ-03, which successfully restored control plane service health in that zone. Based on this, we began rolling back the affected components in AZ-01. By design, rollback operations are executed in stages by Azure Fabric controllers using update domains to ensure platform safety, while recovery proceeds incrementally.
While mitigation was in progress, the platform was unable to maintain two healthy instances of the PubSub service across availability zones simultaneously, which is a requirement for normal replication and control plane operations. This resulted in a loss of quorum of the service. As the system attempted to rebalance, impact shifted between availability zones, leading to periods of degraded behavior across multiple zones.
Similar failure patterns began to appear in AZ-02 and again in AZ-03, expanding the scope of impact across the region. For AZ-02 we initiated and completed a rollback, and although AZ-03 had previously shown recovery following the rollback, subsequent instability indicated that the rollback in that zone had not fully completed, because of an orchestration fault. As impact reemerged, rollback operations in AZ-03 were restarted and then completed, fully restoring service health.
How did we respond?
• 11:30 UTC on 24 April 2026 – Customer impact began. We observed failures or delays when customers attempted to provision, scale, or update resources in the affected region.
• 11:38 UTC on 24 April 2026 – We detected an issue in AZ-01. A control plane partition became unhealthy and automatic failover attempts did not complete successfully.
• 11:38–13:40 UTC on 24 April 2026 – We attempted manual failover in AZ-01. These efforts did not successfully restore service.
• 13:40 UTC on 24 April 2026 – We identified a recently deployed update as the likely cause of the issue.
• 13:50 UTC on 24 April 2026 – We began observing similar symptoms in AZ-03, indicating the issue was affecting multiple availability zones.
• 14:07 UTC on 24 April 2026 – We initiated rollback to a previously known good version in AZ-03.
• 15:03 UTC on 24 April 2026 – We observed significant recovery in AZ-03. Control plane availability exceeded 99%.
• 15:04 UTC on 24 April 2026 – We initiated rollback actions to AZ-01.
• 18:52 UTC on 24 April 2026 – We observed significant improvement in AZ-01 as rollback progressed.
• 19:02 UTC on 24 April 2026 – We confirmed AZ-01 had recovered to greater than 99% availability, while the rollback continued in the background.
• 19:05 UTC on 24 April 2026 – We observed similar symptoms in AZ-02 as load redistributed across the region.
• 19:10 UTC on 24 April 2026 – We initiated rollback to a known good version in AZ-02.
• 21:02 UTC on 24 April 2026 – We observed instability reappear in AZ-03. We determined this was because the rollback had not yet completed across all update domains. Consequently, we manually unblocked the rollback across remaining update domains in AZ03 to ensure stable recovery.
• 22:39 UTC on 24 April 2026 – We confirmed rollback was fully completed in AZ-03.
• 23:22 UTC on 24 April 2026 – We confirmed rollback was fully completed in AZ-02, completing PubSub mitigation across all affected zones.
• 00:15 UTC on 25 April 2026 – We validated downstream service recovery and PubSub health across all zones in the region.
How are we making incidents like this less likely or less impactful?
• We have assessed the risk of occurrence in other high volume regions, and have taken steps to rollback this PubSub service in these regions out of an abundance of caution. (Completed)
• We are investing in improving our test coverage surrounding the failure cases and load patterns that contributed to this incident, to catch issues like this one before they reach production. (Estimated completion: TBD)
• We are working to reduce rollback complexity, to be able to mitigate issues like this more quickly in future. (Estimated completion: TBD)
• This is our Preliminary PIR to share what we know so far. After our internal retrospective is completed (generally within 14 days) we will publish a Final PIR with additional details.
How can customers make incidents like this less impactful?
• Consider using Availability Zones (AZs) to run your services across physically separate locations within an Azure region. To help services be more resilient to localized failures like this one (which predominantly impacted zones at different times) many Azure services support zonal, zone-redundant, and/or always-available configurations: https://docs.microsoft.com/azure/availability-zones/az-overview
• For mission-critical workloads, customers should consider a multi-region geodiversity strategy to avoid impact from incidents like this one that impacted a single region: https://learn.microsoft.com/azure/architecture/patterns/geodes and https://learn.microsoft.com/azure/well-architected/design-guides/regions-availability-zones
• More generally, consider evaluating the reliability of your applications using guidance from the Azure Well-Architected Framework and its interactive Well-Architected Review: https://aka.ms/AzPIR/WAF
• The impact times above represent the full incident duration, so are not specific to any individual customer. Actual impact to service availability varied between customers and resources – for guidance on implementing monitoring to understand granular impact: https://aka.ms/AzPIR/Monitoring
• Finally, consider ensuring that the right people in your organization will be notified about any future service issues – by configuring Azure Service Health alerts. These can trigger emails, SMS, push notifications, webhooks, and more: https://aka.ms/AzPIR/Alerts
Workspot Control Reported Five Minutes Downtime
Début 20 mars 2026 à 18:09 UTC · 3d 11h
Pending
Composants affectés
Workspot Control
investigating
Pingdom reported production was down for Five minutes(4:08 PM UTC - 4:13 PM UTC).
There was a 5 minute intermittent service outage by a subservice provider resetting their system.
The service automatically recovered.
We will share the RCA once we receive from the subservice provider.
resolved
The system has been stable and functioning normally for the past 48 hours. Our internal investigation found no application-level anomalies.
We are closing this incident while the root cause analysis (RCA) from our infrastructure provider is still ongoing. We will share updates as soon as they become available.
Thank you for your patience.
Workspot Control Experiencing Intermittent Availability
Début 2 février 2026 à 09:38 UTC · 1d 17h
Pending
Composants affectés
Workspot Control
investigating
We are currently experiencing intermittent availability issues with the Workspot Control service. The engineering team is actively working on this issue with top priority. We will share more details as they become available.
This is an application‑level issue and not a security incident.
We apologize for any inconvenience this may cause.
identified
Following the recent 26.1 release, we have identified an issue impacting the Workspot Control service. A newly introduced application feature is not behaving as expected, resulting in an increased volume of system requests. This has led to request queuing and intermittent restarts of the Control service.
Our engineering team is actively working on testing and implementing a fix with the highest priority to stabilize the service.
identified
The issue has been identified, and corrective measures are in place. Our engineering team is reviewing remaining impact areas and completing final validation to ensure service stability before rollout.
identified
Our engineering team is still reviewing the impact areas and completing validation to ensure service stability. We will share more details as they become available.
identified
Our engineering team continues to review the impact areas and complete validation to ensure service stability.
There are no new updates since our last communication.
identified
Control is actively accepting requests; however, responses may be intermittent due to a high volume of queued requests.
- Control login: Login requests may occasionally time out. Retrying may be successful.
- User login: Most users can establish a session. If an issue occurs, please retry.
Our Engineering team is also exploring a potential rollback option while ensuring data consistency. We are working to determine an ETA and will provide an update as soon as more information becomes available.
We appreciate your patience as we continue to improve service stability.
identified
The Engineering team is planning to roll back the release deployed over the weekend. We will continue to share status updates approximately every 30 minutes.
An ETA is currently being assessed, and we will provide further updates as more information becomes available.
We appreciate your patience as we work to restore service stability.
identified
The Engineering team is continuing to assess the rollback plan and review the code changes. An ETA is being evaluated based on the outcome of the review and associated testing.
We appreciate your patience as we work to restore service stability.
identified
There is no new update at this time. The Engineering team continues to review the rollback plan and assess next steps.
We will share another update as soon as more information becomes available. We appreciate your continued patience as we work to restore service stability.
identified
Engineering team has made internal progress reviewing the rollback code, and testing activities are ongoing.
We will share another update as soon as more information becomes available.
We appreciate your continued patience as we work to restore service stability.
identified
There is no new update at this time. Engineering team has made internal progress reviewing the rollback code, and testing activities are ongoing. We will share another update as soon as more information becomes available.
We appreciate your continued patience as we work to restore service stability.
identified
Engineering team is estimating 90 minutes to complete rollback testing and then a short maintenance window to rollback at approximately 1030 PM UTC. We will update as timeline firm up.
We appreciate your patience as we work to restore service stability.
monitoring
Engineering team has completed the rollback, and all services have now been successfully restored. We are continuing to monitor the environment and will keep you posted with any further updates.
We greatly appreciate your patience and cooperation.
monitoring
Following the rollback performed, no new issues have been reported. Customers have confirmed that the Workspot service is stable.
Workspot Control is operating as expected, and all requests are being handled properly.
monitoring
We are continuing to monitor for any further issues.
resolved
The service has been fully restored after the rollback was performed. We appreciate your patience.
A detailed Root Cause Analysis (RCA) will be shared once available.
Workspot Control Login Instability
Début 14 novembre 2025 à 15:57 UTC · 2h 33m
OutageIncident majeur
Composants affectés
Workspot Control
investigating
We are currently experiencing intermittent issues with Workspot Control logins and non-persistent VM logins. Our team is actively investigating the issue and has escalated the matter by raising a priority ticket with the platform vendor. We are working closely with them to resolve this as quickly as possible. Further updates will be shared as soon as more information becomes available.
monitoring
Update: We are seeing improvement in both Workspot Control logins and non-persistent VM logins. At this time, we are closely monitoring the situation to ensure stability. Our investigation is ongoing, and we will share additional details as they become available.
resolved
This issue has been resolved, and customers can now log in to Workspot Control and access non-persistent VMs without any issues. The platform is stable, and normal operations have resumed. We will continue to monitor the environment to ensure stability.
We will share a post-incident summary once our investigation ends and relevant details are available.
AWS Operational Issues - No Impact - Workspot is closely monitoring the situation
Début 20 octobre 2025 à 17:11 UTC · 10d 13h
Pending
Composants affectés
Workspot Control
monitoring
We would like to inform you that Amazon Web Services (AWS) has been experiencing some operational issues. At this time, there is no operational impact to Workspot customers or services.
Our engineering and operations teams are closely monitoring the situation to ensure continued stability and performance across all Workspot systems. In the event of any changes or potential service impact, we will provide timely updates.
Latest Updates from AWS:
--> Increased Error Rates and Latencies
Oct 20 10:03 AM PDT: We continue to apply mitigation steps for network load balancer health and recovering connectivity for most AWS services. Lambda is experiencing function invocation errors because an internal subsystem was impacted by the network load balancer health checks. We are taking steps to recover this internal Lambda system. For EC2 launch instance failures, we are in the process of validating a fix and will deploy to the first AZ as soon as we have confidence we can do so safely. We will provide an update by 10:45 AM PDT.
Oct 20 9:13 AM PDT: We have taken additional mitigation steps to aid the recovery of the underlying internal subsystem responsible for monitoring the health of our network load balancers and are now seeing connectivity and API recovery for AWS services. We have also identified and are applying next steps to mitigate throttling of new EC2 instance launches. We will provide an update by 10:00 AM PDT.
Link --> https://health.aws.amazon.com/health/status
We will continue to monitor the situation and provide updates as necessary. Thank you for your patience and cooperation.
resolved
This incident has been resolved, and no Workspot customers have reported experiencing the issue.
Here is the summary of the AWS incident: https://aws.amazon.com/message/101925/
Users are experiencing Authentication failures while using Workspot Windows Client.
Début 15 octobre 2025 à 18:58 UTC · 15d 11h
Pending
Composants affectés
Workspot Control
identified
Issue Summary :
A subset of users attempting to log in to the Workspot Windows Client are experiencing authentication failures after entering the MFA verification code.
Root Cause :
The issue has been traced to the installation of the Preview Patch KB5065789 or the October 2025 Windows Cumulative Update. This update appears to interfere with the authentication flow used by the Workspot Windows Client.
Workaround :
>>Option 1: To restore normal login functionality, affected users should uninstall KB5065789 and the October 2025 Cumulative update from their local Windows device and perform a system reboot. This action has been confirmed to resolve the issue in multiple cases.
>>Option 2: As an alternative, users can access Workspot resources using the Web Client, which remains fully functional and unaffected by this issue.
Next Steps :
We recommend IT administrators review devices for the presence of KB5065789 and coordinate its removal where login issues are reported.
These workarounds will allow users to log in and continue working until we provide a permanent resolution.
Thank you for your patience and understanding. We truly appreciate your cooperation.
identified
Microsoft has confirmed that the issue is caused by recent Windows updates — specifically KB5065789, KB5066835, and KB5066131 — which interfere with the HTTP.sys component used in server-side applications. This affects environments running Windows Server 2025 and Windows 11, and disrupts the authentication flow used by the Workspot Windows Client.
Resolution Steps (as per Microsoft):
Microsoft recommends the following actions to mitigate the issue:
- Step 1: On the affected machine, go to Settings > Windows Update and click “Check for updates.”
- Step 2: Restart the machine, regardless of whether new updates are found. This reboot helps restore normal functionality.
For air-gapped environments, Microsoft has provided a manual rollback installer:
Download Known Issue Rollback MSI https://download.microsoft.com/download/16d61dc0-7e94-4cd2-ba3c-4f59dece8488/Windows 11 24H2, Windows 11 25H2 and Windows Server 2025 KB5066835 251015_22001 Known Issue Rollback.msi
Next Steps:
We recommend IT administrators:
- Review devices for the presence of the affected KB updates.
- Apply the Microsoft-recommended mitigation steps.
- Use the Web Client as a temporary workaround if needed.
We will continue to monitor the situation and provide updates as necessary. Thank you for your patience and cooperation.
resolved
This issue has been resolved in accordance with the Microsoft article: https://learn.microsoft.com/en-us/windows/release-health/status-windows-11-25h2#issue-details.
The problem is addressed in the October Windows non-security preview update (KB5067036) and all subsequent updates. Microsoft recommends installing the latest Windows updates, as they include important improvements and fixes, including this resolution.
Additionally, we have not received any further reports of this issue from customers who have applied the recommended Microsoft updates.
Failure in Azure East US and East US2 region
Début 10 septembre 2025 à 14:43 UTC · 6d 13h
Pending
Composants affectés
Workspot Control
identified
We are aware that some customers are experiencing issues with VM provisioning and resuming operations in the East US and East US 2 Azure regions.
Our team has already engaged Microsoft Azure Support with a Severity 1 ticket. According to Microsoft, the issue is related to unhealthy dependencies that are causing failures. They are actively investigating to determine the root cause and identify possible mitigation steps.
We will share further updates as soon as Microsoft provides more information.
identified
We are continuing to work with Microsoft
identified
As per the latest update from Microsoft, they have observed that two of the three impacted zones have returned to a healthy state, and failure rates are now trending downward. Customers may see signs of recovery over time.
This failure issue is limited to the East US2 region.
identified
Below is the summary provided by Microsoft regarding the ongoing issue in the East US 2 region:
"Current Status:
We detected the issue through automated monitoring following a spike in failure rates. The root cause has been traced to a backend service responsible for managing resource placement, which is experiencing performance degradation. This has led to delays and failures in resource creation and management.
Our engineering teams have attempted several recovery actions, including restarting key service components and shifting workloads away from affected infrastructure. However, these efforts have not yet fully resolved the issue due to system-level constraints.
Mitigation Actions Taken:
• Restarted critical service components to restore functionality.
• Attempted to reroute workloads from affected infrastructure.
• Initiated multiple recovery cycles for the impacted backend service.
Active Recovery Efforts:
• Zone 3 (Az03) is showing signs of improvement after targeted recovery actions. System performance has stabilized and is being closely monitored.
• Zone 2 (Az02) is undergoing similar recovery steps. While new resource deployments remain restricted, existing resources are beginning to recover.
• We are working to redistribute workloads to healthier zones (Az01 and Az03). However, limited capacity in these zones is causing throttling and delays.
• In parallel, we are exploring emergency capacity expansion to alleviate resource constraints and accelerate recovery.
Note that the 'logical' zones used by each customer subscription may correspond to different physical zones - customers can use the Locations API to understand this mapping, to confirm which resources run in this physical Availability Zone(AZ).
Next Steps:
We continue to monitor all zones and prioritize recovery in Az02 to restore capacity in the region."
identified
Below is the summary provided by Microsoft regarding the ongoing issue in the East US 2 region:
"Current Status:
We detected the issue through automated monitoring following a spike in failure rates. We have identified a performance issue in a core infrastructure component responsible for managing resource placement. This is causing delays and failures in virtual machine provisioning. The issue stems from severe transaction delays and high system load in two zones of the region.
Active Recovery Efforts:
• Zone 2 (Az02) - Gradually re-enabling traffic (~50%) to Zone 2 using controlled allocation strategies. This should be introducing more capacity for the region and allowing higher allocation success rates.
• Zone 3 (Az03) and Zone 1 (Az01) – Is recovered but due to partial traffic enabled for Zone 2, customers may still see allocation failures here. As Zone 2 traffic increases, allocation success should increase in all zones.
Note that the 'logical' zones used by each customer subscription may correspond to different physical zones - customers can use the Locations API to understand this mapping, to confirm which resources run in this physical AZ.
The next update will be provided within 60 minutes, or sooner if significant progress is made.
"
identified
As per Microsoft, this issue has been mitigated. Here is the incident summary, shared by Microsoft
SUMMARY OF IMPACT:
What happened?
Between 09:12 UTC and 18:50 UTC on 10 September 2025, a platform issue resulted in an impact to multiple Azure services in the East US 2 region, more specifically two zones (Az02 and Az03). Impacted customers may have experienced error notifications when performing service management operations - such as create, delete, update, scaling, start or stop - for resources hosted in this region. The primary impacted service affected was Virtual Machines or Virtual Machines Scale Sets, but this would have resulted in issues for services dependent upon such Compute resources, such as Azure Databricks, Azure Kubernetes Service, Azure Synapse Analytics, Backup, and Data Factory.
Customers that still see failed or unhealthy resources should attempt to update or redeploy the resource.
What do we know so far?
Our investigation identified that the issue impacting resource provisioning in East US 2 was linked to a failure in the platform component responsible for managing resource placement. The system is designed to recover quickly from transient issues, but in this case, the prolonged performance degradation caused recovery mechanisms themselves to become a source of instability.
The incident was primarily driven by a combination of platform recovery behavior and sustained performance degradation. While customer-generated load remained within expected limits, internal platform services began retrying failed operations aggressively when performance issues emerged. These retries, intended to support resilience, instead created a surge in internal system activity.
How did we respond?
• 09:12 UTC on 10 September 2025 – Customer impact began.
• 09:13 UTC on 10 September 2025– Our monitoring systems observed a rise in failure rates, triggering an alert and prompting our team to initiate an investigation.
• 12:08 UTC on 10 September 2025 – We identified unhealthy dependencies in core infrastructure components as initial contributing factors.
• 13:34 UTC on 10 September 2025 – Began mitigation efforts that included - Restarted critical service components to restore functionality, reroute workloads from affected infrastructure, initiated multiple recovery cycles for the impacted backend service, on recovery, internal workloads processed through backlogs to get to the current healthy state, and executed capacity operations to free up resources.
• 18:50 UTC on 10 September 2025 – After a period of monitoring to validate the health of services, we were confident that the control plane service was restored, and no further impact was observed to downstream services for this issue.
monitoring
A fix has been implemented and we are monitoring the results.
resolved
Microsoft has informed us that the outage in the region has been mitigated. Please, find the summary below, as per Microsoft:
STATUS: Mitigated 9/10/2025 9:55:15 PM UTC
SUMMARY OF IMPACT:
What happened?
Between 09:12 UTC and 18:50 UTC on 10 September 2025, a platform issue resulted in an impact on multiple Azure services in the East US 2 region, more specifically, two zones (Az02 and Az03). Impacted customers may have experienced error notifications when performing service management operations, such as creating, deleting, updating, scaling, starting, or stopping, for resources hosted in this region. The primary impacted service was Virtual Machines or Virtual Machine Scale Sets, but this would have resulted in issues for services dependent on such Compute resources, such as Azure Databricks, Azure Kubernetes Service, Azure Synapse Analytics, Backup, and Data Factory.
Customers who still see failed or unhealthy resources should attempt to update or redeploy the resource.
What do we know so far?
Our investigation identified that the issue impacting resource provisioning in East US 2 was linked to a failure in the platform component responsible for managing resource placement. The system is designed to recover quickly from transient issues, but in this case, the prolonged performance degradation caused recovery mechanisms themselves to become a source of instability.
The incident was primarily driven by a combination of platform recovery behavior and sustained performance degradation. While customer-generated load remained within expected limits, internal platform services began retrying failed operations aggressively when performance issues emerged. These retries, intended to support resilience, instead created a surge in internal system activity.
How did we respond?
09:12 UTC on 10 September 2025 – Customer impact began.
09:13 UTC on 10 September 2025– Our monitoring systems observed a rise in failure rates, triggering an alert and prompting our team to initiate an investigation.
12:08 UTC on 10 September 2025 – We identified unhealthy dependencies in core infrastructure components as initial contributing factors.
13:34 UTC on 10 September 2025 – Began mitigation efforts that included - Restarted critical service components to restore functionality, rerouted workloads from affected infrastructure, initiated multiple recovery cycles for the impacted backend service, on recovery, internal workloads processed through backlogs to get to the current healthy state, and executed capacity operations to free up resources.
18:50 UTC on 10 September 2025 – After a period of monitoring to validate the health of services, we were confident that the control plane service was restored, and no further impact was observed on downstream services for this issue.
What happens next?
Our team will be completing an internal retrospective to understand the incident in more detail. We will publish a Preliminary Post Incident Review (PIR) within approximately 72 hours to share more details on what happened and how we responded. After our internal retrospective is completed, generally within 14 days, we will publish a Final Post Incident Review with any additional details and learnings.
To get notified when that happens, and/or to stay informed about future Azure service issues, make sure that you configure and maintain Azure Service Health alerts – these can trigger emails, SMS, push notifications, webhooks, and more: https://aka.ms/ash-alerts
For more information on Post Incident Reviews, refer to https://aka.ms/AzurePIRs
The impact times above represent the full incident duration, so they are not specific to any individual customer. Actual impact to service availability may vary between customers and resources – for guidance on implementing monitoring to understand granular impact: https://aka.ms/AzPIR/Monitoring
Finally, for broader guidance on preparing for cloud incidents, refer to https://aka.ms/incidentreadiness
Stay informed about your Azure services
- Visit Azure Service Health to get your personalized view of possible impacted Azure resources, downloadable Issue Summaries, and engineering updates.
- Set up service health alerts to stay notified of future service issues, planned maintenance, or health advisories.
GCP Service Alert
Début 12 juin 2025 à 19:10 UTC · 1h 52m
Pending
identified
GCP is reporting 11:46 PDT "Multiple GCP products are experiencing impact due to Identity and Access Management Service Issue".
https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1SsW
monitoring
GCP update 12:09 PDT "Our engineers are continuing to mitigate the issue and we have confirmation that the issue is recovered in some locations."
monitoring
GCP: 12:30 PDT All locations except us-central1 have fully recovered. us-central1 is mostly recovered. We do not have an ETA for full recovery in us-central1.
monitoring
GCP 1241 PDT:
"Our engineers have identified the root cause and have applied appropriate mitigations.
While our engineers have confirmed that the underlying dependency is recovered in all locations except us-central1, we are aware that customers are still experiencing varying degrees of impact on individual google cloud products. All the respective engineering teams are actively engaged and working on service recovery.
We do not have an ETA for full service recovery."
monitoring
GCP 13:16 PDT
We have identified the root cause and applied appropriate mitigations. Our infrastructure has recovered in all regions except us-central1.
resolved
GCP 14:00 PDT
We have implemented mitigation for the issue in us-central1 and multi-region/us and we are seeing signs of recovery.
Please see GCP's status page for more details. https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1SsW
Workspot Control Affected Due to Downstream Platform Issue
Début 10 juin 2025 à 07:34 UTC · 2d 12h
Pending
Composants affectés
Workspot Control
investigating
We are currently investigating this issue. We have escalated it by raising a priority ticket with the platform vendor and are working with them to resolve it as quickly as possible. We will provide updates as we learn more.
investigating
Update: The issue is still ongoing, and our team is actively working on it with the highest priority. We will share further updates as we receive more information.
investigating
Update: The issue is ongoing, and our team is still working closely with the platform vendor to resolve it. We understand the significant impact this has on our customers. We remain committed to restoring service as quickly as possible and will continue to provide updates as we receive more information.
investigating
Update: The issue is still ongoing. We are actively following-up with platform vendor. We will keep you updated in regular interval or as soon as we have more information.
investigating
Heroku (Salesforce) platform is working through intermittent issue since 0603 UTC affecting multiple Heroku customers. For Workspot, this has impacted certain services that resume VMs and broker user connections to desktops. At this time, some customers will have access to persistent desktops.
Users are able to connect to persistent desktops that were powered on (listed as ready, online or disconnected) at the time of impact.
Engineering is working on:
1. Verifying that Admins may be able to power on "paused" VMs so that users can connect by selecting "Reboot" VMs from the Control UI or API service.
2. Verifying disabling pause/resume service so no VMs will pause or stop running until Heroku is recovered.
3. Verifying access to non persistent desktop and applications.
Workspot Support and Operations will be continuing to communicate updates via status.workspot.com every 60 minutes.
investigating
Update: Engineering is still working on:
1. Verifying that Admins may be able to power on "paused" VMs so that users can connect by selecting "Reboot" VMs from the Control UI or API service.
2. Verifying disabling pause/resume service so no VMs will pause or stop running until Heroku is recovered.
3. Verifying access to non persistent desktop and applications.
investigating
Heroku (Salesforce) platform is still working through resolving the intermittent issues since 0603 UTC, affecting multiple Heroku customers.
Workspot is applying mitigations to workaround and limit the impact of the Heroku outage. Some customers are seeing the restoration of their service.
We are still monitoring the environment as Heroku is still working through the process.
Workspot Support and Operations will be continuing to communicate updates via status.workspot.com every 60 minutes.
monitoring
Heroku (Salesforce) platform has been working through intermittent issues since 0603 UTC, affecting multiple Heroku customers.
Workspot has applied mitigations to limit the impact of the Heroku outage. Customers are seeing the restoration of the service for persistent, non-persistent, and applications. Please try accessing your desktops now. Let us know if you have any issues connecting to your desktops.
Customer admins don’t have to reboot the VMs at this time per earlier instructions.
We are still monitoring the environment as Heroku is still working through the process.
Workspot Support and Operations will be continuing to communicate updates via status.workspot.com every 60 minutes.
monitoring
Heroku (Salesforce) has resolved their issue.
We are still monitoring the environment to ensure the Heroku updates are stable. Our internal systems are showing green on functionality and metrics. Our customers are reporting that services have been restored.
We’ll continue to monitor and update as more information and RCA is made available to us.
resolved
Workspot systems are fully recovered from this incident.
postmortem
Heroku Salesforce posted the [official summary](https://www.heroku.com/blog/summary-of-june-10-outage/) and preliminary [Root Cause Analysis \(RCA\)](https://help.salesforce.com/s/articleView?language=en_US&id=005036674&type=1).
Workspot is reviewing the Heroku findings and will provide customers update in a few days.
Azure Service Alert – East US Region VMs might be impacted
Début 29 mai 2025 à 14:03 UTC · 22h 26m
Pending
Composants affectés
Workspot Control
identified
Issue Summary:
Start Time: 09:15 UTC on 29 May 2025
Impact: Errors may occur during service management operations (create, delete, update, scale, start, stop) for VMs.
Cause: A sudden spike in usage has caused backend VM components to hit operational limits, resulting in delays and failures.
Current Status: Microsoft is mitigating the issue by failing over to a healthy backend instance.
Please monitor the updates in Azure subscription under Service Health, if your resources are in East US region.
resolved
As per Microsoft, this issue has been resoved. Here is the incident summary, shared by Microsoft:
Issue Summary:
Between 09:07 UTC and 16:25 UTC on 29 May 2025, a platform issue resulted in an impact to the following services in the East US region:
- Virtual Machines & Virtual Machine Scale Sets: Error notifications when performing service management operations - such as create, delete, update, scaling, start, stop - for resources hosted in this region. This impact was restricted to a single Availability Zone (AZ01), Physical AZ01. Retries may have been successful.
- Azure Synapse Analytics: Issues while executing Spark jobs through Synapse Pipelines or Notebooks, encountering the error code "CLUSTER_CREATION_TIMED_OUT". Retries may have been successful.
- Azure Data Factory: Activity or Pipeline run failures and delays due to dataflow activity failures.
MS Response Timeline:
- 09:07 UTC: Customer impact began.
- 09:12 UTC: Auto-recovery attempts started, including load-shedding and failover.
- 09:15 UTC: Service monitoring detected spikes in VM failures; investigation began.
- 11:45 UTC: Platform engineers terminated problematic service instances to free compute resources.
- 12:53 UTC: Services started processing backlogged VM requests, with some customers still seeing timeouts and throttling.
- 13:15 UTC: Engineers redirected VM deployment traffic to alternate management services to speed recovery.
- 13:48 UTC: Failover progress noted, backlog began draining.
- 13:58 UTC: Azure Data Factory service restored.
- 14:09 UTC: Azure Synapse Analytics service restored.
- 16:25 UTC: All services fully restored; customer impact mitigated.
Workspot API service issue
Début 7 avril 2025 à 15:52 UTC · 13h 7m
Pending
Composants affectés
Workspot Control
investigating
Workspot Service is in maintenance mode
monitoring
The Engineering team has identified the issue and reverted the Control changes implemented over the weekend at approximately 16:00 UTC. Since the rollback, no further issues have been observed. We continue to actively monitor the environment.
resolved
This incident has been resolved. No issues have been observed since the rollback was performed.
Azure Network Infrastructure - Issues accessing a subset of Microsoft services
Début 30 juillet 2024 à 13:36 UTC · 8d 2h
Pending
investigating
Microsoft updated its status page that they are investigating reports of issues connecting to Microsoft services globally. Customers may experience timeouts connecting to Azure services.
Please refer to https://azure.status.microsoft/en-us/status for the latest update.
investigating
Microsoft updated the status page with the below details:
We have implemented networking configuration changes and have performed failovers to alternate networking paths to provide relief. Monitoring telemetry shows improvement in service availability from approximately 14:10 UTC onwards, and we are continuing to monitor to ensure full recovery.
monitoring
As per Microsoft, they have implemented networking configuration changes, telemetry shows improvement in service availability.
resolved
The issue has been mitigated. Microsoft have shared the below preliminary post-incident review regarding this incident (Tracking ID: KTY1-HW8):
This is our Preliminary PIR that we endeavor to publish within 3 days of incident mitigation to share what we know so far. After our internal retrospective is completed (generally within 14 days) we will publish a "Final" PIR with additional details/learnings.
What happened?
Between 11:45 and 13:58 UTC on 30 July 2024, a subset of customers experienced intermittent connection errors, timeouts, or latency spikes while connecting to Microsoft services that leverage Azure Front Door (AFD) and Azure Content Delivery Network (CDN). The two main impacted services were Azure Front Door (AFD) and Azure Content Delivery Network (CDN), and downstream services that rely on these – including the Azure portal, and a subset of Microsoft 365 and Microsoft Purview services. From 13:58 to 19:43 UTC, a smaller set of customers continued to observe a low rate of connection timeouts.
What went wrong and why?
Azure Front Door (AFD) is Microsoft's scalable platform for web acceleration, global load balancing, and content delivery, operating in nearly 200 locations worldwide – including datacenters within Azure regions, and edge sites. AFD and Azure CDN are built with platform defenses against network and application layer Distributed Denial-of-Service (DDoS) attacks. In addition to this, these services rely on the Azure network DDoS protection service, for the attacks at the network layer. You can read more about the protection mechanisms at https://learn.microsoft.com/azure/ddos-protection/ddos-protection-overview and https://learn.microsoft.com/azure/frontdoor/front-door-ddos.
Between 10:15 and 10:45 UTC, a volumetric distributed TCP SYN flood DDoS attack occurred at multiple Azure Front Door and CDN sites. This attack was automatically mitigated by the Azure Network DDoS protection service and had minimal customer impact.
At 11:45 UTC, as the Network DDoS protection service was disengaging and resuming default traffic routing to the Azure Front Door service, the network routes could not be updated within one specific site in Europe. This happened because of Network DDoS control plane failures to that specific site, due to a local power outage. Consequently, traffic inside Europe continued to be forwarded to AFD through our DDoS protection services, instead of returning directly to AFD. This event in isolation would not have caused any impact.
However, an unrelated latent network configuration issue caused traffic from outside Europe to be routed to the DDoS protection system within Europe. This led to localized congestion, which caused customers to experience high latency and connectivity failures across multiple regions. The vast majority of the impact was mitigated by 13:58 UTC, around two hours later when we resolved the routing issue. A small subset of customers without retry logic in their application may have experienced residual effects until 19:43 UTC.
How did we respond?
Our internal monitors detected impact on our Europe edge sites at 11:47 UTC, immediately prompting a series of investigations. Once we identified that the network routes could not be updated within that one specific site, we updated the DDoS protection configuration system to avoid traffic congestion. These changes successfully mitigated most of the impact by 13:58 UTC. Availability returned to pre-incident levels by 19:43 UTC once the default network policies were fully restored.
How we are making incidents like this less likely or less impactful
- We have already added the missing configuration on network devices to ensure a DDoS mitigation issue in one geography cannot spread to other geographies in the Europe region which resulted in traffic redirection. (Completed)
- We are enhancing our existing validation and monitoring in the Azure network, to detect invalid configurations. (Estimated completion: November 2024)
- We are improving our monitoring where our DDoS protection service is unreachable from the control plane, but is still serving traffic. (Estimated completion: November 2024)
- This is our Preliminary PIR that we endeavor to publish within 3 days of incident mitigation to share what we know so far. After our internal retrospective is completed (generally within 14 days) we will publish a "Final" PIR with additional details/learnings.
How can customers make incidents like this less impactful
- For customers of Azure Front Door/Azure CDN products, implementing retry logic in your client-side applications can help handle temporary failures when connecting to a service or network resource during mitigations of network layer DDoS attacks. For more information, refer to our recommended error-handling design patterns: https://learn.microsoft.com/azure/well-architected/resiliency/app-design-error-handling#implement-retry-logic.
- Applications that use exponential-backoff in their retry strategy may have seen success, as an immediate retry during intervals of high packet loss may have also seen high packet loss. A retry conducted during periods of lower loss would likely have succeeded. For more details on retry patterns, refer to https://learn.microsoft.com/azure/architecture/patterns/retry.
- More generally, consider evaluating the reliability of your applications using guidance from the Azure Well-Architected Framework and its interactive Well-Architected Review: https://docs.microsoft.com/azure/architecture/framework/resiliency.
- Finally, ensure that the right people in your organization will be notified about any future service issues by configuring Azure Service Health alerts. These alerts can trigger emails, SMS, push notifications, webhooks, and more: https://aka.ms/ash-alerts
How can we make our incident communications more useful?
You can rate this PIR and provide any feedback using our quick 3-question survey: https://aka.ms/AzPIR/KTY1-HW8
Crowdstrike Issue - General Update.
Début 22 juillet 2024 à 22:43 UTC · 7d 11h
Pending
monitoring
Workspot has identified specific customers with virtual machines impacted by the Crowdstrike and Microsoft issue. The Workspot service is 100% operational, not impacted, and automatically detected the issues affecting customer VMs.
We are actively working with customers affected by this outage to restore their service. If you are impacted by this issue, and we have contacted you, please respond to the communication so that we can schedule time with you to resolve the issue. If you have not received an email and are affected by this outage, please open a support ticket through Control so that we can help. Please allot approximately 30 mins to restore each VM. This is the average time we are currently taking.
While this global outage is not a Workspot issue, we take pride in maintaining our high level of service for our customers. We will continue to monitor this issue this week.
Thank you for understanding and patience.
resolved
The issue has been resolved by fixing the affected Virtual Desktop or providing the steps to our customers to recover the affected desktop.
PaaS Provider issue impacting Agent communications with Workspot Cloud
Début 22 avril 2024 à 21:44 UTC · 53m
Pending
Composants affectés
Workspot ControlWorkspot Watch
investigating
We are currently investigating the following issues:
1. Control and Watch may not show telemetry data (i.e. CPU, Memory, RTT)
2. Gateways may show offline status in Control and Watch.
Users can still connect as normal and are not impacted.
monitoring
A fix has been implemented and we are monitoring the results.
resolved
This incident has been resolved.
Workspot Agents and Enterprise Connectors failing to connect via the new URL after Control 18.2 update
Début 20 avril 2024 à 18:25 UTC · 2d 3h
Pending
Composants affectés
Workspot Control
monitoring
Workspot Control was updated to 18.2 on April 20, 2024. Post update, Workspot observed that some Workspot Agents and Workspot Enterprise Connectors failed to connect to Control via the new https://control.us.workspot.com URL. This was presumably caused by firewalls blocking *.us.workspot.com.
If this was happening to you, you might have seen the following symptoms:
• Enterprise Connectors was showing as “Offline” in Workspot Control.
• Non-persistent desktops were showing as “Offline” and were not able to be assigned to new users (ongoing sessions are unaffected).
• Persistent desktops was showing as “Offline” but users were still connect to them.
Workspot has remediated the issue observed after the 18.2 update and monitoring the environment.
Please contact the Workspot Support for any questions.
monitoring
We are continuing to monitor for any further issues.
resolved
This incident has been resolved.
Workspot Watch System Reboot
Début 23 juin 2023 à 19:08 UTC · 43m
IssuesIncident mineur
Composants affectés
Workspot Control
identified
Workspot Watch may be loading slowly. Resolution in progress.
All other systems are fully operational.
resolved
This incident has been resolved.
Multiple Microsoft Services are down affecting Workspot Customers
Début 25 janvier 2023 à 08:47 UTC · 2h 24m
Pending
Composants affectés
Workspot Control
investigating
Microsoft reported a global outage for Office 365 & Azure Portal services. Customers using Azure AD for authentication may see problems while logging into the Workspot clients or launching their desktops.
You can monitor the status of the ongoing issue on the below Microsoft links:
- https://status.azure.com/
- https://status.office365.com/
We will keep you posted on the status periodically.
investigating
As per the latest update from Microsoft, starting at 07:30 UTC, they identified a networking issue impacting connectivity to Azure for a subset of users. They are actively investigating.
You can monitor the status of the ongoing issue on the below Microsoft links:
- https://status.azure.com/
- https://status.office365.com/
We will keep you posted on the status periodically.
identified
As per the latest Microsoft update on the status, they have determined the network connectivity issue is occurring with devices across the Microsoft Wide Area Network (WAN). This impacts connectivity between clients on the internet to Azure, as well as connectivity between services in data centers, as well as ExpressRoute connections. The issue is causing impact in waves, peaking approximately every 30 minutes.
They have identified a recent WAN update is the likely underlying cause, and they have taken steps to roll back this update. They are seeing signs of recovery across multiple regions and services, and are continuing to actively monitor the situation
You can monitor the status of the ongoing issue on the below Microsoft links:
- https://status.azure.com/
- https://status.office365.com/
We will keep you posted on the status periodically.
resolved
Microsoft confirmed that the issues have been mitigated.
From Microsoft:
Summary of Impact: Between 07:05 UTC and 09:45 UTC on 25 January 2023, customers experienced issues with networking connectivity, manifesting as network latency and/or timeouts when attempting to connect to Azure resources in Public Azure regions, as well as other Microsoft services including M365 and PowerBI.
Preliminary Root Cause: We determined that a change made to the Microsoft Wide Area Network (WAN) impacted connectivity between clients on the internet to Azure, as well as connectivity between services in different regions, as well as ExpressRoute connections.
Mitigation: We identified a recent change to WAN as the underlying cause and have rolled back this change. Networking telemetry shows recovery from 09:00 UTC onwards across all regions and services, with the final networking equipment recovering at 09:35 UTC. Most impacted Microsoft services automatically recovered once network connectivity was restored, and we worked to recover the remaining impacted services.
Next Steps: We will follow up in 3 days with a preliminary Post Incident Report (PIR), which will cover the initial root cause and repair items. We'll follow that up 14 days later with a final PIR where we will share a deep dive into the incident.
You can track the Post Incident Report (PIR) on the Microsoft Azure link - https://status.azure.com/
We are closing this incident from our end. Please reach out to the Workspot Support team in case you need any assistance related to Workspot Cloud Desktops.
Incident Detected - Azure West US 2 Region Connectivity Issues - December 14, 2022 20:16 UTC
Début 14 décembre 2022 à 20:28 UTC · 18h 58m
Pending
Composants affectés
Workspot Control
investigating
Azure has reported that starting at 19:11 UTC on 14 Dec 2022, Virtual Machines in West US 2 may be experiencing connectivity issues. These Virtual Machines may have also restarted unexpectedly since the start of this incident.
A network device has experienced a fault, resulting in network connectivity loss to downstream resources. The unhealthy network device is being isolated from the network and traffic is being rerouted to healthy infrastructure.
We will update this incident every 30 minutes or as warranted.
identified
Azure reports a network device has experienced a fault, resulting in network connectivity loss to downstream resources. The unhealthy network device is being isolated from the network and traffic is being rerouted to healthy infrastructure.
We will update this incident every 30 minutes or as warranted.
identified
Azure is continuing to work to reroute traffic around the affected network device. We are still seeing issues connecting to some VMs in West US 2 region of Azure.
We will update this incident every 30 minutes or as warranted.
identified
Azure have identified that multiple top-of-rack (ToR) network devices, connecting a single rack of servers, experienced an inadvertent power failure. The Azure resources connected to these ToRs were shut down. Customers may have been unable to connect to their resources and Virtual Machines (VMs) that were deployed onto the affected nodes may have experienced a restart. They are actively working to return the ToRs to a normal state. An update will be provided in 60 minutes or as events warrant.
identified
We are still waiting on an update from Azure.
Last update was:
Azure have identified that multiple top-of-rack (ToR) network devices, connecting a single rack of servers, experienced an inadvertent power failure. The Azure resources connected to these ToRs were shut down. Customers may have been unable to connect to their resources and Virtual Machines (VMs) that were deployed onto the affected nodes may have experienced a restart. They are actively working to return the ToRs to a normal state. An update will be provided in 60 minutes or as events warrant.
monitoring
This incident is now mitigated. Between 20:16 UTC on December 14, 2022 and 00:39 UTC on December 15, 2022, Virtual Machines in West US 2 experienced connectivity issues. Affected Virtual Machines may have restarted unexpectedly during this incident.
Root cause: A network device experienced a fault, resulting in network connectivity loss to downstream Virtual Machines.
Mitigation: Service healing was triggered to automatically redeploy affected Virtual Machines to healthy infrastructure.
We are continuing to monitor the situation and will close the incident as resolved if no further issues are reported by our customers or Azure.
resolved
Azure has resolved the incident.
Their last update:
They have identified that multiple top-of-rack (ToR) network devices, connecting a single rack of servers, experienced an inadvertent power failure. A top-of-rack (ToR) network device, connecting a single rack of servers, experienced a fault.
They have restored the power to the top-of-rack (ToR) network devices to mitigate the issue. A traffic routing normally, there should be no further connectivity issues expected.
If you continue to notice connectivity issues or VM rebooting unexpectedly in West US2, please open a support ticket through the Workspot Control Support Portal.