Uipath 中断历史

Between 15:42 UTC and 15:58 UTC, Automation Cloud Data Fabric customers in the Japan region experienced a brief interruption in service. The issue occurred due to database maintenance. Our team responded promptly, mitigated the issue, and service has been fully restored.

Generating a taxonomy with Autopilot on new IXP Unstructured and Complex Documents projects is currently unavailable across all regions. We have identified the root cause and a fix is being rolled out. We will update this page once the fix is fully deployed and the issue is resolved

We are working on deploying the fix

We are deploying fix in progress

Issue has been resolved.

## Customer impact Between July 7, 2026 \(when the defect reached production\) and July 9, 2026 at 2:35 pm UTC \(when the fix finished rolling out\), some attempts to create a new Unstructured and Complex Documents project did not complete normally. The project itself was always created and remained accessible from the project list, but the page showed an error such as "No such source" or "No such project" and returned to the project list instead of opening the new project—interrupting the Autopilot taxonomy generation that runs during project creation. The failure was intermittent, and affected all commercial cloud regions where the feature is available. ## Root cause A recent change introduced a short-lived, in-memory cache of user authorization data in the service, added to improve resilience of authorization checks under load. The cache did not refresh when a user's permissions changed. When a user created a new project, the permissions granted on that new project were not visible to requests served from a stale cache entry \(up to 60 seconds old\), so immediate follow-up requests failed as if the project did not exist. This caused the error, the redirect back to the project list, and the interruption of Autopilot taxonomy generation. ## Detection The issue was identified through internal testing and user reports on the morning of July 9, 2026, and an incident was declared at 11:02 am UTC. The root cause was identified within minutes of the reports. The public status page was updated at 11:34 am UTC, once the affected feature and regional scope were confirmed. We acknowledge the defect was present in production for approximately two days before detection; improving automated detection of this workflow is a follow-up action. ## Response The engineering team traced the failure to the authorization cache change and rolled back that change across all affected regions. The first region was fixed and verified at 12:52 pm UTC, and all regions were remediated by 1:19 pm UTC. Status page updates were posted at 11:34 am, 12:51 pm, and 2:08 pm UTC while the rollout progressed, and the incident was marked resolved at 2:35 pm UTC after verification. ## Follow-up To prevent a recurrence, we are: 1. Replacing the rolled-back change with a corrected version that refreshes cached authorization data when permissions change, validated end-to-end before release. 2. Adding automated end-to-end checks covering project creation with Autopilot taxonomy generation, so a regression in this workflow is detected within minutes.

Automation Solutions Service is experiencing connectivity issues while deploying packages containing integration services entities (connectors). We are currently investigating the issue.

We have identified the issue. Currently working on deploying the fix.

We are currently deploying the fix in lower environment to validate.

We are currently rolling out the fix to the production environment.

Degraded performance impacting deploying packages has been fully resolved. Impact: All affected functionality is operating normally.

We are investigating reports of degraded performance impacting LLM Calls for Agents in the US and EU regions. Impact: Users may experience errors from LLM calls by Agents. Our teams are working to identify the cause and will share more details as the investigation progresses.

Mitigation has been applied and performance is improving for LLM Calls in Agents across the US and EU regions. Impact: Users may still experience failed LLM calls, but this should continue to improve. We are monitoring closely to ensure stability.

The issue has been resolved and Agents performance has returned to expected levels after degraded performance impacted LLM Calls in US and Europe. Impact: No ongoing user impact.

Integration Service Customers Facing intermittent failures during connection creation and activities execution in US and Europe regions

The issue causing degraded performance affecting Integration Service in the US and Europe regions has been identified, and mitigation efforts are currently underway.

Fix has been applyed we are currently monitoring the service to ensure stability

Fix has been applyed we are currently monitoring the service to ensure stability

we are seeing some intermittent failures. We are still investigating further on this.

Issue resolved in Us region. we are seeing some intermittent failures in Europe region. We are investigating further on this.

We are working to deploy the fix

Issue is now resolved in Europe. We are seeing some errors in US and we are continuing to work on the fix.

Issue is mitigated now. we are now monitoring the issue.

Issue is resolved now.

We are investigating an issue affecting the Apps runtime in the Japan region. Impacted customers may see delayed load times or failures when attempting to open deployed Apps. We will provide updates as more information becomes available.

The issue is has been resolved and we are actively monitoring the application.

We have identified the issue and currently working on a fix.

We have identified the cause of the degraded performance impacting Orchestrator's 'configured execution settings for API trigger job ignored' in US and we are working on mitigation. Impact: customers might see configured execution settings for API trigger job ignored in orchestrator.

We are continuing to work on a fix for this issue.

The fix is now live in the US and Japan, and we are monitoring stability in those regions. Concurrently, the team is focused on mitigating the issue across all remaining environments.

The fix is live and working as expected in the US and Japan. We've started deploying it to the remaining affected regions.

Deployment and validation for the remaining regions are underway. We will share further updates as soon as they become available.

Services have been restored to operational status in the EU, Australia, Japan, US and Switzerland. Validation for the remaining regions is underway.

Services have been restored to operational status in India, UAE, Canada, EU, Australia, Japan, US, Switzerland. Validation for the remaining regions (UK, Singapore) is underway.

The fix has now been successfully deployed to the UK. As a result, the following regions are now operational: UK, India, UAE, Canada, EU, Australia, Japan, US and Switzerland

Fix is now live in all regions. Early metrics from monitoring look stable. We'll keep a close eye on everything for the next 30 minutes before declaring full resolution.

This incident has been resolved.

We are currently seeing some instance executions become stuck due to elevated error rates following a recent deployment. The root cause has been identified, and our team is actively working to resolve the issue.

A fix has been identified and we are working on deploying it. Please note that only instances where instance management operations were performed (Pause, resume, retry, goto, migrate, cancel, patchvariable) were impacted.

A fix has been deployed to all impacted regions.

We are currently observing a partial outage on IXP and team is investigating this issue.

The issue has been mitigated and we are now monitoring the service currently. Please note that some functionality might take some time to recover as it catches up.

The issue has been resolved and we are actively monitoring the service.

This issue is causing degraded performance impacting maestro instances and team is currently investigating the issue. Impact: Workflows may experience slowness or get stuck.

The immediate issue has been mitigated. However, because this specific failure mode develops gradually over time, we have additional monitoring in place and collecting more diagnostic data to prevent a recurrence.

The issue remains quiet and has not resurfaced. We are implementing our active monitoring and rapid-response guardrails in place while the engineering team continues investigating the root cause.

Increased latency and degraded performance in US, Investigation is in progress.

Mitigation has been applied and performance is improving for the issue that impacted Orchestrator in US. Impact: Users may still experience degraded performance, but this should continue to improve. We are monitoring closely to ensure stability.

Performance has continued to improve for the issue impacting Orchestrator in the US, and latency has decreased. Impact: Some users may still experience intermittent degraded performance while we continue to investigate. We are monitoring closely to ensure stability.

The issue has been mitigated and Orchestrator performance has returned to expected levels after degraded performance impacted US. Impact: No ongoing user impact.

The regional outage has been resolved and all services in US are fully operational. Impact: No ongoing user impact.

We have identified the cause of the degraded performance impacting the GetJobs Activity for Orchestrator in Europe, Switzerland, Canada, India, Australia and UAE. We are actively working on a mitigation, which will be applied shortly Additional updates will be provided as we move toward resolution.

We have a mitigation available, and are beginning to roll it out to affected regions. We will provide more updates as the mitigation takes effect

The mitigation is taking effect, and we are seeing performance return to normal

All regions have recovered, we will monitor to ensure stability

The incident has been resolved.

Increased latency affecting Maestro workflow executions in the North Europe

We have applied a mitigation and seeing significant performance improvements in Maestro workflow executions. We will continue to monitor to ensure the situation has stabilized.

The issue has been resolved. The mitigation has been effective, and Maestro workflow executions have returned to normal.

## Customer impact On June 28, 2026, between 16:00 and 18:00 UTC, a subset of customers using UiPath Maestro experienced increased latency in workflow executions. This included delays in the start of execution of Maestro instances, and slower task execution generally. The impact was concentrated in the Europe region, and the total customer-impact duration was approximately two hours. As a precaution, capacity adjustments were also applied in the US region. All times are in UTC. ## Root cause Maestro's workflow execution relies on an underlying workflow orchestration platform made up of multiple components — components that process and execute workflow tasks, and a central component that coordinates and dispatches that work between them. In response to an earlier, unrelated load issue, the capacity of the task-processing components had been increased to handle higher demand. The capacity of the central coordinating component, however, was not increased in proportion. As the additional task-processing capacity came online, it generated more concurrent requests to the coordinating component than that was configured to accept. Once its concurrency limit was reached, the latter began rejecting \(rate-limiting\) a portion of the incoming task-polling requests. When these requests were rejected, the system repeatedly retried, which triggered further rate-limiting, and slowed the overall pace of Maestro workflow execution. ## Detection The incident was detected from an internal report from of the UiPath internal teams. An incident bridge was started at 16:00 UTC. A public status update was posted at 16:39 UTC, notifying customers of increased latency affecting Maestro workflow executions in Europe. ## Response Immediately after detection, engineering began investigating. By 17:05 UTC, the team had identified the underlying cause as the workflow orchestration service's concurrency limit being exceeded. Mitigation was applied by increasing the capacity of the workflow execution and orchestration components in Europe. Performance improved significantly following the change, and the status page was updated to **monitoring** at 17:35 UTC. After confirming stability, the incident was marked as **resolved** at 18:00 UTC, with Maestro workflow executions returning to normal. ## Follow-up 1. **Capacity validation through load testing**: Run focused load testing to establish the correct orchestration concurrency limits and capacity settings for Maestro under realistic production demand, so these limits are validated ahead of time rather than tuned during an incident. 2. **Targeted alerting**: Add earlier, more direct alerting on orchestration concurrency saturation and Maestro workflow-execution latency, so similar conditions are caught before they reach customer impact.

The Orchestrator service in the East US region experienced partial degradation due to connectivity issues with the database between June 25, 14:06 UTC and June 25, 14:12 UTC. During this period, customers may have encountered slower response times and intermittent request failures. The issue has been mitigated, and the service is now operating normally. We are continuing to review the incident and publish the root cause. Affected Services: East US – Orchestrator

We observed increased latency affecting Maestro workflow executions in the Europe region between June 12, 2026, 20:00 UTC and June 24, 2026, 15:00 UTC. Impact: Some Maestro workflows experienced slower-than-normal execution times during this period. No other services were impacted. The issue was isolated to Maestro workflow execution in the Europe region. Our team has taken corrective actions to restore normal performance and continues to monitor the service to ensure stability. We apologize for any inconvenience this may have caused.

We observed increased latency affecting Maestro workflow executions in the Europe region between June 12, 2026, 20:00 UTC and June 24, 2026, 15:00 UTC. Impact: Some Maestro workflows experienced slower-than-normal execution times during this period. No other services were impacted. The issue was isolated to Maestro workflow execution in the Europe region. Our team has taken corrective actions to restore normal performance and continues to monitor the service to ensure stability. We apologize for any inconvenience this may have caused.

We are investigating an issue causing increased workflow execution times for some Maestro workflows in the East US region. Impact Window: June 24, 2026, from 6:38 AM PST to 9:46 AM PST During this period, a subset of workflow executions may experience higher-than-normal latency. We are actively monitoring the situation and working to mitigate the impact. Currently the system is stable and operation and we will provide more information as it becomes available

We are investigating reports of degraded performance impacting [impacted functionality / user symptom] for [Service] in [Region]. Impact: Users may experience [describe impacted functionality]. Our teams are working to identify the cause and will share more details as the investigation progresses.

We are investigating reports of degraded performance impacting workflows for Maestro in east US. Impact: Users may experience intermittent failures or stuck workflows. Our teams are working to identify the cause and will share more details as the investigation progresses.

We have scaled up and applied a fix and are seeing positive trends. We will continue to investigate and monitor

Recovery in TaaS East US began at approximately 12:45 PM PST. The service is currently stable, and we will continue monitoring to ensure sustained recovery.

Operations have been stabilized for new workflows. We are now actively working through workflows that became pending as a result of the incident.

The issue has been fully resolved, and all operations are now functioning normally. All affected workflows have been processed successfully

## Customer Impact Between 14:03 and 21:26 UTC on June 18, 2026, and between 17:02 and 23:59 UTC on June 23, 2026, a subset of customers in the U.S. Region, using Maestro and Agent executions—including Conversational Agents and some previously deployed Autonomous Agents—experienced performance degradation and workflows getting stuck in pending state. The two events, a week apart, shared the same underlying root cause. In each case, new workflow operations were stabilized before all pending workflows were fully recovered. During these windows, affected workflows could experience higher-than-normal latency, fail intermittently, or remain stuck in a pending state until recovery processing completed. This RCA covers two related incidents in the U.S. Region that were subsequently determined to stem from the same root cause: an initial event on June 18, 2026 and a recurrence on June 23, 2026. ## Root Cause The incident was caused by our workflow execution backend exhausting its configured memory capacity and restarting repeatedly. Those restarts disrupted workflow state processing, causing some workflow executions to experience latency, fail intermittently, or remain pending. The primary driver of memory growth was an experimental multi-cluster \(active/passive\) replication feature for disaster recovery purpose. This feature accumulated large, long-lived workflow-history objects in memory faster than they could be released. Each time a history-processing component exceeded its memory limit and restarted, it had to reload its in-flight work—including the replication workload — which drove memory back over the limit and triggered another restart. This self-reinforcing cycle continued until responders intervened, and it was not fully resolved by the initial mitigation applied on June 18, which is why the condition recurred on June 23. ## Detection Both events were detected quickly through automated monitoring, including synthetic tests that continuously exercise the platform and alerts for pending/stuck workflow tasks. At the time of the incidents, there was no dedicated alerting on the memory saturation and restart behavior of the affected components, which lengthened the time required to identify the true root cause. Public status updates began after customer impact was confirmed. ## Response and Recovery A cross-team engineering bridge was established to investigate and mitigate the incidents. Early mitigation attempts focused on scaling backend infrastructure and restarting the affected history-processing components. These provided limited relief because they did not address the underlying memory-growth mechanism. Once the memory-retention path was pinpointed through diagnostic heap analysis, the incidents were mitigated by two actions: **increasing the memory capacity allocated to the affected history-processing components**, and **disabling the multi-clusters replication feature** that was the main contributor to the memory growth. After these changes, new workflow operations returned to normal and the region's capacity to process workflows was restored. Workflows that had become stuck in a pending state were recovered by the engineering team where possible. A small number of workflows that customers had already canceled were not replayed, and a limited set of Agent executions that had failed during the incident had to be retried manually by customers. Both incidents were marked resolved after all recoverable workflows had been processed successfully. ## Follow-Up To prevent recurrence and improve detection, we are implementing the following: 1. Right-size the memory capacity and limits for the affected components, and add automatic memory-cap safeguards so the components stay within safe bounds under load. 2. Redesign and re-validate the cross-region replication feature—including load testing with replication enabled—before it is re-enabled, so it can support disaster-recovery needs without causing memory pressure. 3. Add repeatable memory-diagnostic collection capability and operational runbooks for the service, so memory analysis and mitigation can be performed quickly during future incidents.

We are investigating reports of degraded performance impacting workflows for Maestro in east US. Impact: Users may experience intermittent failures or stuck workflows. Our teams are working to identify the cause and will share more details as the investigation progresses.

We have resolved the situation that was causing the degraded performance for Maestro workflows. We had to scale up the capacity for the underlying services. We are actively monitoring the backlog of requests for latencies and will take the action as needed.

Mitigation has been applied and performance is improving for the issue that impacted [impacted functionality / user symptom] in Maestro across US. Impact: Users may still experience degraded perfornance but this should continue to improve. We are monitoring closely to ensure stability.

We have resolved the situation that was causing the degraded performance for Maestro workflows. Impact: Users may still experience degraded performance, but this should continue to improve. We are monitoring closely to ensure stability.

The issue has been resolved and Maestro workflow performance has returned to expected levels after degraded performance impacted Maestro workflow in US. Impact: No ongoing user impact. Agent executions that failed will have to be retried manually.

## Customer Impact Between 14:03 and 21:26 UTC on June 18, 2026, and between 17:02 and 23:59 UTC on June 23, 2026, a subset of customers in the U.S. Region, using Maestro and Agent executions—including Conversational Agents and some previously deployed Autonomous Agents—experienced performance degradation and workflows getting stuck in pending state. The two events, a week apart, shared the same underlying root cause. In each case, new workflow operations were stabilized before all pending workflows were fully recovered. During these windows, affected workflows could experience higher-than-normal latency, fail intermittently, or remain stuck in a pending state until recovery processing completed. ## Root Cause The incident was caused by our workflow execution backend exhausting its configured memory capacity and restarting repeatedly. Those restarts disrupted workflow state processing, causing some workflow executions to experience latency, fail intermittently, or remain pending. The primary driver of memory growth was an experimental multi-cluster \(active/passive\) replication feature for disaster recovery purpose. This feature accumulated large, long-lived workflow-history objects in memory faster than they could be released. Each time a history-processing component exceeded its memory limit and restarted, it had to reload its in-flight work—including the replication workload — which drove memory back over the limit and triggered another restart. This self-reinforcing cycle continued until responders intervened, and it was not fully resolved by the initial mitigation applied on June 18, which is why the condition recurred on June 23. ## Detection Both events were detected quickly through automated monitoring, including synthetic tests that continuously exercise the platform and alerts for pending/stuck workflow tasks. At the time of the incidents, there was no dedicated alerting on the memory saturation and restart behavior of the affected components, which lengthened the time required to identify the true root cause. Public status updates began after customer impact was confirmed. ## Response and Recovery A cross-team engineering bridge was established to investigate and mitigate the incidents. Early mitigation attempts focused on scaling backend infrastructure and restarting the affected history-processing components. These provided limited relief because they did not address the underlying memory-growth mechanism. Once the memory-retention path was pinpointed through diagnostic heap analysis, the incidents were mitigated by two actions: **increasing the memory capacity allocated to the affected history-processing components**, and **disabling the multi-clusters replication feature** that was the main contributor to the memory growth. After these changes, new workflow operations returned to normal and the region's capacity to process workflows was restored. Workflows that had become stuck in a pending state were recovered by the engineering team where possible. A small number of workflows that customers had already canceled were not replayed, and a limited set of Agent executions that had failed during the incident had to be retried manually by customers. Both incidents were marked resolved after all recoverable workflows had been processed successfully. ## Follow-Up To prevent recurrence and improve detection, we are implementing the following: 1. Right-size the memory capacity and limits for the affected components, and add automatic memory-cap safeguards so the components stay within safe bounds under load. 2. Redesign and re-validate the cross-region replication feature—including load testing with replication enabled—before it is re-enabled, so it can support disaster-recovery needs without causing memory pressure. 3. Add repeatable memory-diagnostic collection capability and operational runbooks for the service, so memory analysis and mitigation can be performed quickly during future incidents.

We are currently investigating reports of an issue in IXP in the Switzerland region. Impact: Customers using IXP in the Switzerland may experience degraded performance and failed requests. Our teams are actively working to identify the cause. Further updates will be shared as soon as more information becomes available.

A fix has been deployed and the service is recovering. We are monitoring the system to ensure stability and full recovery. Further updates will be shared soon

The issue has been resolved. The system has remained stable during the monitoring period.

We have identified the cause of the degraded performance impacting write operations for Document Understanding in US region and are working on mitigation. Impact: Users may experience failures on some of the write operations. Additional updates will be provided as we move toward resolution.

Impact: Users may be unable to write and perform certain operations in US region on Document Understanding. Next update: Our teams are working to understand the cause and scope and will share updates as available.

Fix is underway and additional updates will be provided as we move toward resolution. Impact: Users may be unable to write and perform certain operations in US region on Document Understanding.

The outage has been resolved and Document Understanding in US region is fully operational.