### Summary
On September 8, 2026, customers in Prod 1 and Prod 2 experienced elevated platform latency and pipeline failures. The issue was caused by a regression in a newly released capability that triggered cascading failures under high load. Because the capability was behind a feature flag, it was quickly disabled, and service was restored after a brief monitoring period.
### Customer Impact
* Customers encountered slowness and failures during pipeline execution and UI operations. Some API calls returned errors or timed out.
* No data loss or corruption occurred.
### Root Cause
The new capability introduced a regression that created contention on a shared backend resource used by multiple Harness components. This saturated the shared platform infrastructure and caused the cascading failures.
### Mitigation
* Disabled the capability across all environments
* Temporarily increased platform capacity to restore stability
### Next Steps
To prevent recurrence, Harness will:
1. **Permanently fix the capability** by profiling and eliminating the sub-optimal code path and query
2. **Improve detection** by enhancing alerting for resource-intensive queries on high-frequency platform paths
公式のインシデント更新を自動翻訳しています。
Prod2は断続的に利用できませんでした
開始 2026年9月5日 9:40 UTC · 1m
Pending
影響を受けたコンポーネント
Platform
investigating
現在、この問題を調査中です.
resolved
この事件は解決しました.
postmortem
## **Summary**
Between 12:34am PST and 12:38am PST on 5th September, the Delegate service manager experienced some elevated exceptions when attempting to write to the database. Consequently, delegate connections were dropped, causing them to disconnect. Delegate automatically re-attempts registration back to the `delegate service manager` and majority of the delegates got connected back after the incident. For Docker and ECS delegates the automatic restart is not enabled unless these delegates have health monitoring enabled. For these delegates a manual restart is needed and was recommended. Post restart the delegate would re-connect and the issue was resolved.
## **Root cause**
On Prod2 cluster we identified a performance bottleneck in the delegate service that, under certain conditions, can increase database write latency and delay heartbeat processing which leads to delegates being disconnected.
## **Impact**
All K8s delegates and \`Docker/ECS\` delegates got connected back immediately within 4 mins and started to function normally. The impact can be scoped to those specific types of delegates that didn’t have health monitoring enabled.
## **Remediation**
* Immediate: We have added additional monitoring and increased resources for handling the influx of traffic.
* Permanent: We have identified a hotspot in the code that can cause high latency when writing to a database which we are actively working on resolving.
## **Action Items**
To prevent such issues from happening again, Harness will work on the following:
1. Increased targeted monitoring and alerting to initiate timely mitigation and prevent this from happening again.
2. Fix the identified delegate service managers database client reconnect failures
3. Fix the hotpots that can cause query latency.
Infrastructure as Code Management (IaCM)Infrastructure as Code Management (IaCM)Infrastructure as Code Management (IaCM)Infrastructure as Code Management (IaCM)
We are currently investigating a Harness component that is experiencing issues. We are working to identify the cause and restore normal operations as soon as possible.
investigating
We are continuing to investigate this issue.
identified
The issue has been identified and a fix is being implemented.
resolved
This incident has been resolved.
postmortem
## Summary
Customers on Prod1, Prod2, and Prod3 \(US\) clusters experienced failures when loading SEI 2.0 dashboards on August 6, 2026, from 7:22 AM PDT to 9:03 AM PDT. Customers calling the SEI 2.0 API also experienced similar failures.
No customer data was lost, and ingestion of all integration data continued to work uninterrupted. SEI customers using 1.0 were not impacted.
## Root Cause
The incident was caused by resource exhaustion on the nodes serving queries. This resource degradation developed in a pattern that did not cross our existing alerting thresholds early enough to provide sufficient warning or allow mitigation before customer impact occurred.
## Impact
Customers on Prod1, Prod2, and Prod3 \(US\) clusters were unable to load SEI 2.0 dashboards during the incident window.
**Duration:** August 6, 2026, 7:22 AM PDT – 9:03 AM PDT \(~1 hour 41 minutes\)
### What was not impacted?
* Data ingestion and processing
* SEI 1.0 customers
* Integrations and metadata flows
No customer data was lost.
## Remediation
Upon identifying the root cause, our team took immediate corrective action by adding capacity to restore the affected systems. Services were fully recovered, and all dashboards resumed normal operation at 9:03 AM PDT.
## Action Items
To prevent from such issues happening again, Harness is/has
Proactively added capacity updates have been applied to prevent this issue from recurring
#### Enhanced Monitoring and Alerting
Additional monitoring and alerting have been put in place to detect anomalies early, focused on a leading indicator, which in this case was thread pool exhaustion, before they can impact dashboard availability and data rendering.
#### System Patch in Progress
We are working with our vendor to apply a patch to remediate this and similar issues completely.
Editing 'Variable Sets' in the IaCM module is experiencing issue
開始 2026年8月4日 12:07 UTC · 3h 21m
Issues軽微なインシデント
影響を受けたコンポーネント
Infrastructure as Code Management (IaCM)Infrastructure as Code Management (IaCM)Infrastructure as Code Management (IaCM)Infrastructure as Code Management (IaCM)
investigating
We are currently investigating this issue.
investigating
We are continuing to investigate this issue.
investigating
We have identified the issue and started to implement the fix , prod2 is restored.
investigating
We are continuously rolling out fixes to all Clusters, Prod EU1 has been resolved.
investigating
We are continuously rolling out fixes to all Clusters, Prod3 has been resolved.
resolved
This incident has been resolved.
postmortem
# Executive Summary
On August 4, 2026, between approximately 3:36 PM and 9:00 PM IST, customers using Infrastructure as Code Management \(IaCM\) on Prod0 and Prod1 were unable to access the Variable Sets settings page. The page rendered blank with no error message, and customers with Variable Sets attached to their workspaces could not view or manage them for the duration of the incident. Prod2, Prod3, and EU1 were not affected.
Separately, during the same window, a scheduled maintenance action caused the IaCM settings tab to temporarily disappear across all environments. This was identified and reversed within the incident bridge call before significant customer impact occurred.
We deployed a hotfix that restored full access to the Variable Sets page on Prod0 and Prod1 the same evening, and we are implementing permanent safeguards described below to prevent this class of issue from recurring.
# Impact
* Customers with the Variable Sets feature enabled on Prod0 and Prod1 were unable to view or manage Variable Sets for approximately 5–6 hours.
* No data was lost or corrupted, this was a UI routing failure only; underlying Variable Sets data and configuration were not affected.
* Prod2, Prod3, and EU1 were not affected by this issue.
* A secondary issue, a scheduled feature flag operation caused the IaCM settings tab to temporarily disappear across all environments during the incident bridge call. This was identified and reversed within minutes. External customer exposure for this secondary issue is still being confirmed.
# Root Cause
A platform routing change released on July 18, 2026 updated how IaCM settings pages are resolved in the user interface. As part of that change, any settings page that had not been explicitly re-registered in the new routing structure became unreachable.
The Variable Sets page had not been re-registered under the new routing structure, making it inaccessible in the environments where the routing change had been deployed — Prod0 and Prod1. Because the failure occurred at the routing layer rather than within the page itself, the page rendered blank with no visible error rather than showing a clear failure message.
# Remediation
## Immediate
We deployed a hotfix that re-registered the Variable Sets page in the updated routing structure, restoring access for all affected customers on Prod0 and Prod1.
## Permanent
We are adding automated end-to-end tests that navigate to settings pages with relevant feature flags enabled, configured as a required gate in our release pipeline. We are also documenting and enforcing the routing constraint through static analysis so that settings pages are never inadvertently left out of the routing structure during future platform changes.
# Action Items
To prevent such issues from happening again,
1. Enhance automated tests, that navigate to settings pages with relevant feature flags enabled, configured as a blocking gate in the release pipeline, so this class of regression is caught before it reaches production.
2. Establish an explicit checklist step for future platform-wide architectural changes that verifies all existing settings pages remain accessible in the updated routing structure before the change is promoted to production.
レガシーダッシュボード - 劣化
開始 2026年8月3日 3:08 UTC · 2h 45m
Issues軽微なインシデント
影響を受けたコンポーネント
Custom Dashboards
investigating
現在、この問題を調査中です.
monitoring
修正を実装し、結果を監視しています.
resolved
この事件は解決しました.
公式のインシデント更新を自動翻訳しています。
UIダッシュボードは後ろにラギングされます(CI)
開始 2026年7月31日 20:22 UTC · 12h 48m
Pending
影響を受けたコンポーネント
Continuous Integration Enterprise(CIE) - Linux Cloud BuildsContinuous Integration Enterprise(CIE) - Windows Cloud BuildsContinuous Integration Enterprise(CIE) - Mac Cloud BuildsContinuous Integration Enterprise(CIE) - Self Hosted RunnersContinuous Integration Enterprise(CIE) - Self Hosted RunnersContinuous Integration Enterprise(CIE) - Self Hosted RunnersContinuous Integration Enterprise(CIE) - Windows Cloud BuildsContinuous Integration Enterprise(CIE) - Windows Cloud BuildsContinuous Integration Enterprise(CIE) - Linux Cloud BuildsContinuous Integration Enterprise(CIE) - Mac Cloud BuildsContinuous Integration Enterprise(CIE) - Linux Cloud BuildsContinuous Integration Enterprise(CIE) - Mac Cloud Builds
# **Summary**
On July 31, 2026, artifact uploads performed through pipeline in the EU1 cluster began failing with an authentication error. Uploads initiated manually \(outside of a pipeline\) were not affected, and the ability to retrieve existing artifacts \(downloads\) was also unaffected — this was isolated to the specific pipeline upload path in one cluster.
# **Impact**
* Artifact uploads performed through pipeline in the EU1 cluster failed with an authentication error for approximately 4 hours and 34 minutes.
* Retrieving existing artifacts \(downloads\) was not affected.
* Manually uploading artifacts outside of a pipeline was not affected.
* Other clusters/regions were not affected by this issue.
# **Root Cause**
The component responsible for handling pipeline-based artifact uploads is distributed as a container image. In the EU1 cluster, this image is retrieved from an internal registry that mirrors a public image source; in other clusters, the same image is retrieved directly from the public source.
A publishing error in our release process caused a new build of this component to be published using a version label that was already in use, rather than being assigned a new, unique version. As a result, two different images ended up associated with the same version label in the public source.
Our internal registry mirrors images from the public source via an automated replication process. Because of how that replication was triggered, it copied the original \(earlier\) image associated with that version label rather than the corrected one. This meant the EU1 cluster — which pulls from the internal mirror — ended up running a different, defective image than other clusters, which pull directly from the public source and therefore received the corrected image. The defective image contained an authentication issue that caused pipeline uploads to fail.
# **Mitigation**
* Reverted the affected account to the last known-good version of the upload component, immediately restoring pipeline uploads.
* Published a corrected, permanent version of the component to resolve the issue across all clusters.
# **Next steps**
* Fix the upload step to remove the underlying container-related defect that made this failure mode possible.
* Update our release pipeline for this component so that publishing an image can never overwrite an existing version — every publish must create a new, distinct version going forward.
公式のインシデント更新を自動翻訳しています。
断続的な外部ネットワーク接続の問題 ビルドVM の不具合
開始 2026年7月30日 5:59 UTC · 21h 24m
Issues軽微なインシデント
影響を受けたコンポーネント
Continuous Integration Enterprise(CIE) - Linux Cloud BuildsContinuous Integration Enterprise(CIE) - Linux Cloud BuildsContinuous Integration Enterprise(CIE) - Linux Cloud BuildsContinuous Integration Enterprise(CIE) - Linux Cloud BuildsContinuous Integration Enterprise(CIE) - Linux Cloud Builds
investigating
概要 - ビルド VM の外部リソースに接続できないネットワーク接続の問題に直面しています。
現在、課題を調査中です.
monitoring
修正を実装し、結果を監視しています.
monitoring
今後の問題がないか引き続き監視しています。
resolved
この事件は解決しました.
postmortem
## Summary
Starting on August 4, 2026, CI runners in the us-west1 and us-central1 regions intermittently experienced connection timeouts of approximately 134 seconds when reaching external services such as GitHub and Bitbucket over outbound network gateways.
## Impact
* CI runners in the affected regions intermittently experienced connection timeouts of approximately 134 seconds when reaching external services \(e.g., GitHub, Bitbucket\) over our outbound network gateways.
* The issue was intermittent rather than constant — connections succeeded under normal load, and failures clustered during periods of high outbound traffic volume.
* No data was lost or corrupted. This was a network-connectivity and capacity issue, not a data-integrity issue.
* us-west1 and us-central1 were the affected regions; other regions were not impacted by this issue.
## Root Cause
Our load balancer distributes outbound traffic across multiple NAT gateways using a hashing method based on connection details \(source/destination address and port\). For any single connection, these details stay constant for that connection's lifetime. We had a sustainted traffic surge for a few seconds which congested the gateways
## Action Items
To prevent such issues from happening again Harness will,
Increase outbound connection capacity on our NAT gateways by provisioning additional external network interfaces, giving each gateway a substantially larger pool of connections it can serve concurrently..
# Summary
During a recent production deployment, a defect in our internal deployment tooling caused two critical services to run with incorrect, non-production configuration values in our production environment This led to a related set of four distinct symptoms: incorrect configuration behavior, intermittent login/access failures, a filestore access issue affecting one customer environment, and delayed pipeline status updates in the UI.
We have identified and are implementing a permanent fix for the underlying configuration defect, and have already put in place resource and capacity changes that resolve the UI delay symptom.
At no point during this incident were pipeline executions themselves lost, corrupted, or left in a stuck state. Where execution behavior was affected, it was limited to delays in status visibility, not in the underlying processing.
# Incident Details
## Incorrect Production Configuration Values Applied
Our engineering team confirmed a defect in the Service Manager deployment pipeline that caused certain production services to be deployed using configuration values intended for a different environment, rather than the correct production configuration.
**Root Cause**
The service responsible for fetching configuration overrides during deployment queries an internal API that returns a maximum of 1,000 results per request. The total number of services in the environment recently grew beyond that limit. As a result, any service beyond the first 1,000 returned was not included in the response, and the deployment pipeline silently fell back to default configuration values for those services. This is a confirmed pagination defect in the deployment tooling, not an issue with the configuration values themselves.
**Resolution**
Engineering has confirmed the mechanism and is implementing a permanent fix to remove this limit-related gap in the deployment pipeline.
## Intermittent Login / Access Failures
During the Service Manager deployment referenced above, some users experienced intermittent login or access failures. Under normal operation, previously running instances should continue serving traffic without interruption while a new deployment is in progress. In this incident, that fallback behavior did not occur as expected, contributing to access failures during the deployment window.
## Filestore Access Issue
A filestore access issue was identified that was specific to the Prod-3 environment and affected a single customer's environment.
**Root Cause**
This is related to an IAM / storage-bucket permission configuration on Service Manager, potentially triggered by rollback activity.
## Delayed Pipeline Execution Status Updates in UI
Some users observed that the pipeline execution graph in the UI was slow to refresh and did not reflect the latest status promptly. Importantly, this was a visibility delay only: there was no impact to actual pipeline executions, and no executions were stuck or failed as a result of this issue.
**Root Cause**
The pipeline execution graph relies on a message stream \(the orchestration log\) to receive status updates. During the incident window, consumer processing of this stream fell behind \(high consumer lag\), which delayed how quickly status updates reached the UI. This was caused by the fact that the underlying database was in the middle of a planned scaling operation at the same time, and a traffic spike during that window further exacerbated the delay. Users experienced this as apparent pipeline slowness, even though the underlying executions were running normally.
**Resolution**
We have increased resource capacity for the affected components to maintain more than 50% spare headroom going forward, reducing sensitivity to similar load spikes. This change has been implemented and is currently being validated as part of longer-term hardening for this part of the platform.
# Impact Summary
* Service Manager and License Manager ran with incorrect configuration values in the Prod-1 and Prod-3 environments.
* Some users experienced intermittent login or access failures during the affected deployment window.
* One customer environment in Prod-3 experienced a filestore access issue.
* Users across affected environments saw delayed pipeline execution status updates in the UI; underlying pipeline executions continued to run correctly and were not lost, stuck, or corrupted.
# Preventive Actions
The following corrective and preventive actions have been identified.
| **Corrective / Preventive Action** |
| --- |
| Correct the pagination limit in the configuration-lookup service so that all services are returned and evaluated, regardless of total count. |
| Add safeguards so that a service which cannot retrieve its configuration fails safely \(e.g. alerts and blocks the deployment\) rather than silently falling back to non-production defaults. |
| Increase Postgres and messaging-pipeline resource headroom \(target: greater than 50% spare capacity\) to reduce sensitivity to concurrent load and scaling events. |
_We recognize the impact this incident had across multiple areas of the platform and appreciate your patience as we work through a complete resolution._