Split outage history

We are currently investigating this issue.

This incident has been resolved.

Pipeline execution is stuck in Prod1. We are investigating the issue.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

We are currently investigating this issue.

The issue has been identified and a fix is being implemented.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

## Summary Between August 27 and August 28, 2026, customers experienced an issue where some pipelines, deployments, and related resources appeared as not found in the Harness UI and API, even though the underlying data remained intact. ‌ The issue occurred during a planned internal infrastructure update that affected communication between internal platform services. As a result, requests that depended on account, organization and project scope resolution were unable to complete successfully, which led to incorrect not found responses being returned to customers for existing entities. ‌ Engineering identified the issue, rolled back the change, and restored normal service. No customer data was lost or deleted during the incident. ‌ ## Root Cause The issue was caused by a configuration error introduced during a planned internal service routing update in Production. An internal platform service responsible for resolving account, organization, and project context was unable to validate requests from other Harness services after the change was applied. Because that validation step is required before many entity reads and pipeline-related actions can proceed, the failed requests surfaced to customers as not found errors for resources that continued to exist normally. The issue was limited to the affected production environment and was resolved by reverting the change and restoring the previous service communication path. ‌ ## Impact * Some customers saw existing pipelines, deployments, and related entities appear as not found in the UI and API. * Some pipeline-related operations, including execution progression, webhook-triggered starts, scheduled trigger evaluation, and entity listing, were temporarily disrupted. * The issue affected availability and visibility of existing entities, but it did not remove data or change customer configurations. * No unauthorized access occurred, and no customer data loss was observed. ## Remediation * **Immediate:** Reverted the infrastructure configuration update and restored the previously working service communication path. * **Recovery validation:** Verified that affected entity lookups, pipeline operations, and dependent APIs were functioning normally after rollback. * **Permanent:** Corrected the configuration handling associated with the update so similar issues do not interfere with service-to-service authentication in future rollouts. ## Action Items To prevent such issues from happening again, Harness will  1. Improve configuration validation by enhancing the pre-deployment tests to verify internal service communication before shifting production traffic. 2. Enhance monitoring and alerting for internal authentication failures so issues can be detected earlier. 3. Improve error handling so dependency failures are less likely to appear to customers as resource not found errors.

We are currently investigating an issue reported in IACM pipelines in Prod-1 , Prod-2 ,Prod-4 and EU1 harness clusters.

We are continuing to investigate this issue.

We have reverted the change that caused this issue in all clusters.

This incident has been resolved.

We are currently investigating this issue.

We are currently investigating reports that the Feature Management & Experimentation (FME) user interface is failing to load. Customers attempting to access the FME console may encounter errors or unresponsive pages. Feature flag evaluation and SDK traffic are not believed to be affected. A further update will follow shortly.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

## Summary * Starting at **23:42 UTC** on August 23, 2026, several FME customers reported failures loading the FME UI. * FME UI Artifacts served from the CDN expired due to a retention policy, causing FME UI to fail to load. * Any flag request changes through the API, change delivery, and the data pipeline continued to work with no interruption. ## Root Cause * The FME UI is served from a CDN. The UI artifacts got evicted due to a retention policy, causing the UI to fail to load for all users. ## Impact * The FME UI was unable to load for all users across all production environments. ### What was not impacted? * SDK functionality and runtime flag evaluation * Admin API calls * Customer flag configuration data * No data loss occurred ## Remediation * FME UI got restored in the CDN through a deployment * Recovery confirmed across all production environments before closing the incident. ## Action Items * Improve the asset retention policy so that the currently active version is never subject to eviction.

We are currently investigating this issue.

The slowness could cause either of below symptoms: - Pipelines not starting - Delays in execution - Pipelines being cancelled due to timeouts

Our cloud provider is facing an active incident and we are following up.

We are continuing to investigate this issue.

Our cloud provider has confirmed an ongoing incident impacting multiple regions. Harness pipelines have not experienced failures as a result, though some users may continue to experience slowness. We are monitoring the situation closely and will provide updates as more information becomes available.

We are observing improved latencies across the board following the fix implemented by our cloud provider. We are continuing to monitor the situation closely and will provide further updates as warranted. We noted some stuck executions for CI for couple of customers, which we are investigating

This incident has been resolved.

# Summary On 20 August 2026, beginning at approximately 15:00 UTC, the Harness platform experienced widespread performance degradation across all production environments. Pipeline executions that normally complete in around two minutes took seven to ten minutes. Continuous Delivery, Continuous Integration, pipeline orchestration, and Feature Management & Experimentation were all affected. Google Cloud Platform experienced a multi-product incident in the us-west1 region affecting Bigtable, Compute Engine, Google Kubernetes Engine, and persistent-disk I/O. Harness production infrastructure runs on persistent disks in that region. The degradation raised database operation latency from approximately 2 ms to over 10 ms at the 95th percentile, which in turn caused message-queue processing lag and propagated to every service that depends on timely database access. ‌ # Impact This was a degradation, not an outage. Pipelines continued to execute and complete successfully throughout; they were slow rather than failing. No data was lost, and no customer work was dropped as a result of this incident. # **Root cause** Harness production infrastructure in the affected environments runs on Google Cloud Platform persistent disks in the us-west1 region. When that storage layer degraded, the effect propagated through the platform in a predictable chain: **Persistent-disk I/O degradation in us-west1.** Google Cloud Platform experienced a multi-product incident affecting Bigtable, Compute Engine, Google Kubernetes Engine, and persistent-disk performance. This was an infrastructure failure in the provider’s environment, outside Harness’s control. # **Preventive actions** Although Harness cannot prevent a cloud provider infrastructure failure. The actions below are aimed at detecting one faster and being better positioned to act on it. | **Action** | | --- | | Continue routine pre-testing of targeted cross-region database failovers, as performed during this incident, to keep failover readiness verified rather than assumed | | Assess full-stack multi-region failover readiness for future scenarios in which cross-region latency would be unacceptable |

We are currently investigating this issue.

The issue has been identified and a fix is being implemented.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

### Summary On August 20, 2026, between 10:24 and 14:55 UTC, a subset of FME writes failed. Writes made from the FME UI and writes made with Harness access tokens \(PATs and SATs\) were not affected. Runtime flag evaluation continued to work normally. The issue was mitigated by reverting a recent authentication change in a shared governance service, and affected writes returned to normal by 14:55 UTC. Status: [https://status.harness.io/incidents/rhthgm7d5dkz](https://status.harness.io/incidents/rhthgm7d5dkz) ### Root Cause A change in how a shared governance service authenticated inbound calls resulted in some FME writes being rejected. Those writes used service-to-service credentials that the governance service could no longer verify after the change. FME surfaces a governance failure to the client as HTTP 499, the same status used when a governance policy intentionally denies a change. Because 499 is a valid, expected response in that deny path, the failures did not look like an outage on our alerts, and the incident was identified from customer reports rather than internal detection. ### Impact * A subset of FME writes failed during the window, primarily those made using legacy Split API keys or change request scheduling. * Writes made from the FME UI were not impacted. * Writes using Harness access tokens \(PATs and SATs\) were not impacted. * Runtime flag evaluation continued normally. * No data loss occurred. Failed writes did not apply. ‌ ### Remediation Reverted the governance-service authentication change. Affected writes returned to normal immediately. ### Action Items To prevent such issues from happening again, * Harness will return a distinct error \(not 499\) when a write fails because governance could not be evaluated, so it is not confused with an intentional policy denial. * Add alerting on the governance evaluation call itself, rather than relying on the client-facing status code. * Expand authentication support for policy evaluations. * Expand automated coverage for additional write scenarios.

We are currently investigating this issue.

We are continuing to investigate this issue.

The issue has been identified and a fix is being implemented.

A fix has been implemented and we are monitoring the results.

We are continuing to monitor for any further issues.

This incident has been resolved.

**Summary** On 19 August 2026 between 12:35 and 17:29 UTC, the Harness Application Security service experienced a significant disruption affecting both the customer-facing console and the data ingestion pipeline in the SaaS Production and US1 regions. ‌ **Root Cause** The internal configuration service that supplies runtime settings to nearly every other component became overloaded and entered a repeated restart cycle. Because so many services depend on it, the effects were broad: console pages such as protection policies, posture views, activity logs, API inventory, and custom policy failed to load or timed out, and downstream processing stalled while waiting for configuration it could not obtain. # **Customer impact** | **Dimension** | **Detail** | | --- | --- | | Console \(UI\) impact | Multiple pages failed to load or timed out, including protection policies, posture event pages and posture views inside dashboards and insight pages, activity log queries, API inventory screens, custom policy, and sensitive-data views and widgets. | | Ingestion impact | Security telemetry processing degraded severely and, in some paths, stopped entirely. Consumer lag grew across normalisation, grouping, anomaly detection, generation, and related processing stages. | | Data loss | A subset of telemetry ingested during the disruption was permanently dropped. | ‌ **Mitigation** Several intermediate mitigations additional CPU and memory, relaxed health-check thresholds, a database restart, and a larger connection pool ameliorated the issue. Disabling the new feature in both affected regions restored throughput sharply and durably. The incident was resolved at 17:29 UTC. ‌ # **Preventive actions** The following actions are committed and tracked internally to completion. The feature that triggered this incident remains disabled and will not be re-enabled until the work below is complete and validated. | **Action** | | --- | | | | OPtimize the code by tuning parameters such as cache eviction and retention , evaluate cursor-based pagination for bulk rule retrieval as rule counts grow | | Add a purpose-built database index for the service-scoping access pattern | | Remediate pipeline recovery semantics so consumers replay safely after position-marker loss instead of skipping backlog | | Mandate staged rollout for configuration overrides that alter downstream request patterns: low-volume cluster, then mid-volume, then high-volume | | Add backpressure and concurrency protection to the configuration service: circuit breaking, bounded queues, and timeout isolation | | Enhance observability by Instrumenting more detailed metrics |

We are currently investigating a Harness component that is experiencing issues. We are working to identify the cause and restore normal operations as soon as possible.

We are continuing to investigate this issue.

The issue has been identified and a fix is being implemented.

This incident has been resolved.

## Summary Customers on Prod1, Prod2, and Prod3 \(US\) clusters experienced failures when loading SEI 2.0 dashboards on August 6, 2026, from 7:22 AM PDT to 9:03 AM PDT. Customers calling the SEI 2.0 API also experienced similar failures. No customer data was lost, and ingestion of all integration data continued to work uninterrupted. SEI customers using 1.0 were not impacted. ## Root Cause The incident was caused by resource exhaustion on the nodes serving queries. This resource degradation developed in a pattern that did not cross our existing alerting thresholds early enough to provide sufficient warning or allow mitigation before customer impact occurred. ## Impact Customers on Prod1, Prod2, and Prod3 \(US\) clusters were unable to load SEI 2.0 dashboards during the incident window. **Duration:** August 6, 2026, 7:22 AM PDT – 9:03 AM PDT \(~1 hour 41 minutes\) ### What was not impacted? * Data ingestion and processing * SEI 1.0 customers * Integrations and metadata flows No customer data was lost. ## Remediation Upon identifying the root cause, our team took immediate corrective action by adding capacity to restore the affected systems. Services were fully recovered, and all dashboards resumed normal operation at 9:03 AM PDT. ## Action Items To prevent from such issues happening again, Harness is/has Proactively added capacity updates have been applied to prevent this issue from recurring #### Enhanced Monitoring and Alerting Additional monitoring and alerting have been put in place to detect anomalies early, focused on a leading indicator, which in this case was thread pool exhaustion, before they can impact dashboard availability and data rendering. #### System Patch in Progress We are working with our vendor to apply a patch to remediate this and similar issues completely.

We are monitoring the stuck pipelines in prod2. The new executions are passing as we are continuously monitoring the services.

We are monitoring the stuck pipelines in prod2. For the customers who are still seeing stuck pipelines, we request you to abort and re-trigger.

This incident has been resolved.

## **Summary** On August 6, 2026 \(morning PDT\), some customers running pipelines in the Prod2 production environment observed pipeline executions that stopped making progress — stages that did not advance and produced no further output or status updates. The issue was reported by affected customers. Harness engineers identified the cause, mitigated the impact, and pipeline executions returned to normal operation. The issue was caused by a self-referential pipeline expression. A Git webhook triggered a pipeline that referenced the contents of the webhook payload, and the payload itself contained further copies of that same expression. Each round of expression resolution therefore produced more expressions to resolve, doubling the amount of work each time. This exhausted the resources of the service instance processing that execution, and other executions assigned to the same instance were unable to progress while it was in that state. ## **Impact** During the incident window \(approximately 6:11 AM to 11:23 AM PDT on August 6, 2026\): * Some customers' pipeline executions on Prod2 stalled mid-execution and made no further progress. * Affected executions produced no new step output or status updates, and had to be aborted and re-run after mitigation. * Behavior was limited to executions being processed by the affected service instance — pipelines handled by other instances continued to execute normally. There was **no data loss**. Pipeline definitions, execution history, and stored state were unaffected. The majority of pipelines on Prod2 continued to execute successfully throughout the incident; the primary impact was that some in-flight executions could not complete and needed to be re-run once the issue was mitigated. ## **Root Cause** Harness pipelines support expressions that are resolved at runtime — for example, an expression that inserts the contents of the Git webhook payload that triggered the pipeline. In this case, a Git commit message contained the literal text of the payload expression itself, twice, and the pipeline referenced that same payload expression. Because the commit message is part of the webhook payload, resolving the expression inserted the entire payload — including the two literal copies of the expression carried in the commit message. Those newly inserted copies were then treated as expressions to be resolved, and each pass inserted two more full copies of the payload. The size of the value being processed, and the work required to process it, therefore doubled on every pass and grew exponentially rather than converging. Harness has a safeguard intended to stop exactly this: expression resolution is bounded by a maximum nesting depth, beyond which resolution halts and the pipeline fails with an explicit error. A defect in that safeguard meant the limit was not applied in this specific self-referential case, so resolution continued unchecked. Expression resolution runs inline on the threads that start pipeline steps. As each pass consumed progressively more memory and CPU without ever completing, the service instance performing that work stopped making progress, and every execution assigned to that instance stalled — which is what customers reported. ## **Mitigation** Harness completed the following immediate mitigation steps: * Identified the pipeline and the expression pattern responsible for the runaway resolution. * Stopped the affected service instance so that it would take on no further work. The remaining healthy instances picked up and processed queued executions normally. * Confirmed that pipeline executions returned to normal and closed the incident. These actions restored normal pipeline execution behavior and resolved the customer-facing impact. ## **Action Items** To reduce the risk of recurrence and improve detection, the following actions are in various stages of being implemented: * Fix the defect in the expression depth and loop-detection safeguard so that self-referential expressions are caught and fail fast with a clear error instead of consuming resources without bound. * Prevent payload expressions from being resolved out of trigger payload content, removing the self-referential path entirely. * Tighten the maximum expression nesting depth and evaluate explicit loop detection in addition to the existing depth limit. * Enhance automated tests in pre-production environments that reproduce self-referential expression patterns and verify that the safeguard detects and stops them. * Add monitoring for this pattern in pipeline executions so that it is detected proactively.

We are currently investigating this issue.

We are continuing to investigate this issue.

We have identified the issue and started to implement the fix , prod2 is restored.

We are continuously rolling out fixes to all Clusters, Prod EU1 has been resolved.

We are continuously rolling out fixes to all Clusters, Prod3 has been resolved.

This incident has been resolved.

# Executive Summary On August 4, 2026, between approximately 3:36 PM and 9:00 PM IST, customers using Infrastructure as Code Management \(IaCM\) on Prod0 and Prod1 were unable to access the Variable Sets settings page. The page rendered blank with no error message, and customers with Variable Sets attached to their workspaces could not view or manage them for the duration of the incident. Prod2, Prod3, and EU1 were not affected. Separately, during the same window, a scheduled maintenance action caused the IaCM settings tab to temporarily disappear across all environments. This was identified and reversed within the incident bridge call before significant customer impact occurred. We deployed a hotfix that restored full access to the Variable Sets page on Prod0 and Prod1 the same evening, and we are implementing permanent safeguards described below to prevent this class of issue from recurring. # Impact * Customers with the Variable Sets feature enabled on Prod0 and Prod1 were unable to view or manage Variable Sets for approximately 5–6 hours. * No data was lost or corrupted, this was a UI routing failure only; underlying Variable Sets data and configuration were not affected. * Prod2, Prod3, and EU1 were not affected by this issue. * A secondary issue, a scheduled feature flag operation caused the IaCM settings tab to temporarily disappear across all environments during the incident bridge call. This was identified and reversed within minutes. External customer exposure for this secondary issue is still being confirmed. # Root Cause A platform routing change released on July 18, 2026 updated how IaCM settings pages are resolved in the user interface. As part of that change, any settings page that had not been explicitly re-registered in the new routing structure became unreachable. The Variable Sets page had not been re-registered under the new routing structure, making it inaccessible in the environments where the routing change had been deployed — Prod0 and Prod1. Because the failure occurred at the routing layer rather than within the page itself, the page rendered blank with no visible error rather than showing a clear failure message. # Remediation ## Immediate We deployed a hotfix that re-registered the Variable Sets page in the updated routing structure, restoring access for all affected customers on Prod0 and Prod1. ## Permanent We are adding automated end-to-end tests that navigate to settings pages with relevant feature flags enabled, configured as a required gate in our release pipeline. We are also documenting and enforcing the routing constraint through static analysis so that settings pages are never inadvertently left out of the routing structure during future platform changes. # Action Items To prevent such issues from happening again, 1. Enhance automated tests, that navigate to settings pages with relevant feature flags enabled, configured as a blocking gate in the release pipeline, so this class of regression is caught before it reaches production. 2. Establish an explicit checklist step for future platform-wide architectural changes that verifies all existing settings pages remain accessible in the updated routing structure before the change is promoted to production.

We are currently investigating this issue.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

We are currently investigating this issue.

The issue has been identified and a fix is being implemented.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

# **Summary** Between 25 July and 4 August 2026, pipeline execution dashboards and overview pages in the Harness Prod 2 and Prod 3 clusters displayed data that was between behind real time. Pipelines themselves continued to build, deploy, and execute normally throughout; the issue was confined to how quickly execution records were copied into the database that serves reporting and dashboard views. ‌ **No customer data was lost.** Every affected record remained durably stored and was replayed into the analytics datastore once the underlying limitation was removed. Harness migrated the affected clusters to a horizontally scalable, queue-backed version of the replication component on 1 August 2026 and completed targeted data backfills for all affected accounts. # **Root cause** Harness maintains a change-data-capture component that continuously replicates pipeline execution records from the primary operational datastore into a separate time-series datastore optimised for dashboards and reporting queries. Dashboards read exclusively from the analytics datastore. When replication falls behind, dashboards render an accurate but older view of the world, while execution itself is unaffected. This was caused by sharp, sustained increase in database write volume from another Harness platform module sharing the same replication path exceeded the throughput ceiling of the older, single-instance version of that component still running in Prod 2 and Prod 3. A backlog formed and grew. ‌ ‌ # **Preventive actions** Harness has completed or committed to the following actions to prevent such issues. | **Action** | | --- | | Fine tune the replication lag alerting so that any delay beyond a defined threshold is notified | | Add a replication lag panel to the standard platform monitoring board so pipeline health is visible to on-call by default | | Reduce write amplification from co-tenant modules through per-module rate limiting or entity filtering on the replication stream |

We are currently investigating this issue.

The issue has been identified and a fix is being implemented.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

# **Summary** On July 31, 2026, artifact uploads performed through pipeline in the EU1 cluster began failing with an authentication error. Uploads initiated manually \(outside of a pipeline\) were not affected, and the ability to retrieve existing artifacts \(downloads\) was also unaffected — this was isolated to the specific pipeline upload path in one cluster. # **Impact** * Artifact uploads performed through pipeline in the EU1 cluster failed with an authentication error for approximately 4 hours and 34 minutes. * Retrieving existing artifacts \(downloads\) was not affected. * Manually uploading artifacts outside of a pipeline was not affected. * Other clusters/regions were not affected by this issue. # **Root Cause** The component responsible for handling pipeline-based artifact uploads is distributed as a container image. In the EU1 cluster, this image is retrieved from an internal registry that mirrors a public image source; in other clusters, the same image is retrieved directly from the public source. A publishing error in our release process caused a new build of this component to be published using a version label that was already in use, rather than being assigned a new, unique version. As a result, two different images ended up associated with the same version label in the public source. Our internal registry mirrors images from the public source via an automated replication process. Because of how that replication was triggered, it copied the original \(earlier\) image associated with that version label rather than the corrected one. This meant the EU1 cluster — which pulls from the internal mirror — ended up running a different, defective image than other clusters, which pull directly from the public source and therefore received the corrected image. The defective image contained an authentication issue that caused pipeline uploads to fail. # **Mitigation** * Reverted the affected account to the last known-good version of the upload component, immediately restoring pipeline uploads. * Published a corrected, permanent version of the component to resolve the issue across all clusters. # **Next steps** ‌ * Fix the upload step to remove the underlying container-related defect that made this failure mode possible. * Update our release pipeline for this component so that publishing an image can never overwrite an existing version — every publish must create a new, distinct version going forward.

Summary - We are intermittently facing network connectivity issues with our Build VM's unable to connect to external resources. We are currently investigating the issue.

A fix has been implemented and we are monitoring the results.

We are continuing to monitor for any further issues.

This incident has been resolved.

## Summary Starting on August 4, 2026, CI runners in the us-west1 and us-central1 regions intermittently experienced connection timeouts of approximately 134 seconds when reaching external services such as GitHub and Bitbucket over outbound network gateways. ## Impact * CI runners in the affected regions intermittently experienced connection timeouts of approximately 134 seconds when reaching external services \(e.g., GitHub, Bitbucket\) over our outbound network gateways. * The issue was intermittent rather than constant — connections succeeded under normal load, and failures clustered during periods of high outbound traffic volume. * No data was lost or corrupted. This was a network-connectivity and capacity issue, not a data-integrity issue. * us-west1 and us-central1 were the affected regions; other regions were not impacted by this issue. ## Root Cause ‌ Our load balancer distributes outbound traffic across multiple NAT gateways using a hashing method based on connection details \(source/destination address and port\). For any single connection, these details stay constant for that connection's lifetime. We had a sustainted traffic surge for a few seconds which congested the gateways ‌ ## Action Items To prevent such issues from happening again Harness will, Increase outbound connection capacity on our NAT gateways by provisioning additional external network interfaces, giving each gateway a substantially larger pool of connections it can serve concurrently..

We are currently investigating this issue.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

We are currently investigating this issue.

We are continuing to investigate this issue.

The issue has been identified and a fix is being implemented.

We are continuing to work on a fix for this issue.

A fix has been implemented and we are monitoring the results.

We are continuing to monitor for any further issues.

This incident has been resolved.

# Summary During a recent production deployment, a defect in our internal deployment tooling caused two critical services   to run with incorrect, non-production configuration values in our production environment This led to a related set of four distinct symptoms: incorrect configuration behavior, intermittent login/access failures, a filestore access issue affecting one customer environment, and delayed pipeline status updates in the UI. ‌ We have identified and are implementing a permanent fix for the underlying configuration defect, and have already put in place resource and capacity changes that resolve the UI delay symptom. ‌ At no point during this incident were pipeline executions themselves lost, corrupted, or left in a stuck state. Where execution behavior was affected, it was limited to delays in status visibility, not in the underlying processing. # Incident Details ## Incorrect Production Configuration Values Applied Our engineering team confirmed a defect in the Service Manager deployment pipeline that caused certain production services to be deployed using configuration values intended for a different environment, rather than the correct production configuration. ‌ **Root Cause** ‌ The service responsible for fetching configuration overrides during deployment queries an internal API that returns a maximum of 1,000 results per request. The total number of services in the environment recently grew beyond that limit. As a result, any service beyond the first 1,000 returned was not included in the response, and the deployment pipeline silently fell back to default configuration values for those services. This is a confirmed pagination defect in the deployment tooling, not an issue with the configuration values themselves. ‌ **Resolution** ‌ Engineering has confirmed the mechanism and is implementing a permanent fix to remove this limit-related gap in the deployment pipeline. ## Intermittent Login / Access Failures During the Service Manager deployment referenced above, some users experienced intermittent login or access failures. Under normal operation, previously running instances should continue serving traffic without interruption while a new deployment is in progress. In this incident, that fallback behavior did not occur as expected, contributing to access failures during the deployment window. ## Filestore Access Issue A filestore access issue was identified that was specific to the Prod-3 environment and affected a single customer's environment. **Root Cause** This is related to an IAM / storage-bucket permission configuration on Service Manager, potentially triggered by rollback activity.  ## Delayed Pipeline Execution Status Updates in UI Some users observed that the pipeline execution graph in the UI was slow to refresh and did not reflect the latest status promptly. Importantly, this was a visibility delay only: there was no impact to actual pipeline executions, and no executions were stuck or failed as a result of this issue. **Root Cause** The pipeline execution graph relies on a message stream \(the orchestration log\) to receive status updates. During the incident window, consumer processing of this stream fell behind \(high consumer lag\), which delayed how quickly status updates reached the UI. This was caused by the fact that the underlying database was in the middle of a planned scaling operation at the same time, and a traffic spike during that window further exacerbated the delay. Users experienced this as apparent pipeline slowness, even though the underlying executions were running normally. **Resolution** We have increased resource capacity for the affected components to maintain more than 50% spare headroom going forward, reducing sensitivity to similar load spikes. This change has been implemented and is currently being validated as part of longer-term hardening for this part of the platform. # Impact Summary * Service Manager and License Manager ran with incorrect configuration values in the Prod-1 and Prod-3 environments. * Some users experienced intermittent login or access failures during the affected deployment window. * One customer environment in Prod-3 experienced a filestore access issue. * Users across affected environments saw delayed pipeline execution status updates in the UI; underlying pipeline executions continued to run correctly and were not lost, stuck, or corrupted. # Preventive Actions The following corrective and preventive actions have been identified. ‌ | **Corrective / Preventive Action** | | --- | | Correct the pagination limit in the configuration-lookup service so that all services are returned and evaluated, regardless of total count. | | Add safeguards so that a service which cannot retrieve its configuration fails safely \(e.g. alerts and blocks the deployment\) rather than silently falling back to non-production defaults. | | Increase Postgres and messaging-pipeline resource headroom \(target: greater than 50% spare capacity\) to reduce sensitivity to concurrent load and scaling events. | ‌ _We recognize the impact this incident had across multiple areas of the platform and appreciate your patience as we work through a complete resolution._

We are investigating an issue impacting AIDI dashboards. Users may experience increased load times or intermittent failures when accessing dashboards. Our team is actively working to identify the root cause and restore normal performance. We will provide updates as more information becomes available.

The issue has been identified and a fix is being implemented.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

## Summary Customers on Prod1, Prod2, and Prod3 clusters experienced intermittent widget load failures and increased load times when accessing AIDI 2.0 dashboards on July 22, 2026. Not all widgets were affected simultaneously the issue manifested as sporadic failures rather than a full outage. No customer data was lost. SEI 1.0 customers were not impacted. ## Root Cause Over time, a routine database maintenance process failed to run on certain tables in our analytics database, causing those tables to accumulate a large volume of internal metadata used to track deleted records. When the database planned queries against these tables, it loaded all of this accumulated metadata into memory, causing memory usage on the affected nodes to spike repeatedly. These repeated spikes triggered an automatic safety mechanism that restarts a node when it detects excessive memory pressure, and the affected nodes began restarting in a loop as a result. This caused intermittent, degraded query performance on AIDI 2.0 dashboards for the duration of the incident. ## Impact Customers on Prod1, Prod2, and Prod3 clusters may have experienced intermittent widget load failures or increased load times on AIDI 2.0 dashboards. **Duration:** July 22, 2026, 07:58 PDT – 16:16 PDT \(~8 hours 18 minutes\), with intermittent widget failures; system was restarted and under active monitoring from 08:25 PDT onward. ### What was not impacted? * Data ingestion and processing * SEI 1.0 customers * Integrations and metadata flows No customer data was lost. ## Remediation Upon identifying the issue, the affected database nodes were restarted at 08:25 PDT, which restored initial stability. We continued to monitor the system closely, and when intermittent degradation was still observed afterward, we applied several additional fixes: * Adjusted database configuration settings to limit the amount of memory used for processing accumulated metadata, and tuned query-planning settings to reduce memory pressure. * Ran cleanup jobs to reduce the backlog of accumulated metadata on the affected tables. * Increased capacity on the affected database nodes to provide additional headroom. These changes progressively stabilized the system, and the incident was fully resolved at 16:16 PDT. ## Action Items To prevent recurrence, we are implementing the following: 1. We have upgraded the backend which includes underlying improvements that handle memory spikes caused by excessive delete files. 2. We have rolled out automated compaction jobs for newly introduced tables to prevent delete file accumulation going forward.

We are currently investigating this issue.

The issue has been identified and a fix is being implemented.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

# Summary On 17 July 2026, following a routine code deployment, customers on older delegate versions \(858xx and below\) began experiencing delayed CI builds on Harness Cloud-hosted builds using our global build-queueing capability. Affected builds experienced an unexpected pause of up to approximately 8 minutes at the "waiting for infrastructure" stage before continuing, rather than proceeding within the expected sub-second time. Overall build slowness was intermittent. # Impact * All CI builds were potentially subject to delay; impact was most pronounced for builds on Harness Cloud-hosted infrastructure using the global build-queueing feature. * Affected builds experienced an unexplained pause of up to approximately 8 minutes before continuing, followed by a slower "cold start" since a pre-reserved compute slot was not available — this presented to users as slow builds rather than build failures. * Accounts running on newer delegate versions \(858xx and above\) were not impacted. * No builds failed outright as a direct result of this issue, and no data was lost. # Root Cause The root cause was an internal code change that inadvertently broke how a specific build-queueing record was read back from our database once builds that had already been queued under the previous version of the code encountered the newly deployed version. We resolved the immediate impact by cleaning up the affected records and reverting the underlying code change, and we are implementing several safeguards to prevent this class of issue from recurring. ‌ # Next Steps We assess the risk of a similar recurrence as low as the following actions are being understaken.The specific code path that caused this incident has already been reverted, and we are implementing structural safeguards so that this general class of issue cannot recur, regardless of where in the codebase it might otherwise occur. | **Corrective / Preventive Action** | | --- | | Add explicit, stable identifiers to all internal data classes that get stored in our database, so that future internal code reorganizations cannot break the system's ability to read back previously stored records. | | Introduce rollback and backward-compatibility testing in our pre-production environment, specifically designed to catch this class of issue before it reaches production. |

We are currently investigating this issue.

The issue has been identified and a fix is being implemented.

A fix has been implemented and we are monitoring the results.

We are continuing to monitor for any further issues.

This incident has been resolved.

# Summary Between June 19 and July 17, 2026, the built-in Git Clone step — and any pipeline step using the drone-git clone plugin — failed on ARM64 Kubernetes build infrastructure with the error exec /usr/local/bin/clone: exec format error. AMD64 \(Intel/AMD\) builds, Windows builds, and the VM containerless binary path were not affected. The root cause was a defect in our internal image publishing process that caused ARM64-tagged drone-git images to actually contain AMD64 binaries. We identified and mitigated the issue the same day it was reported by reverting the drone-git image to the last known-good version. No customer action or configuration change was required. # Root Cause On June 19, 2026, a security remediation restructured how the drone-git image is built. AMD64 builds were updated correctly, but the ARM64 build pipeline didn't build ARM64 files directly — it adapted the AMD64 build file via text substitution and compiled it on ARM64 infrastructure. The June 19 change altered the AMD64 file so that substitution silently no-op'd instead of failing, so the pipeline published an image tagged ARM64 whose Git Clone and Git LFS binaries were still compiled for AMD64. # Impact * Affected: The built-in Git Clone step, and any pipeline step using the drone-git clone plugin, running on ARM64 Kubernetes build infrastructure, across all accounts, between July 6 and July 17, 2026. * Symptom: Builds failed at the Git Clone step with exec /usr/local/bin/clone: exec format error. * Not affected: AMD64 \(Intel/AMD\) Kubernetes and VM builds, Windows builds, the VM containerless execution path, and our hardened image variant. # Mitigation We reverted the drone-git image version used across all affected services to the last known-good release. This fully resolved the ARM64 execution failures; no customer configuration changes were required. # Next Steps To prevent such issues from happening again. * Rebuild the ARM64 image publishing pipeline to build our dedicated ARM64 build files directly, rather than adapting the AMD64 build files. * Enhance automated post-publish validation to every image release: verify binary architecture matches the image tag, and run a functional smoke test before an image is considered releasable. * Expand automated test coverage to include ARM64 Kubernetes build scenarios. * Remove the affected intermediate image versions from circulation once the corrected release is validated.