Harness outage history

We are currently investigating this issue.

The issue has been identified and a fix is being implemented.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

## **Summary** On October 19, 2026, between approximately 12:10 AM and 12:55 AM PDT, customers on Prod4 experienced errors and intermittent unavailability of Harness Pipelines services. Heavy pipeline executions drove a sharp rise in CPU and memory use on the pipeline service, and for a short period all service instances were unavailable. Harness engineering scaled the service both vertically and horizontally, which restored service. A fix for the underlying behavior is being rolled out. ## **Root Cause** On October 19th, high number of concurrent executions with very large execution graphs drove a sharp increase in cluster resources. The service stores and retrieves this full execution graph for visualization purposes and further progressing the pipelines to next states. Our auto-scalaers on Prod4 took longer than usual for scaling resources due to a sharp rise in CPU and memory.  As a part of the fix, the execution graph update will now process graph changes incrementally instead of the full graph. It is being rolled out in a phased manner and will be completely rolled out at the earliest. ## **Impact** * Affected users: Customers on Prod4. * Symptom: unavailability of pipeline functionality in the Harness platform. * Data: No data loss. * Suggested workaround: None required from customers. Harness engineering resolved the issue. * Duration of impact: Approximately 20 minutes of full unavailability and degraded performance for 25 minutes. Harness continued monitoring after recovery and confirmed platform stability with customers. ## **Remediation** **Immediate** Harness engineering scaled the pipeline service both vertically \(more resources per instance\) and horizontally \(more instances\), and removed instances in an unhealthy state. The instance count stabilized and normal service was restored. The system will stay scaled up till the fix is completely rolled out.  **Permanent fix and Action Items** To prevent such issues from happening again, Harness is 1. Tuning the auto heal/scaling process so that impact is mimimal to none. 2. Optimizing the execution graph flow so that large graphs can be process faster.

We are currently investigating this issue.

The issue has been identified and a fix is being implemented.

This incident has been resolved.

We are currently investigating this issue.

The issue has been identified and a fix is being implemented.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

We are currently investigating this issue.

The issue has been identified and a fix is being implemented.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

. .

We are currently experiencing HTTP 429 (rate-limiting) errors when Harness Cloud builds access Maven Central, which may impact build execution. Our team is actively working with Sonatype to resolve the issue as quickly as possible. We will provide further updates as we make progress.

This incident has been resolved.

tldr: \(There was a brief rate limiting by maven central\) More details to follow

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

We are currently investigating this issue.

We are continuing to investigate this issue.

The issue has been identified and a fix is being implemented.

A fix has been implemented and we are monitoring the results.

A fix has been implemented and we are monitoring the results. we have confirmation of this fixing the issue.

This incident has been resolved.

# 1. Summary On October 6, 2026, pipelines containing an IaCM stage were unable to start on Prod1 for approximately two hours. Affected users saw an error when the pipeline attempted to create its execution plan. Pipelines that did not use an IaCM stage continued to run normally throughout. The underlying cause was an internal logging defect in the service that handles IaCM stage execution: under a specific, rare timing condition, that service's own logging system could end up waiting on itself indefinitely, which in turn stalled the broader process responsible for preparing pipeline execution plans. The issue was triggered by an unusually large burst of log activity from one customer's pipeline. We resolved the immediate impact by rolling the service back to its previous stable version, and a permanent code fix is currently in testing. # 2. Impact * All pipelines using an IaCM stage on Prod1 were unable to start for approximately two hours. * Multiple customers on the Prod1 cluster were affected. * Pipelines that did not use an IaCM stage were not affected and continued running normally. * No data was lost as a result of this incident. * As a side effect of the rollback used to resolve this incident, an unrelated fix for one customer \(extending how long a certain authentication token remains valid during long-running pipelines\) was temporarily reverted. That fix was reapplied the same evening. # 3. Root Cause The service responsible for IaCM pipeline stages logs its activity through an internal queue: log messages are placed on the queue, and a single background worker continuously reads from it and writes the messages out. Separately, that service has a filter that can rewrite the severity of certain log messages back onto that same queue. Under a specific, narrow timing condition, this created a deadlock, if the queue happened to be completely full at the exact moment a rewritten message was being submitted back onto it, the one worker responsible for freeing up space on the queue would itself get stuck waiting for space to free up, a condition it could never resolve, since it was the only thing that could have freed that space. Once that happened, every other part of the service that needed to log anything also became stuck waiting, which cascaded into the pipeline-plan-creation process failing for any pipeline using an IaCM stage. # 4. Resolution * Rolled the service back to its last stable version, which resolved the immediate incident. # 5. Preventive Actions 1. **Optimize the underlying defect**: Optimize and tune the logging filter so that rewriting a message's severity happens on the thread that generated the log line, rather than being resubmitted through the same background worker's queue, eliminating the deadlock condition entirely. 2. **Enhance monitoring on logging-queue health**: add metrics and alerting on this internal queue so that this class of saturation pages our on-call team automatically rather than relying on customer reports. 3. **Evaluate limits and perform high stress tests with very large pipelines**:

On October 7, 2026, from 08:00 UTC to 11:00 UTC (1:30 PM to 4:30 PM IST), some pipeline executions on the Prod 7 cluster saw intermittent failures in Policy (OPA) step evaluations.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

### Summary Customers on Prod 2 experienced intermittent failures of a few Verify steps at startup. Affected steps failed shortly after launch with a message indicating Continuous Verification\(CV\) could not reach an internal Harness service. The issue stemmed from a recent service update that introduced connection pooling to improve performance and reliability with a keep-alive setting incompatible with the receiving service's timeout. No customer data was lost or corrupted. ### Customer Impact Customers using CV Verify steps on Prod-2 saw intermittent step failures during the incident window. The failure rate was low — the majority of executions completed successfully throughout the window. Pipelines that did not use CV Verify steps were unaffected. ### Root Cause A recent CV service update introduced connection pooling to an internal platform component, with a keep-alive duration longer than that component's idle timeout. Connections idle long enough were silently closed by the server; the CV service did not detect this and had no retry logic for such failures, causing them to surface directly as Verify step failures. The issue was intermittent because it only affected connections that happened to be idle long enough to expire. ### Mitigation The connection keep-alive duration was reduced to fall within the receiving service's idle timeout. Failures dropped to near zero immediately, and no further step failures of this type have been observed on any production environment. ### Next Steps To prevent recurrence, Harness will: * **Fix connection handling:** Tuned the default keep-alive and enabled automatic retry on connection failures so stale-connection errors are handled transparently. * **Add resilience at step startup:** Introduce retry with backoff for the internal service call at the Verify step start, so isolated transient failures do not surface as immediate step failures. * **Improve alerting:** Enhanced monitoring for internal connection error rates.

We are currently investigating this issue.

We are continuing to investigate this issue.

The issue has been identified and a fix is being implemented.

This incident has been resolved.

## Summary On **Tuesday, 6 October 2026**, IaCM pipelines that use the **OpenTofu plugin** in **Prod1** failed intermittently for a production accountAffected runs injected secrets successfully, then later steps timed out or failed authentication. The same pipeline often succeeded on retry, with no configuration change. Typical failing executions ran for about 90 minutes. Other Harness modules were not broadly impacted. We have no report of customer data loss. Pipelines that finished inside the token lifetime were unaffected by this defect. ## Root Cause The IaCM stages minted authentication tokens that expired after one hour. Policy checks later in a long run then rejected the token, so the step timed out or failed authentication. That one-hour lifetime was fixed. Runs of about 90 minutes crossed the one-hour mark, which is why the failures showed up on longer pipelines and often cleared on a shorter retry. A separate set of plan errors in the same account came from credential setup in the customer workspace and is outside this incident. ## Impact * **Product area:** IaCM pipelines using the IaCM plugins talking to OPA sdk. * **Customer experience:** intermittent timeouts and authentication failures after secret injection, on runs longer than about one hour. * **Environment:** Prod1, one production account * **Data:** No data loss reported. ‌ ## Remediation * **Immediate:** Opened a status page for the affected Prod1 account, and shipped an interim IaCM manager change for one class of the observed failures while the plugin fix was built. * **Permanent:** Shipped an OpenTofu plugin hotfix so token expiry is configurable. In production the lifetime was raised to about 6 hours, so a long pipeline does not outlive the token. The hotfix was validated in QA and deployed to Prod1. We apologise for the disruption caused. Please reach out to your Harness support contact if you have any further questions.

We are currently investigating this issue.

The issue has been identified and a fix is being implemented.

A fix has been implemented and we are monitoring the results.

We are continuing to monitor for any further issues.

Verified the fix and resolved SSO issues.

## Okta Application SP-Initiated SSO Login Failures ‌ # Summary ‌ Between October 5, 2026, 18:18 UTC and October 6, 2026, 00:26 UTC, some Harness customers were unable to sign in to the Harness Hosted MCP server via SAML Single Sign-On \(SSO\). The main Harness Portal user authentication remained fully operational. Affected users saw login errors or were redirected to an invalid SSO page instead of completing sign-in. ‌ The issue was caused by breaking changes in a version upgrade of the Harness identity service HarnessID. We fixed it with a series of hotfixes, and SSO login was fully restored for all affected customers. ‌ # Impact ‌ Users signing in to Harness MCP via SAML SSO were unable to complete authentication. Existing MCP sessions that relied on SSO could not be refreshed, so some users had to sign in again and could not. Non-SSO logins to Hosted MCP worked during most of the incident. No customer data was lost or exposed. Customers using their own generic SAML 2.0 application configuration in Okta were not affected. # Root Cause On September 29, 2026, an updated version of the Harness application was published to the Okta Integration Network. The update added support for registering multiple reply \(ACS\) URLs, so that customers running Harness across several regions or environments could configure more than one callback URL. ‌ Harness includes an AssertionConsumerServiceURL value in the SAML authentication request it sends to Okta. Before this update, Okta did not validate that value for application instances without additional ACS URLs configured and instead returned the SAML assertion to the application's default ACS URL. After the update, this validation became active across all existing application instances, including those where the customer had never configured an additional ACS URL. Because the requested value was not present in the instance's registered list, Okta returned an HTTP 400 error. ‌ The updated application went through Okta's standard OIN publishing review, which includes backward-compatibility verification against existing installations of the published integration. This change in validation behavior for instances with no additional ACS URLs configured was not surfaced during that review, so it reached customers at the time of release. # Remediation **Immediate Response** ‌ Two workarounds were published on the status page: sign in from the Harness tile in Okta \(IdP-initiated login\), which was unaffected, or register the application's Okta SSO URL in the Additional ACS URLs field of the Harness Okta application. Harness engaged each reported affected customer directly and validated their sign-in. ‌ **Resolution** ‌ Rolling the application back to the previous version was deliberately avoided, as that would only defer the same disruption to the next release. Every affected customer was instead unblocked through the ACS URL registration, and we are working with Okta to make the ACS URL validation conditional again, so that instances with no additional ACS URLs configured behave exactly as they did before, and no customer action is required. # Action Items To prevent such issues from happening again, Harness will take the following action items ‌ 1. Working with Okta to restore conditional ACS URL validation, so instances with no additional ACS URLs configured behave exactly as before and no customer action is required. 2. Publish a dedicated configuration guide for the Harness OIN application in Harness documentation, including reply \(ACS\) URL requirements. 3. Add an independent pre-release verification step against existing customer configurations for any future update to the published Okta application, alongside Okta's own review. 4. Migrate user authentication to Harness ID, which removes the dependency on this validation behavior entirely.

We are currently investigating this issue.

The issue has been identified and a fix is being implemented.

We identified an issue that applies only to SP-initiated login through the published Harness Okta application. After a recent update to that application, those login attempts returned an HTTP 400 error when the configured SSO URL was missing from Additional ACS URLs. General SAML 2.0 configurations in Okta are not affected. Immediate Solutions: Use IDP-initiated login. Start the login from the Harness application tile in Okta. Or Add your Harness SSO URL to Additional ACS URLs in the published Harness Okta application.

This incident has been resolved.

# 1. Summary Starting September 29, 2026, customers who sign in to Harness through the published Harness application in the Okta Integration Network \(OIN\) were unable to complete SP-initiated SSO login. Affected users received an HTTP 400 error from Okta instead of being signed in. The issue was introduced by a change in how Okta validated the requested reply \(ACS\) URL for existing installations of the published Harness application. Workarounds were published within an hour of the first report, and every affected customer was unblocked. # 2. Impact ‌ * Impact was limited to customers using the published Harness application from the Okta Integration Network for SP-initiated login \(starting sign-in from Harness\). Those attempts returned an HTTP 400 error from Okta. * IdP-initiated login, started from the Harness tile in Okta, continued to work throughout. * Customers using their own generic SAML 2.0 application configuration in Okta were not affected. * No customer data was lost or exposed. # 3. Root Cause On September 29, 2026, an updated version of the Harness application was published to the Okta Integration Network. The update added support for registering multiple reply \(ACS\) URLs, so that customers running Harness across several regions or environments could configure more than one callback URL. Harness includes an AssertionConsumerServiceURL value in the SAML authentication request it sends to Okta. Before this update, Okta did not validate that value for application instances that had no additional ACS URLs configured, and simply returned the SAML assertion to the application's default ACS URL. After the update, this validation became active on all existing instances of the application, including those where the customer had never configured an additional ACS URL. Because the requested value was not present in the instance's registered list, Okta rejected the authentication request with an HTTP 400 error. The updated application went through Okta's standard OIN publishing review, which includes backward compatibility verification against existing installations of a published integration. This change in validation behavior for instances with no additional ACS URLs configured was not surfaced during that review, so it reached customers at release. # 4. Remediation **Immediate Response** Two workarounds were published on the status page: sign in from the Harness tile in Okta \(IdP-initiated login\), which was unaffected; or register the application's Okta SSO URL in the Additional ACS URLs field of the Harness Okta application. Support engaged each affected customer directly and validated sign-in with them. ‌ **Resolution** Rolling the application back to the previous version was deliberately avoided, as that would only defer the same disruption to the next release. Every affected customer was instead unblocked through the ACS URL registration, and we are working with Okta to make the ACS URL validation conditional again, so that instances with no additional ACS URLs configured behave exactly as they did before and no customer action is required. # 5. Action Items To prevent such issues from happening again, we are working on | **Action Item** | | --- | | Working with Okta to restore conditional ACS URL validation, so instances with no additional ACS URLs configured behave exactly as before and no customer action is required. | | Publish a dedicated configuration guide for the Harness OIN application in Harness documentation, including reply \(ACS\) URL requirements. | | Add an independent pre-release verification step against existing customer configurations for any future update to the published Okta application, alongside Okta's own review. | | Migrate user authentication to Harness ID, which removes the dependency on this validation behavior entirely. |

We are currently investigating this issue.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

### Summary Between approximately **11:37 AM and 3:01 PM PDT \(18:37–22:01 UTC\) on September 29, 2026**, customers using Harness CI self-hosted Linux cloud runners in **prod1/prod2 \(us-west1\)** experienced intermittent networking failures during pipeline execution — primarily Git/network timeouts when fetching source code. **Root Cause** This was primarily due to saturation of our NAT which was futher exacerbated by a temporary cloud provider compute capacity, preventing new VM provisioning there. ### Impact * Customers running self-hosted Linux CI builds in us-west1 saw **intermittent build/pipeline failures**, not a full outage. * Observed customer-facing errors included Git fetch timeouts, e.g. `Failed to connect to github.com port 443: Connection timed out` **Remediation** * **Immediate:** Rebalanced ARM build-VM load from the saturated us-west1 path toward central1 \(partially limited by central1 capacity\); restarted the affected Cloud NAT gateways, which stabilized pipeline networking. **Action Items** 1. Secure additional NAT gateway/IP capacity \(In progress\) 2. Improve NAT saturation detection and alerting ahead of customer impact. 3. Evaluate cross-region capacity contingencies so a single region's compute shortage doesn't block failover mitigation.

We are currently investigating this issue.

A fix has been implemented and we are monitoring the results.

We are continuing to monitor for any further issues.

This incident has been resolved.

**Summary** Customers on Prod-2 experienced elevated failure rates in Continuous Verification pipelines. Verification jobs failed to complete or reported errors because CV data-collection workers could not communicate with the platform service that coordinates their work. This was caused by an unexpected traffic spike to one of our backend systems.  Service was restored by scaling out the affected platform component. No data loss or corruption occurred. ‌ **Customer Impact** Customers using Continuous Verification on Prod-2 saw verification pipeline jobs fail or time out during the incident window. A significant fraction of verification job instances that ran during this period ended in an operational failure. Pipelines not using CV continued to operate normally. ‌ **Root Cause** A large automated provisioning run created a significant number of persistent CV monitoring workers in a short window, which overwhelmed a shared platform component. The sudden increase in concurrent outbound calls exhausted the available network connections on each service instance.   ‌ **Mitigation** The maximum scaling limit for the CV service was raised, allowing it to expand capacity to meet demand. Failures stopped shortly after the scale-out was completed. ‌ **Next Steps** To prevent recurrence, Harness will: * **Increase capacity headroom and add provisioning guardrails:** Raise the CV service scaling limit to maintain headroom under burst loads, and introduce rate limiting for bulk provisioning of monitored services. * **Improve Resiliency:** Add exponential backoff and jitter for worker retries, and use a reusable connection pool to reduce overhead and prevent transient failures from compounding under high concurrency. * **Improve monitoring and alerting:** Enhance alerting for abnormal per-account worker growth rates, so similar conditions are detected and acted on earlier.

On 28th September 2026, we observed an issue with copying files from the Harness Filestore during the export step, where the files were being copied as null/empty files. As a result, all pipelines using Filestore APIs were blocked during the incident window. The issue was caused due to a Feature Flag rollout which was reverted once the issue was discovered. All the operations are back to normal.

### Summary On September 28, 2026, some customers on Prod-2 experienced failures in CD pipeline steps that read File Store content using pipeline expressions \(e.g., ` `\). Affected expressions returned null instead of the expected file content, causing dependent steps to fail or behave incorrectly. The issue was traced to a platform update that tightened authorization checks on File Store APIs. The update was rolled back, and File Store expressions have been working normally since. ### Customer Impact * Only pipeline steps on Prod-2 that read File Store content via expressions were affected. Pipelines that do not use File Store expressions were not impacted. * Affected expressions resolved to null, causing steps that depend on configuration or settings files to fail or produce incorrect output. * No File Store content was modified. No files, secrets, credentials, or artifacts were exposed, and no data was lost. ### Root Cause As part of an ongoing security initiative, Harness deployed stricter authorization checks across several platform APIs, including the API that serves File Store content. When a pipeline evaluates a File Store expression, one internal Harness service calls another to retrieve the file. The new authorization checks failed, resulting in the requests being rejected and the expressions returning null. ### Mitigation Once the platform update was identified as the cause, it was rolled back across the production environment. File Store expressions have resolved correctly since the rollback. No customer action is required. ### Preventive Measures & Next Steps * **Audit affected APIs:** Review all APIs where the stricter authorization checks were applied to ensure internal service-to-service calls are correctly accounted for. * **Improve test coverage:** Enhance automated test suites to include File Store expression evaluation across pipeline execution paths

We are currently investigating connectivity issues affecting customers whose environments are protected by Palo Alto Networks firewalls. These firewalls are blocking connections to Harness APIs, which may result in: - Failed pipeline executions - Inability to deploy applications - Intermittent timeouts when accessing Harness platform features - Integration failures with third-party tools - Delegate connectivity with Harness platform This issue is not caused by a Harness service outage, but rather by firewall rules on your network side. We are advising customers to make an exception for our DNS endpoints.

Mitigation Available: Harness Traffic Blocked by Network Security Policy We have identified an issue where some customers may be unable to access Harness because Harness traffic is being categorized as “AI-Platform” by Palo Alto network security controls and subsequently blocked by the customer's firewall/INET policy. If you are experiencing this issue: Please contact your organization's Network/Firewall/CyberSecurity team and request that the required Harness traffic be allowed through your Palo Alto firewall policy. As a mitigation, customers can allow the “AI-Platform” URL category for the required Harness endpoints, or add the affected Harness domains/FQDNs to their organization's approved/allow list, in accordance with their security policies. After the firewall policy is updated, access to Harness should be restored. Important: The firewall policy is managed by the customer's network/security team. If the Harness endpoint is incorrectly categorized by Palo Alto, customers can also request a URL-category review from Palo Alto Networks.

After consultation with Palo Alto Networks, we can confirm that on Sept 23rd 2026, app.harness.io was categorized as an AI Platform Service by the Palo Alto Networks Team.  Palo Alto Networks sent a notice to customers in August about the reclassification. This change may place AI Platform Service under a different policy than app.harness.io's previous ruleset categorization, Computer and Internet Info.  The policy applied is dependent on customer settings. Harness Customers should discuss this categorization and the alerts with their internal network team and Palo Alto Networks.

.

We are currently investigating this issue.

The issue has been identified and a fix is being implemented.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

# Summary On September 22, 2026, three of our outbound network gateways in one region became unhealthy and began rejecting a small percentage of new outbound connection attempts. The issue was resolved the same day problem completely with no further recurrence or follow-on impact.The underlying cause was a saturated internal traffic-monitoring buffer on those gateways, which caused their health checks to respond too slowly and get marked unhealthy.  # Impact * Only three gateway instances in one region were affected; other regions and other gateways in the same region were not impacted. * There was no cascading failure , the issue was fully resolved with a single round of gateway restarts, and no follow-on issues occurred afterward. # Root Cause  Our outbound network gateways run an internal component that monitors network traffic for operational visibility. Under sustained high traffic, this component's internal buffer became saturated, which created backpressure severe enough to slow down the gateway's own health-check responses beyond the timeout our load balancer allows. Once that timeout was exceeded, the load balancer marked the affected gateways unhealthy and began rejecting a portion of new connections destined for them. Actual packet forwarding continued to work normally throughout the incident. This was specifically a health-check and monitoring-buffer issue, not a failure of the gateways' core forwarding capability. # Resolution The three affected gateway instances were manually restarted, which cleared the saturated monitoring buffer and immediately restored normal health-check behavior. Connection rejections stopped as soon as the restart completed, and no further intervention was needed. # Preventive Actions | **Action** | | --- | | Replace the current traffic-monitoring approach on all gateways with a more efficient, lower-overhead implementation that cannot saturate in this way, and that alerts well before any customer-visible impact. | | Implement automatic, periodic replacement of long-running gateway instances on a fixed schedule, so no instance can accumulate the kind of age-related degradation seen in this incident. | | Migrate gateway instances to network-optimized infrastructure with substantially more dedicated network capacity, better suited to sustained high-throughput traffic. | | Enhance existing monitoring buffer's utilization while the longer-term replacement is rolled out. | | Enhance alerting specifically on gateway health-check response latency, to catch early signs of degradation before it becomes customer-visible. |

The issue has been identified and a fix is being implemented.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

Same as the previous root cause.

We are currently investigating this issue.

The issue has been identified and a fix is being implemented.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

## Summary On Sep 21, 2026, customers in Prod 2 experienced degraded and then largely unavailable deployment dashboards for approximately three hours. The issue was caused by a set of analytics queries behind those dashboards reading far more data than they needed, which exhausted the capacity of the database serving them. A rolling restart of the affected service restored dashboards fully, and the platform has been stable since. ## Customer Impact Dashboard pages loaded slowly or timed out between 14:48 and 18:11 UTC. Pipeline execution and deployment recording run on separate infrastructure and continued to operate normally. No data loss or corruption occurred. ## Root Cause The dashboard queries were missing a time-range filter on the column used to split the largest table into weekly segments, so each query read all historical data instead of only the requested period. Correctness was never affected — the dashboards returned the right results; only the amount of data read to produce them was excessive. As data volume grew, the cost of those reads exceeded what the database could serve, and retries of the failing queries added further load, so the condition did not clear until the service was restarted. ## Mitigation * Performed a rolling restart of the affected service, which stopped the retry cycle and allowed the database to complete its outstanding work * Confirmed dashboard latency and database connection usage returned to normal levels ## Next Steps To prevent recurrence, Harness will: * Add the missing time-range filter to the affected dashboard queries so each reads only the requested period * Replace immediate retries with bounded, backing-off retries * Improve detection by adding alerting on database connection usage and per-query execution cost

The issue has been identified and a fix is being implemented.

We are currently investigating this issue.