Harness Status

Testing Dev in QA

This incident has been resolved.

This incident has been resolved.

We are currently investigating this issue.

This incident has been resolved.

## **Summary** _Between 13:30 UTC on June 5, 2026, and 00:27:00 UTC on June 6, 2026, customers using Harness IDP experienced intermittent 503 errors when accessing IDP web pages and APIs. The incident was mitigated through a configuration update and normal operation was restored by 00:27 UTC._ ## **Root Cause** _The Root Cause was traced back to a performance optimization update deployed around 10:35 UTC on June 5, 2026. This network transport configuration change altered idle TCP connection behavior, causing long-lived connections to be terminated prematurely under certain conditions. Consequently, requests failed to reach backend services, leading to intermittent 503 errors in the Harness IDP service. Stability was restored by recalibrating connection timeouts and lifecycle settings to synchronize connection management across the network path and eliminate stale connection reuse._ ## **Impact** _Starting at approximately ~13:30 UTC, customers experienced intermittent 503 service errors while using the Harness Internal Developer Portal \(IDP\) until we resolved the issue at 00:27:00 UTC. The issue impacted both UI and API traffic, resulting in failed requests and a degradation of service availability until mitigation measures were implemented._ ## **Remediation** _The team implemented a configuration change to connection management settings, reducing idle connection timeouts and limiting connection lifetime._  _This prevented the reuse of stale connections and restored service stability._ ## **Action Items** 1. _**Enhance monitoring & alerting** – Enhance synthetic alerting and coverage for upstream connection failures, connection resets, and elevated 503 error rates to enable earlier detection of network transport issues._ 2. _**Pre-production validation of networking changes** – Expand pre-production checks for configuration alignment and testing for network timeout and connection management scenarios to reduce the risk of similar issues in the future._

This incident has been resolved.

This incident has been resolved.

This incident has been resolved.

### **Summary** Between May 27 and June 1, 2026, some Harness CI customers experienced pipeline execution failures with the error: `failed to call LE.RetryStartStep: context deadline exceeded` ### **Impact** Affected customers saw intermittent CI pipeline failures during step execution. Existing pipeline definitions, customer data, source code, and artifacts were not impacted. ### **Root Cause** The root cause was a deadlock in the Light Engine logging path. When the log service returned an error, the Light Engine log writer attempted to reacquire a mutex it already held. This caused the Light Engine process to freeze, which led to step execution timeouts and pipeline failures.   ### **Mitigation and Resolution** Harness Engineering took multiple mitigation steps during the incident, including: * Rolled back affected runner versions where needed * Increased relevant timeout configurations * Reduced log-service load and latency * Temporarily disabled the affected livelog streaming path * Migrated selected workloads across regions and infrastructure providers * Pinned a fixed Light Engine version through runner configuration The final fix addressed the mutex deadlock in the Light Engine log writer and prevented the same lock from being reacquired while already held.   ### **Prevention and Follow-Up Actions** Harness is taking the following actions to reduce recurrence risk: * Improve deadlock detection in critical concurrent code paths * Strengthen error handling for log-service interactions * Add better monitoring for Light Engine process health * Improve safeguards around logging-path failures * Continue reviewing runner rollout and validation processes

Our cloud provider is going through a maintenance and during the maintenance window dashboards will not be available to view.

We will be undergoing scheduled maintenance during this time.

Testing Dev Maintenance

The scheduled maintenance has been completed.

The scheduled maintenance has been completed.

The scheduled maintenance has been completed.

The scheduled maintenance has been completed.

The scheduled maintenance has been completed.

The scheduled maintenance has been completed.

The scheduled maintenance has been completed.