Split 장애 기록

1. 2026년 7월 9일 PM 8:21 UTCinvestigating

   우리는 현재이 문제를 조사하고 있습니다.

2. 2026년 7월 9일 PM 9:10 UTCmonitoring

   수정이 구현되어 결과를 모니터링하고 있습니다.

3. 2026년 7월 9일 PM 9:46 UTCresolved

   이 사건은 해결되었습니다.

공식 인시던트 업데이트를 자동 번역했습니다.

1. 2026년 7월 9일 AM 9:22 UTCinvestigating

   자주 묻는 질문

공식 인시던트 업데이트를 자동 번역했습니다.

1. 2026년 7월 8일 AM 7:49 UTCinvestigating

   우리는 현재 Prod3 클러스터의 문제점을 투자하고 있습니다.

2. 2026년 7월 8일 PM 2:40 UTCinvestigating

   우리는 현재이 문제를 조사하고 있습니다.

3. 2026년 7월 8일 PM 4:51 UTCmonitoring

   수정이 구현되어 결과를 모니터링하고 있습니다.

4. 2026년 7월 8일 PM 6:56 UTCresolved

   이 사건은 해결되었습니다.

5. 2026년 7월 14일 AM 12:14 UTCpostmortem

   ## 요약 7월 7일부터 7월 8일까지, 2026년까지, 당사의 Prod 3 환경에서 호스팅되는 고객 계정은 파이프 라인 및 리필 템플릿을 로딩할 때 간헐적인 슬픔과 실패를 경험했습니다. 우리의 핵심 플랫폼 백엔드 서비스에 우리의 템플렛 가공 서비스에서 내부 특징 플랩 보기 교통에 있는 지속적인 증가는 날카로운 원인이었습니다. 2개의 별도의 날에 발표된 문제점. Day 1 \ (7 월 7 \)에서, 우리의 팀은 영향을받는 서비스 및 용량을 추가하여 인프라 수준의 완화를 통해 서비스를 복원했습니다. 그 원인은 여전히 존재했기 때문에, 같은 실패 모드는 Day 2 \ (7 월 8 \)에 재발했다. Deeper investigation on Day 2 특정 기능 플래그 및 코드 경로 책임, 그리고 엔지니어링은 고정을 배송 그래서 플래그 체크는 이제 로컬 캐시에서 제공. 이 수정은 모든 생산 클러스터를 통해 배포되었으며 완전히 문제를 해결했습니다. 그것은 이후 재발하지 않았습니다. 우리는 동일한 캐싱 간격을 위한 플랫폼에 다른 모든 특징 깃발을 감사하고, 이전 문제점을 붙잡기 위하여 예방적인 측정 단면도에서 설명된 과정과 감시 개선을 실행하고 반복의 위험을 감소시킵니다. ## 충격 **석사:** Intermittent Slowness 또는 고장 로드 파이프라인; 느린 또는 실패 템플릿 유효성 및 해결; 플랫폼 서비스 간의 연결 / 시간 오류. **기능 충격:** 템플릿 해상도에 따라 파이프 실행 및 파이프라인 스튜디오 작업이 지연되거나 위의 타임 라인에 설명 된 활성 창에서 실패했습니다. 몇몇 실행 필수 설명서 retry ## 뿌리 원인 우리의 핵심 백엔드 서비스는 최근 릴리스를 따르는 내부 교통의 크고 성장한 볼륨을 구동하지 않기 때문에 지속적인, uncached feature-flag lookup 때문에 요청에 도착 비율로 유지될 수 없습니다. ## 소송 * 서비스 및 추가 용량에서 포화 백엔드 인스턴스(s\)를 제거하여 급성 병목을 구호합니다. * 영구 수정 \ (일 2 \) : 모든 검색에 백엔드 서비스를 호출하는 대신 지역 캐시에서 책임있는 기능 표시 표시를 사용하여 템플릿 처리 서비스를 업데이트하여 소스에서 과잉 내부 트래픽을 제거하십시오. * 수정은 모든 생산 클러스터에 배포하고 정상 내부 트래픽 볼륨 및 응답 시간에 반환을 통해 확인되었습니다. ## 예방 조치 및 다음 단계 * 특징 플랩 캐싱 감사 : 우리는이 문제의 반복을 일으킬 수 있기 전에 유사한 캐싱 간격을 식별하고 닫는 플랫폼을 통해 고주파 코드 경로에 사용되는 모든 기능 플래그를 검토하고 있습니다. * 방출 유효성: 우리는 새로운 특징 flag-gated 부호 경로를 위한 미리 방출 짐 테스트를 강화하고 이렇게 이 종류의 통화 부피 회귀는 생산을 도달하기 전에 붙잡습니다. * 수용량 경고: 우리는 backend request-queue 포화에 proactive 경고를 추가하고 병목의 이 종류가 붙잡고 mitigated 자동적으로, 그것 때문에 소비자 만족도에 영향을 미치.

공식 인시던트 업데이트를 자동 번역했습니다.

1. 2026년 7월 7일 AM 10:40 UTCinvestigating

   현재 IACM 모듈을 사용하여 문제가있는 EU1의 문제점을 조사하고 있습니다.

2. 2026년 7월 7일 PM 2:00 UTCmonitoring

   우리는이 문제를 해결 한 우리의 측면에 대한 수정을 구현했습니다. 우리는이 문제를 모니터링하고 있습니다.

3. 2026년 7월 7일 PM 2:10 UTCresolved

   이 사건은 해결되었습니다.

공식 인시던트 업데이트를 자동 번역했습니다.

1. 2026년 7월 7일 AM 10:22 UTCinvestigating

   우리는 현재 Prod3 클러스터의 문제점을 조사하고 있습니다. 파이프라인 구성 요소가 예상보다 느리게로드됩니다.

2. 2026년 7월 7일 PM 12:19 UTCmonitoring

   수정은 문제를 해결한 파이프라인 템플릿을 다루는 우리의 마이크로서비스 중 하나입니다. 우리는 문제를 감시하는 것을 계속합니다.

3. 2026년 7월 7일 PM 12:36 UTCresolved

   이 사건은 해결되었습니다.

4. 2026년 7월 14일 AM 12:04 UTCpostmortem

   ## 요약 7월 7일부터 7월 8일까지, 2026년까지, 당사의 Prod 3 환경에서 호스팅되는 고객 계정은 파이프 라인 및 리필 템플릿을 로딩할 때 간헐적인 슬픔과 실패를 경험했습니다. 우리의 핵심 플랫폼 백엔드 서비스에 우리의 템플렛 가공 서비스에서 내부 특징 플랩 보기 교통에 있는 지속적인 증가는 날카로운 원인이었습니다. 2개의 별도의 날에 발표된 문제점. Day 1 \ (7 월 7 \)에서, 우리의 팀은 영향을받는 서비스 및 용량을 추가하여 인프라 수준의 완화를 통해 서비스를 복원했습니다. 그 원인은 여전히 존재했기 때문에, 같은 실패 모드는 Day 2 \ (7 월 8 \)에 재발했다. Deeper investigation on Day 2 특정 기능 플래그 및 코드 경로 책임, 그리고 엔지니어링은 고정을 배송 그래서 플래그 체크는 이제 로컬 캐시에서 제공. 이 수정은 모든 생산 클러스터를 통해 배포되었으며 완전히 문제를 해결했습니다. 그것은 이후 재발하지 않았습니다. 우리는 동일한 캐싱 간격을 위한 플랫폼에 다른 모든 특징 깃발을 감사하고, 이전 문제점을 붙잡기 위하여 예방적인 측정 단면도에서 설명된 과정과 감시 개선을 실행하고 반복의 위험을 감소시킵니다. ## 충격 **석사:** Intermittent Slowness 또는 고장 로드 파이프라인; 느린 또는 실패 템플릿 유효성 및 해결; 플랫폼 서비스 간의 연결 / 시간 오류. **기능 충격:** 템플릿 해상도에 따라 파이프 실행 및 파이프라인 스튜디오 작업이 지연되거나 위의 타임 라인에 설명 된 활성 창에서 실패했습니다. 몇몇 실행 필수 설명서 retry ## 뿌리 원인 우리의 핵심 백엔드 서비스는 최근 릴리스를 따르는 내부 교통의 크고 성장한 볼륨을 구동하지 않기 때문에 지속적인, uncached feature-flag lookup 때문에 요청에 도착 비율로 유지될 수 없습니다. ## 소송 * 서비스 및 추가 용량에서 포화 백엔드 인스턴스(s\)를 제거하여 급성 병목을 구호합니다. * 영구 수정 \ (일 2 \) : 모든 검색에 백엔드 서비스를 호출하는 대신 지역 캐시에서 책임있는 기능 표시 표시를 사용하여 템플릿 처리 서비스를 업데이트하여 소스에서 과잉 내부 트래픽을 제거하십시오. * 수정은 모든 생산 클러스터에 배포하고 정상 내부 트래픽 볼륨 및 응답 시간에 반환을 통해 확인되었습니다. ## 예방 조치 및 다음 단계 * 특징 플랩 캐싱 감사 : 우리는이 문제의 반복을 일으킬 수 있기 전에 유사한 캐싱 간격을 식별하고 닫는 플랫폼을 통해 고주파 코드 경로에 사용되는 모든 기능 플래그를 검토하고 있습니다. * 방출 유효성: 우리는 새로운 특징 flag-gated 부호 경로를 위한 미리 방출 짐 테스트를 강화하고 이렇게 이 종류의 통화 부피 회귀는 생산을 도달하기 전에 붙잡습니다. * 수용량 경고: 우리는 backend request-queue 포화에 proactive 경고를 추가하고 병목의 이 종류가 붙잡고 mitigated 자동적으로, 그것 때문에 소비자 만족도에 영향을 미치.

공식 인시던트 업데이트를 자동 번역했습니다.

1. 2026년 7월 2일 PM 3:48 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

2. 2026년 7월 2일 PM 8:10 UTCmonitoring

   We are seeing the backlogged items worked through and continuing to monitor results.

3. 2026년 7월 2일 PM 11:31 UTCmonitoring

   We are continuing to monitor results while the backlog is worked through.

4. 2026년 7월 3일 PM 4:11 UTCresolved

   This incident has been resolved.

5. 2026년 7월 9일 PM 7:24 UTCpostmortem

   **Incident Summary** Harness FME \(Feature Management & Experimentation\) faced substantial delays in scheduled impressions data exports on Jul 2, 2026. This issue stemmed from performance degradation within Tinybird's infrastructure, affecting query execution crucial to our export pipeline. **Root Cause** The core issue was a temporary **performance degradation on the provider's infrastructure**. This directly impacted the metadata retrieval endpoint, which failed to return job IDs intermittently. Consequently, completed export jobs underwent repeated retries, leading to a backlog in the export queue. **Impact** * The export backlog accumulated, peaking at more than 6 hours behind the planned schedule. * No data loss was detected during the incident. ‌ **Mitigation** 1. **Adjusted Temporal workflows:** Extended timeouts from ~30 to ~50 minutes and increased polling attempts from ~180 to ~300. 2. **Expanded resource capacity.** 3. **Traffic redirection:** Systematically shifted export workloads over to an optimized in-house cluster. 4. **Manual intervention:** Terminated stalling workflows stuck in retry loops from failed metadata lookups. ‌ **Next Steps** Complete the migration of all remaining export jobs to the internal analytics cluster \(In progress\).

1. 2026년 6월 30일 AM 9:18 UTCscheduled

   Testing Maintenance

2. 2026년 7월 1일 AM 9:16 UTCin_progress

   Scheduled maintenance is currently in progress. We will provide updates as necessary.

3. 2026년 7월 1일 AM 10:16 UTCcompleted

   The scheduled maintenance has been completed.

1. 2026년 6월 30일 AM 9:16 UTCinvestigating

   Dev Statuspage incident - Testing

1. 2026년 6월 29일 PM 2:56 UTCinvestigating

   We are currently investigating this issue.

2. 2026년 6월 29일 PM 2:57 UTCidentified

   Harness is investigating an issue affecting our Prod 4 environments. Our log-services are affected and harness is currently working on restoring function.  This can affect our CI, Chaos, IACM modules.

3. 2026년 6월 29일 PM 3:20 UTCidentified

   We are continuing to work on a fix for this issue.

4. 2026년 6월 29일 PM 3:20 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

5. 2026년 6월 29일 PM 3:41 UTCresolved

   This incident has been resolved.

6. 2026년 7월 2일 AM 12:25 UTCpostmortem

   ### **Summary** During a planned security improvement to rotate service authentication secrets in the Prod 4 environment, an incorrect secret value was inadvertently applied to a subset of internal services. This resulted in authentication failures between platform components, impacting CI initialization, delegate task creation, and log streaming for certain pipeline executions. The issue was resolved by rolling back the token configuration to the previous known-good version and validating service recovery. ### **Impact** **Affected environment:** Prod 4 **Customer-visible impact:** * CI builds failed during initialization with errors such as: * _“Could not fetch token from log service”_ * _“token in request not authorized for receiving tokens”_ \(HTTP 400\) * CD and IaCM pipeline stages continued to execute but did not stream logs. * Some Shell Script steps failed during execution. * Delegate tasks could not be created or assigned for some pipeline executions. No customer data was lost or corrupted. ### **Root Cause** As part of a planned security initiative, authentication secrets used for communication between internal platform services were rotated in the Prod 4 environment. During the rotation process, an incorrect secret value was inadvertently configured for one of the services. Instead of applying the newly generated secret in the expected format, a default/incorrect value was introduced. This created a mismatch between services that authenticate using the shared token, causing authentication requests to be rejected with HTTP 400 errors. The authentication failures prevented CI components from obtaining log service tokens, disrupted delegate task creation, and prevented log streaming for affected pipeline executions. ### **Resolution** Engineering responded by: * Reverting the token configuration to the previous known-good secret. * Restoring consistent authentication across affected services. * Validating recovery through CI pipeline execution tests. * Verifying CD and IaCM pipeline execution and log streaming. * Executing a comprehensive validation suite to confirm platform functionality before closing the incident. ### **Follow-up Actions** To reduce the likelihood of similar incidents in the future, Harness will: * Implement additional pre-rotation and post-rotation validation checks to verify that the correct secret values and formats are applied before changes are activated. * Introduce automated verification of authentication between dependent services immediately following secret rotations. * Strengthen operational safeguards and deployment validation for security credential rotation procedures to detect configuration mismatches before they can impact customer workloads.

1. 2026년 6월 29일 PM 12:38 UTCinvestigating

   Testing CD outage

2. 2026년 6월 29일 PM 12:51 UTCinvestigating

   We are continuing to investigate this issue.

3. 2026년 6월 29일 PM 12:56 UTCresolved

   This incident has been resolved.

1. 2026년 6월 26일 AM 7:20 UTCinvestigating

   We are currently experiencing intermittent login issues with the Traceable cluster due to an issue with our login provider. We are actively working with the provider to resolve the issue. This incident does not impact data ingestion, and all ingestion pipelines continue to function normally. We will provide updates as we have more information.

2. 2026년 6월 26일 AM 7:20 UTCidentified

   The issue has been identified and a fix is being implemented.

3. 2026년 6월 26일 AM 7:29 UTCmonitoring

   Issue has been fixed and we are closely monitoring

4. 2026년 6월 26일 AM 7:29 UTCmonitoring

   We are continuing to monitor for any further issues.

5. 2026년 6월 26일 AM 7:31 UTCresolved

   This incident has been resolved.

6. 2026년 6월 30일 PM 5:13 UTCpostmortem

   ## **Summary** ‌ _Between **06:50 UTC and 07:20 UTC on June 26, 2026**, customers experienced intermittent login failures while accessing Traceable environments across multiple US clusters. The incident was traced to an issue with the external authentication provider \(Auth0\), where elevated socket timeouts caused increased login latency and authentication request failures. Login functionality gradually recovered as the upstream issue stabilized, and the incident was resolved after successful login validation across multiple impacted clusters._ ## **Root Cause** _The root cause was an issue with the external authentication provider \(Auth0\), which experienced elevated socket timeouts while processing authentication requests. These upstream timeouts increased login latency and caused intermittent authentication failures across multiple US-region clusters. Since authentication requests depended on the external provider, affected login attempts failed despite Traceable platform services remaining healthy. The incident was resolved once the upstream authentication service recovered and login requests consistently completed successfully._ ## **Impact** _Starting at approximately 06:50 UTC, customers experienced intermittent login failures when accessing Traceable environments across multiple US clusters. The issue affected user authentication, preventing some users from accessing the platform while underlying application services remained operational. Login functionality progressively recovered during the incident, and normal authentication was restored by 07:20 UTC._ ## **Remediation** _The engineering team worked with the external authentication provider while continuously monitoring authentication health across affected clusters. Login functionality was validated through platform metrics and manual verification across representative environments. After confirming consistent authentication success across impacted clusters, the incident was declared resolved._ ## **Action Items** _To prevent such issues going forward, Harness will,_   _Increase authentication resilience: Evaluate improvements to authentication request handling, including timeout tuning, retry strategies where appropriate, and graceful degradation for transient upstream failures._

1. 2026년 6월 25일 AM 5:30 UTCscheduled

   We’ll be performing a backend infrastructure upgrade to improve reliability and compliance. During this maintenance window, Harness platform may experience brief, intermittent interruptions. We appreciate your understanding and are working to minimize any impact.

2. 2026년 6월 29일 AM 3:30 UTCscheduled

   Maintenance will begin as scheduled in 60 minutes.

1. 2026년 6월 24일 PM 12:19 UTCinvestigating

   Harness login is failing. We are investigating the problem.

2. 2026년 6월 24일 PM 12:26 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

3. 2026년 6월 24일 PM 12:27 UTCmonitoring

   We are continuing to monitor for any further issues.

4. 2026년 6월 24일 PM 2:37 UTCresolved

   This incident has been resolved.

5. 2026년 7월 2일 AM 12:26 UTCpostmortem

   ### **Summary** During a planned security improvement to rotate service authentication secrets in the EU1 environment, an out of sequence secret value was inadvertently applied to a subset of internal services. This resulted in authentication failures between platform components, causing login failures. The issue was resolved by rolling back the token configuration to the previous known-good version and validating service recovery. ### **Impact** **Affected environment:** EU1 **Customer-visible impact: Login Failures** No customer data was lost or corrupted. ### **Root Cause** As part of a planned security initiative, authentication secrets used for communication between internal platform services were rotated in the EU1 . During the rotation process, an secret value was inadvertently misconfigured for one of the services. There was a newer format of secret and had to be applied in a certain order. This created a mismatch between services that authenticate using the shared token, causing authentication requests to be rejected with HTTP 400 errors. The authentication failures prevented CI components from obtaining log service tokens, disrupted delegate task creation, and prevented log streaming for affected pipeline executions. ### **Resolution** Engineering responded by: * Reverting the token configuration to the previous known-good secret. * Restoring consistent authentication across affected services. ### **Follow-up Actions** To reduce the likelihood of similar incidents in the future, Harness will: * Implement additional pre-rotation and post-rotation validation checks to verify that the correct secret values and formats are applied before changes are activated. * Introduce automated verification of authentication between dependent services immediately following secret rotations. * Strengthen operational safeguards and deployment validation for security credential rotation procedures to detect configuration mismatches before they can impact customer workloads.

1. 2026년 6월 23일 AM 6:46 UTCidentified

   We have identified an issue affecting a small subset of accounts in the EU1 region. As a result, data ingestion for impacted integrations may have stopped.

1. 2026년 6월 22일 PM 5:55 UTCscheduled

   FME will be undergoing scheduled critical maintenance during this window. No downtime is expected, and all services are anticipated to remain available. As part of this maintenance, updates or modifications to Large Segments will be temporarily unavailable until the activity is completed.

2. 2026년 6월 26일 PM 3:07 UTCscheduled

   We will be undergoing scheduled maintenance during this time.

3. 2026년 6월 30일 PM 3:30 UTCscheduled

   Maintenance will begin as scheduled in 60 minutes.

4. 2026년 6월 30일 PM 4:30 UTCin_progress

   Scheduled maintenance is currently in progress. We will provide updates as necessary.

5. 2026년 6월 30일 PM 8:30 UTCcompleted

   The scheduled maintenance has been completed.

1. 2026년 6월 19일 AM 11:18 UTCscheduled

   We’ll be performing a backend infrastructure upgrade to improve reliability and compliance. During this maintenance window, Harness platform may experience brief, intermittent interruptions. We appreciate your understanding and are working to minimize any impact.

2. 2026년 6월 22일 AM 2:30 UTCscheduled

   Maintenance will begin as scheduled in 60 minutes.

3. 2026년 6월 22일 AM 3:30 UTCin_progress

   Scheduled maintenance is currently in progress. We will provide updates as necessary.

4. 2026년 6월 22일 AM 4:36 UTCcompleted

   The scheduled maintenance has been completed.

1. 2026년 6월 19일 AM 9:10 UTCinvestigating

   Testing Dev in QA

1. 2026년 6월 17일 PM 4:44 UTCinvestigating

   We are currently investigating this issue.

2. 2026년 6월 17일 PM 4:45 UTCidentified

   The issue has been identified and a fix is being implemented.

3. 2026년 6월 17일 PM 4:45 UTCidentified

   IDP Systems' communication was interrupted due to an update.  Harness is rolling back changes. Impacted environments - prod0, prod1, prod2

4. 2026년 6월 17일 PM 4:47 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

5. 2026년 6월 17일 PM 5:50 UTCresolved

   This incident has been resolved.

6. 2026년 6월 24일 AM 12:50 UTCpostmortem

   # **Customer RCA: Executive Summary** On June 17, 2026, between 9:30 PM and 10:15 PM IST, customers in certain production environments experienced failures when accessing IDP \(Internal Developer Portal\) workflows. The issue was caused by a configuration mismatch in service authentication settings. ## **Impact** * **Affected Service:** IDP workflows * **Duration:** Approximately 45 minutes \(9:30 PM - 10:15 PM IST\) * **Customer Impact:** Users were unable to load IDP workflows during this period. * **Data Loss:** None * **Security Impact:** None ## **Root Cause** During a planned service deployment, a configuration mismatch occurred in the authentication settings used for internal service-to-service communication. Specifically, the authentication credentials configured in the newly deployed service did not match the credentials expected by existing services in prod environments. This mismatch caused authentication failures when services attempted to communicate with each other, resulting in workflow loading failures for customers. ## **Mitigation** The issue was resolved by reverting the configuration change to restore the previous working authentication settings. The engineering team was proactively monitoring logs immediately after deployment and identified error patterns several minutes before the first customer report. ## **Action Items/Next Steps** 1. **Enhanced Pre-Deployment Validation:** Automated validation checks will be introduced to verify that all required configurations, secrets, and service dependencies are correctly provisioned and consistent across services before any production deployment. 2. **Improved Deployment Procedures:** Updated deployment checklists with explicit verification of service authentication settings across services before rollout.

1. 2026년 6월 17일 PM 1:42 UTCinvestigating

   We are investigating an issue affecting artifact downloads from Maven Central following recent rate-limiting changes implemented by Sonatype. Sonatype has recently implemented rate limits and builds downloading artifacts from central sonatype  https://central.sonatype.org/faq/429-contact-support/. Customers downloading artifacts would be impacted.  We are actively engaging with Sonatype to mitigate the issue here.

2. 2026년 6월 17일 PM 2:03 UTCidentified

   The issue has been identified and a fix is being implemented.

3. 2026년 6월 17일 PM 2:21 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

4. 2026년 6월 17일 PM 3:21 UTCmonitoring

   Maven Central is rate-limiting requests originating from our us-central1 NAT egress IP. Other regions (us-west1, us-east5) are not affected. The Maven/Sonatype team has been engaged and is actively working on lifting the rate limit for our IP range.

5. 2026년 6월 17일 PM 4:12 UTCmonitoring

   We are continuing to monitor for any further issues.

6. 2026년 6월 17일 PM 5:13 UTCresolved

   This incident has been resolved.

7. 2026년 6월 23일 PM 11:51 UTCpostmortem

   ## Summary On June 17, 2026, a subset of Harness Hosted CI customers experienced intermittent build failures when downloading Java/Gradle dependencies from Maven Central. Affected builds failed with `429 Too Many Requests` errors and could not proceed until the issue was resolved. The incident was not caused by a Harness code change or platform defect. It was triggered by a policy change implemented by Sonatype \(the operator of Maven Central\) that tightened IP-based rate limits for unauthenticated access. Because Harness Hosted CI routes builds through shared network egress infrastructure, the aggregate outbound traffic from multiple customers sharing a common IP address exceeded the new, lower thresholds — causing all builds behind that IP to be temporarily blocked. This was an industry-wide change. Other CI/CD platforms experienced identical incidents in the weeks prior to Harness being affected. ## Root Cause **Primary cause:** Sonatype tightened IP-based rate limiting policies on Maven Central \([repo1.maven.org](http://repo1.maven.org) and [repo.maven.apache.org](http://repo.maven.apache.org)\). Harness Hosted CI uses shared network egress infrastructure where multiple customer build environments route outbound traffic through common IP addresses. When the aggregate Maven Central request volume from all customers behind a given IP exceeded Sonatype's revised threshold, that IP was hard-blocked for approximately 30 minutes, causing all builds attempting to download Maven dependencies to fail with `429` errors. ‌ ## Mitigation **Immediate actions taken during the incident:** 1. **Regional redistribution:** For customers experiencing active failures, builds were routed to infrastructure regions whose outbound IPs had not yet hit the rate limit threshold, providing immediate relief for those customers. 2. **Sonatype engagement and IP whitelisting:** The Harness team contacted Sonatype support and provided the full list of Harness Hosted CI outbound IP addresses for whitelisting. Sonatype applied a "warning" rate-limit policy in place of hard blockingeliminating the `429` errors across all affected regions. Post-mitigation validation confirmed zero throttling across all tested IPs. ## Next Steps The following actions are in progress or planned to prevent recurrence and reduce exposure to similar incidents: **Near-term \(in progress\):** * Harness is in the process of establishing a commercial agreement with Sonatype for elevated rate limits on Maven Central and advance notification of future policy changes. * Harness is engaging other major dependency registries — including Docker Hub, npm, GitHub Packages, and the Gradle Plugin Portal — to proactively whitelist Harness egress IPs before similar incidents can occur. * We are publishing documentation on best practices for managing external dependency access in Hosted CI, including guidance on dependency proxies, mirrors, and caching strategies. * Harness plans to implement caching infrastructure for common dependencies at the platform level \(using Harness Artifact Registry\), which will significantly reduce direct Maven Central request volume and provide resilience against future external rate-limit changes. * We are enhancing our alerting alerting for `429` error patterns from external registries so that future rate-limit incidents are detected internally before customers are impacted. ‌ ## What This Means for You **To reduce future exposure**, we recommend: * **Enable Cache Intelligence** in Harness CI to cache dependency artifacts between builds, reducing how often Maven Central is contacted. * **Use Harness Save/Restore Cache steps** to persist your local dependency cache across pipeline runs. If you have questions or need help implementing any of these recommendations, please contact your Customer Success Manager or reach out to Harness Support.

1. 2026년 6월 16일 AM 10:42 UTCscheduled

   We will be undergoing scheduled maintenance during this time.