Loading
Applying language and content...
Loading status page
Checking service and reports...
66 Split incidents · 2026年4月 — official updates, affected components, duration and resolution details.
我们目前正在调查这一问题.
一项措施已经执行,我们正在监测结果.
这一事件已经得到解决.
自动翻译自官方事件更新。
请检查dev
自动翻译自官方事件更新。
我们正在调查Prod3集群中一个问题,其中Harness平台组件的加载速度比预期的要慢.
我们目前正在调查这一问题.
一项措施已经执行,我们正在监测结果.
这一事件已经得到解决.
□ 总结 2026年7月7日至7月8日,我们Prod 3环境中的客户账户在装载管道和溶解模板时,时有间断地缓慢和故障. 其根本原因是,从我们的模板处理服务到我们的核心平台后端服务,内部功能旗搜索流量持续急剧增加。 这个问题分两天提出。 第1天(7月7日),我们团队通过恢复受影响的服务并增加能力,通过基础设施水平的缓解恢复服务。 由于这一根本原因仍然存在,同样的故障模式在第2天\(7月8日)重现. 第2天更深的调查确定了负责具体特征旗和代码路径,而工程运出一个固定,以便现在从本地缓存中提供国旗检查。 在所有生产集群中都采用了这种固定方法,并充分解决这一问题;此后没有再出现这种固定方法。 我们正在审计平台上的所有其他特征标志,以填补同样的缓存缺口,并正在执行下文预防措施一节所描述的程序和监测改进措施,以更早地抓住这一类问题并减少再次发生的风险。 □ 影响 ** 现象:** 间歇性慢入或故障加载管道;慢出或故障模板验证和解析;平台服务间歇性连接/超时出错. ** 功能影响:** 取决于模板解析度的管道处决和管线-管线操作在上述时间线所描述的活性窗口中被拖延或失败. 一些处决需要人工重试 □ 根源 我们的核心后端服务无法跟上请求的到达率,因为在最近发布之后,由于持续、无缝地物旗的搜索,导致大量且不断增加的内部流量。 □ 缓解 * 立即缓解\\(第1天和第2天):将饱和后端实例\\(s\)从服务中取消,并增加了缓解急性瓶颈的能力。 *永久固定\(第2天\):更新了模板处理服务,因此负责的特征-旗子检查从本地缓存读取,而不是在每次查询上调用后端服务,去除其源头的过多内部流量. * 固定装置已部署到所有生产集群,并通过恢复正常内部流量和反应时间加以核实。 □预防措施和下一步 * Feature-flag craching审计:我们正在审查跨平台高频码道中使用的所有地物标志,以查明并关闭任何相类似的craching漏洞,以免导致这一问题的重复. * 释放验证:我们正在加强新特性旗码路径的释放前载荷测试,以便在达到生产前捕获到此类呼叫量回归。 * 能力警报:我们正在对后端请求排队饱和情况增加主动警报,以便在这种瓶颈影响客户反应时间之前自动被抓住并减轻.
自动翻译自官方事件更新。
我们目前正在调查欧盟1的一个问题,在这个问题上,用户正在使用国际咨询和监测中心模块.
我们已经在我们一方执行了一项解决了该问题的规定。 我们正在继续监测这一问题.
这一事件已经得到解决.
自动翻译自官方事件更新。
我们目前正在调查Prod3集群中一个管道部件加载速度比预期慢的问题.
我们的一个负责处理管道模板的微观服务机构已经落实了一项固定措施,解决了这一问题。 我们正在继续监测这一问题.
这一事件已经得到解决.
□ 总结 2026年7月7日至7月8日,我们Prod 3环境中的客户账户在装载管道和溶解模板时,时有间断地缓慢和故障. 其根本原因是,从我们的模板处理服务到我们的核心平台后端服务,内部功能旗搜索流量持续急剧增加。 这个问题分两天提出。 第1天(7月7日),我们团队通过恢复受影响的服务并增加能力,通过基础设施水平的缓解恢复服务。 由于这一根本原因仍然存在,同样的故障模式在第2天\(7月8日)重现. 第2天更深的调查确定了负责具体特征旗和代码路径,而工程运出一个固定,以便现在从本地缓存中提供国旗检查。 在所有生产集群中都采用了这种固定方法,并充分解决这一问题;此后没有再出现这种固定方法。 我们正在审计平台上的所有其他特征标志,以填补同样的缓存缺口,并正在执行下文预防措施一节所描述的程序和监测改进措施,以更早地抓住这一类问题并减少再次发生的风险。 □ 影响 ** 现象:** 间歇性慢入或故障加载管道;慢出或故障模板验证和解析;平台服务间歇性连接/超时出错. ** 功能影响:** 取决于模板解析度的管道处决和管线-管线操作在上述时间线所描述的活性窗口中被拖延或失败. 一些处决需要人工重试 □ 根源 我们的核心后端服务无法跟上请求的到达率,因为在最近发布之后,由于持续、无缝地物旗的搜索,导致大量且不断增加的内部流量。 □ 缓解 * 立即缓解\\(第1天和第2天):将饱和后端实例\\(s\)从服务中取消,并增加了缓解急性瓶颈的能力。 *永久固定\(第2天\):更新了模板处理服务,因此负责的特征-旗子检查从本地缓存读取,而不是在每次查询上调用后端服务,去除其源头的过多内部流量. * 固定装置已部署到所有生产集群,并通过恢复正常内部流量和反应时间加以核实。 □预防措施和下一步 * Feature-flag craching审计:我们正在审查跨平台高频码道中使用的所有地物标志,以查明并关闭任何相类似的craching漏洞,以免导致这一问题的重复. * 释放验证:我们正在加强新特性旗码路径的释放前载荷测试,以便在达到生产前捕获到此类呼叫量回归。 * 能力警报:我们正在对后端请求排队饱和情况增加主动警报,以便在这种瓶颈影响客户反应时间之前自动被抓住并减轻.
自动翻译自官方事件更新。
A fix has been implemented and we are monitoring the results.
We are seeing the backlogged items worked through and continuing to monitor results.
We are continuing to monitor results while the backlog is worked through.
This incident has been resolved.
**Incident Summary** Harness FME \(Feature Management & Experimentation\) faced substantial delays in scheduled impressions data exports on Jul 2, 2026. This issue stemmed from performance degradation within Tinybird's infrastructure, affecting query execution crucial to our export pipeline. **Root Cause** The core issue was a temporary **performance degradation on the provider's infrastructure**. This directly impacted the metadata retrieval endpoint, which failed to return job IDs intermittently. Consequently, completed export jobs underwent repeated retries, leading to a backlog in the export queue. **Impact** * The export backlog accumulated, peaking at more than 6 hours behind the planned schedule. * No data loss was detected during the incident. **Mitigation** 1. **Adjusted Temporal workflows:** Extended timeouts from ~30 to ~50 minutes and increased polling attempts from ~180 to ~300. 2. **Expanded resource capacity.** 3. **Traffic redirection:** Systematically shifted export workloads over to an optimized in-house cluster. 4. **Manual intervention:** Terminated stalling workflows stuck in retry loops from failed metadata lookups. **Next Steps** Complete the migration of all remaining export jobs to the internal analytics cluster \(In progress\).
Testing Maintenance
Scheduled maintenance is currently in progress. We will provide updates as necessary.
The scheduled maintenance has been completed.
Dev Statuspage incident - Testing
We are currently investigating this issue.
Harness is investigating an issue affecting our Prod 4 environments. Our log-services are affected and harness is currently working on restoring function. This can affect our CI, Chaos, IACM modules.
We are continuing to work on a fix for this issue.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
### **Summary** During a planned security improvement to rotate service authentication secrets in the Prod 4 environment, an incorrect secret value was inadvertently applied to a subset of internal services. This resulted in authentication failures between platform components, impacting CI initialization, delegate task creation, and log streaming for certain pipeline executions. The issue was resolved by rolling back the token configuration to the previous known-good version and validating service recovery. ### **Impact** **Affected environment:** Prod 4 **Customer-visible impact:** * CI builds failed during initialization with errors such as: * _“Could not fetch token from log service”_ * _“token in request not authorized for receiving tokens”_ \(HTTP 400\) * CD and IaCM pipeline stages continued to execute but did not stream logs. * Some Shell Script steps failed during execution. * Delegate tasks could not be created or assigned for some pipeline executions. No customer data was lost or corrupted. ### **Root Cause** As part of a planned security initiative, authentication secrets used for communication between internal platform services were rotated in the Prod 4 environment. During the rotation process, an incorrect secret value was inadvertently configured for one of the services. Instead of applying the newly generated secret in the expected format, a default/incorrect value was introduced. This created a mismatch between services that authenticate using the shared token, causing authentication requests to be rejected with HTTP 400 errors. The authentication failures prevented CI components from obtaining log service tokens, disrupted delegate task creation, and prevented log streaming for affected pipeline executions. ### **Resolution** Engineering responded by: * Reverting the token configuration to the previous known-good secret. * Restoring consistent authentication across affected services. * Validating recovery through CI pipeline execution tests. * Verifying CD and IaCM pipeline execution and log streaming. * Executing a comprehensive validation suite to confirm platform functionality before closing the incident. ### **Follow-up Actions** To reduce the likelihood of similar incidents in the future, Harness will: * Implement additional pre-rotation and post-rotation validation checks to verify that the correct secret values and formats are applied before changes are activated. * Introduce automated verification of authentication between dependent services immediately following secret rotations. * Strengthen operational safeguards and deployment validation for security credential rotation procedures to detect configuration mismatches before they can impact customer workloads.
Testing CD outage
We are continuing to investigate this issue.
This incident has been resolved.
We are currently experiencing intermittent login issues with the Traceable cluster due to an issue with our login provider. We are actively working with the provider to resolve the issue. This incident does not impact data ingestion, and all ingestion pipelines continue to function normally. We will provide updates as we have more information.
The issue has been identified and a fix is being implemented.
Issue has been fixed and we are closely monitoring
We are continuing to monitor for any further issues.
This incident has been resolved.
## **Summary** _Between **06:50 UTC and 07:20 UTC on June 26, 2026**, customers experienced intermittent login failures while accessing Traceable environments across multiple US clusters. The incident was traced to an issue with the external authentication provider \(Auth0\), where elevated socket timeouts caused increased login latency and authentication request failures. Login functionality gradually recovered as the upstream issue stabilized, and the incident was resolved after successful login validation across multiple impacted clusters._ ## **Root Cause** _The root cause was an issue with the external authentication provider \(Auth0\), which experienced elevated socket timeouts while processing authentication requests. These upstream timeouts increased login latency and caused intermittent authentication failures across multiple US-region clusters. Since authentication requests depended on the external provider, affected login attempts failed despite Traceable platform services remaining healthy. The incident was resolved once the upstream authentication service recovered and login requests consistently completed successfully._ ## **Impact** _Starting at approximately 06:50 UTC, customers experienced intermittent login failures when accessing Traceable environments across multiple US clusters. The issue affected user authentication, preventing some users from accessing the platform while underlying application services remained operational. Login functionality progressively recovered during the incident, and normal authentication was restored by 07:20 UTC._ ## **Remediation** _The engineering team worked with the external authentication provider while continuously monitoring authentication health across affected clusters. Login functionality was validated through platform metrics and manual verification across representative environments. After confirming consistent authentication success across impacted clusters, the incident was declared resolved._ ## **Action Items** _To prevent such issues going forward, Harness will,_ _Increase authentication resilience: Evaluate improvements to authentication request handling, including timeout tuning, retry strategies where appropriate, and graceful degradation for transient upstream failures._
We’ll be performing a backend infrastructure upgrade to improve reliability and compliance. During this maintenance window, Harness platform may experience brief, intermittent interruptions. We appreciate your understanding and are working to minimize any impact.
Maintenance will begin as scheduled in 60 minutes.
Harness login is failing. We are investigating the problem.
A fix has been implemented and we are monitoring the results.
We are continuing to monitor for any further issues.
This incident has been resolved.
### **Summary** During a planned security improvement to rotate service authentication secrets in the EU1 environment, an out of sequence secret value was inadvertently applied to a subset of internal services. This resulted in authentication failures between platform components, causing login failures. The issue was resolved by rolling back the token configuration to the previous known-good version and validating service recovery. ### **Impact** **Affected environment:** EU1 **Customer-visible impact: Login Failures** No customer data was lost or corrupted. ### **Root Cause** As part of a planned security initiative, authentication secrets used for communication between internal platform services were rotated in the EU1 . During the rotation process, an secret value was inadvertently misconfigured for one of the services. There was a newer format of secret and had to be applied in a certain order. This created a mismatch between services that authenticate using the shared token, causing authentication requests to be rejected with HTTP 400 errors. The authentication failures prevented CI components from obtaining log service tokens, disrupted delegate task creation, and prevented log streaming for affected pipeline executions. ### **Resolution** Engineering responded by: * Reverting the token configuration to the previous known-good secret. * Restoring consistent authentication across affected services. ### **Follow-up Actions** To reduce the likelihood of similar incidents in the future, Harness will: * Implement additional pre-rotation and post-rotation validation checks to verify that the correct secret values and formats are applied before changes are activated. * Introduce automated verification of authentication between dependent services immediately following secret rotations. * Strengthen operational safeguards and deployment validation for security credential rotation procedures to detect configuration mismatches before they can impact customer workloads.
We have identified an issue affecting a small subset of accounts in the EU1 region. As a result, data ingestion for impacted integrations may have stopped.
FME will be undergoing scheduled critical maintenance during this window. No downtime is expected, and all services are anticipated to remain available. As part of this maintenance, updates or modifications to Large Segments will be temporarily unavailable until the activity is completed.
We will be undergoing scheduled maintenance during this time.
Maintenance will begin as scheduled in 60 minutes.
Scheduled maintenance is currently in progress. We will provide updates as necessary.
The scheduled maintenance has been completed.
We’ll be performing a backend infrastructure upgrade to improve reliability and compliance. During this maintenance window, Harness platform may experience brief, intermittent interruptions. We appreciate your understanding and are working to minimize any impact.
Maintenance will begin as scheduled in 60 minutes.
Scheduled maintenance is currently in progress. We will provide updates as necessary.
The scheduled maintenance has been completed.
Testing Dev in QA
We are currently investigating this issue.
The issue has been identified and a fix is being implemented.
IDP Systems' communication was interrupted due to an update. Harness is rolling back changes. Impacted environments - prod0, prod1, prod2
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
# **Customer RCA: Executive Summary** On June 17, 2026, between 9:30 PM and 10:15 PM IST, customers in certain production environments experienced failures when accessing IDP \(Internal Developer Portal\) workflows. The issue was caused by a configuration mismatch in service authentication settings. ## **Impact** * **Affected Service:** IDP workflows * **Duration:** Approximately 45 minutes \(9:30 PM - 10:15 PM IST\) * **Customer Impact:** Users were unable to load IDP workflows during this period. * **Data Loss:** None * **Security Impact:** None ## **Root Cause** During a planned service deployment, a configuration mismatch occurred in the authentication settings used for internal service-to-service communication. Specifically, the authentication credentials configured in the newly deployed service did not match the credentials expected by existing services in prod environments. This mismatch caused authentication failures when services attempted to communicate with each other, resulting in workflow loading failures for customers. ## **Mitigation** The issue was resolved by reverting the configuration change to restore the previous working authentication settings. The engineering team was proactively monitoring logs immediately after deployment and identified error patterns several minutes before the first customer report. ## **Action Items/Next Steps** 1. **Enhanced Pre-Deployment Validation:** Automated validation checks will be introduced to verify that all required configurations, secrets, and service dependencies are correctly provisioned and consistent across services before any production deployment. 2. **Improved Deployment Procedures:** Updated deployment checklists with explicit verification of service authentication settings across services before rollout.
We are investigating an issue affecting artifact downloads from Maven Central following recent rate-limiting changes implemented by Sonatype. Sonatype has recently implemented rate limits and builds downloading artifacts from central sonatype https://central.sonatype.org/faq/429-contact-support/. Customers downloading artifacts would be impacted. We are actively engaging with Sonatype to mitigate the issue here.
The issue has been identified and a fix is being implemented.
A fix has been implemented and we are monitoring the results.
Maven Central is rate-limiting requests originating from our us-central1 NAT egress IP. Other regions (us-west1, us-east5) are not affected. The Maven/Sonatype team has been engaged and is actively working on lifting the rate limit for our IP range.
We are continuing to monitor for any further issues.
This incident has been resolved.
## Summary On June 17, 2026, a subset of Harness Hosted CI customers experienced intermittent build failures when downloading Java/Gradle dependencies from Maven Central. Affected builds failed with `429 Too Many Requests` errors and could not proceed until the issue was resolved. The incident was not caused by a Harness code change or platform defect. It was triggered by a policy change implemented by Sonatype \(the operator of Maven Central\) that tightened IP-based rate limits for unauthenticated access. Because Harness Hosted CI routes builds through shared network egress infrastructure, the aggregate outbound traffic from multiple customers sharing a common IP address exceeded the new, lower thresholds — causing all builds behind that IP to be temporarily blocked. This was an industry-wide change. Other CI/CD platforms experienced identical incidents in the weeks prior to Harness being affected. ## Root Cause **Primary cause:** Sonatype tightened IP-based rate limiting policies on Maven Central \([repo1.maven.org](http://repo1.maven.org) and [repo.maven.apache.org](http://repo.maven.apache.org)\). Harness Hosted CI uses shared network egress infrastructure where multiple customer build environments route outbound traffic through common IP addresses. When the aggregate Maven Central request volume from all customers behind a given IP exceeded Sonatype's revised threshold, that IP was hard-blocked for approximately 30 minutes, causing all builds attempting to download Maven dependencies to fail with `429` errors. ## Mitigation **Immediate actions taken during the incident:** 1. **Regional redistribution:** For customers experiencing active failures, builds were routed to infrastructure regions whose outbound IPs had not yet hit the rate limit threshold, providing immediate relief for those customers. 2. **Sonatype engagement and IP whitelisting:** The Harness team contacted Sonatype support and provided the full list of Harness Hosted CI outbound IP addresses for whitelisting. Sonatype applied a "warning" rate-limit policy in place of hard blockingeliminating the `429` errors across all affected regions. Post-mitigation validation confirmed zero throttling across all tested IPs. ## Next Steps The following actions are in progress or planned to prevent recurrence and reduce exposure to similar incidents: **Near-term \(in progress\):** * Harness is in the process of establishing a commercial agreement with Sonatype for elevated rate limits on Maven Central and advance notification of future policy changes. * Harness is engaging other major dependency registries — including Docker Hub, npm, GitHub Packages, and the Gradle Plugin Portal — to proactively whitelist Harness egress IPs before similar incidents can occur. * We are publishing documentation on best practices for managing external dependency access in Hosted CI, including guidance on dependency proxies, mirrors, and caching strategies. * Harness plans to implement caching infrastructure for common dependencies at the platform level \(using Harness Artifact Registry\), which will significantly reduce direct Maven Central request volume and provide resilience against future external rate-limit changes. * We are enhancing our alerting alerting for `429` error patterns from external registries so that future rate-limit incidents are detected internally before customers are impacted. ## What This Means for You **To reduce future exposure**, we recommend: * **Enable Cache Intelligence** in Harness CI to cache dependency artifacts between builds, reducing how often Maven Central is contacted. * **Use Harness Save/Restore Cache steps** to persist your local dependency cache across pipeline runs. If you have questions or need help implementing any of these recommendations, please contact your Customer Success Manager or reach out to Harness Support.
We will be undergoing scheduled maintenance during this time.