Copado 장애 기록

1. 2026년 8월 27일 AM 8:46 UTCinvestigating

   우리는 현재 예상치 못한 작업 실행에 관한 고객의 작은 수에서 보고서를 조사하고 있습니다. 우리의 팀은 적극적으로 행동을 검토하고 더 나은 문제를 이해하기 위해 노력하고 있습니다. 우리는 현재 투자하는 동안, 당신은 어떤 관련 문제든지 경험하고 있는 경우에 우리의 성공 공동체를 통해 케이스를 열게 자유롭게 느낍니다.

2. 2026년 8월 27일 PM 12:25 UTCinvestigating

   우리는 여전히이 문제를 조사하고 해결을 위해 노력하고 있습니다. 이 상태를 더 많은 정보로 업데이트합니다.

3. 2026년 8월 28일 AM 6:12 UTCmonitoring

   우리는 예상치 못한 작업 실행의 더 이상 수신 보고서가 없습니다. 우리는 뿌리 원인을 결정하기 위해 Salesforce와 일하고 어떤 recurrence든지를 위해 계속할 것입니다.

4. 2026년 8월 28일 AM 7:21 UTCresolved

   우리는이 문제의 뿌리 원인은 알려진 Salesforce 제품 버그입니다. Salesforce는 현재 초기 릴리스로 사용할 수 있도록 계획하고 있습니다. 세부사항은 여기에서 찾아낼 수 있습니다: https://help.salesforce.com/s/issue?id=a02g7000007hHG5AAM 우리는 해결로이 사건을 표시하고 있습니다.

공식 인시던트 업데이트를 자동 번역했습니다.

1. 2026년 8월 21일 PM 6:24 UTCinvestigating

   우리는 현재 Salesforce 서비스 degradation에 영향을 미치는 org 인증 및 Copado 배포를 추적하고 있습니다. 우리는 상황을 밀접하게 감시하고 더 많은 업데이트를 제공 할 것입니다. status.salesforce.com 을 클릭하여 자세한 정보를 확인하세요.

2. 2026년 8월 21일 PM 9:57 UTCmonitoring

   Salesforce degradation invalidated active Refresh Tokens, Copado 작업이 연결 된 orgs에 액세스 잃는 원인이됩니다. 기능을 복원하려면 다음 단계를 수행하십시오. 1. 새로운, 유효한 갱신 토큰을 생성하기 위하여 Copado 사용자 자격 증명을 재조정하십시오. 2. 실패한 Copado 일을 재기하십시오. 재조정 후 문제가 발생하면 지원팀에 연락해 주십시오.

3. 2026년 8월 22일 AM 2:16 UTCresolved

   우리는 마지막 시간에 추가 발생이 없다는 것을 결정했습니다. 이 사건은 해결되었습니다.

공식 인시던트 업데이트를 자동 번역했습니다.

1. 2026년 7월 21일 AM 9:58 UTCinvestigating

   Git 인증 오류가 발생하거나 구성품을 홍보할 때, Git 인증 오류를 경험하고 있는 고객의 서브셋에 영향을 미치는 문제점을 조사하고 있습니다. 우리의 엔지니어링 팀은 뿌리 원인을 확인하고 가능한 한 빨리 정상적인 기능을 복원하기 위해 적극적으로 노력하고 있습니다. 더 많은 정보로 업데이트를 계속 제공 할 것입니다.

2. 2026년 7월 21일 AM 10:46 UTCidentified

   고객에 영향을 미치는 Git 인증 오류가 지속적인 GitHub 서비스 사건과 관련되었는지 확인했습니다. https://www.githubstatus.com/ 우리는 밀접하게 감시하고 이 상태를 더 많은 정보가 유효할 때 새롭게 할 것입니다.

3. 2026년 7월 21일 PM 12:09 UTCmonitoring

   GitHub의 상태는 사건이 해결되는 것을 나타냅니다. 우리는 이제 모니터링하고 우리가 전체 기능을 복원 확인 한 번 해결이 사건을 업데이트합니다.

4. 2026년 7월 21일 PM 1:11 UTCresolved

   우리는 마지막 시간에 추가 발생이 없다는 것을 결정했습니다. 이 사건은 해결되었습니다.

공식 인시던트 업데이트를 자동 번역했습니다.

1. 2026년 7월 15일 PM 9:25 UTCinvestigating

   우리는 CICD 파이프라인과 관련한 문제점을 경험하는 소량의 고객에게서 보고를 받았습니다. 이 보고서의 패턴으로 인해 우리는 현재 잠재적 인 서비스 평가로 이것을 조사하고 있습니다. 특정 Orgs 및 CICD 운영은 영향을받지 않을 수 있지만, 문제가 발생하면 성공 커뮤니티의 사례를 무료로 열 수 있습니다.

2. 2026년 7월 15일 PM 11:34 UTCresolved

   해결되었습니다.

공식 인시던트 업데이트를 자동 번역했습니다.

1. 2026년 7월 15일 PM 8:54 UTCinvestigating

   Metadata Format Pipeline 관련 프로모션에 영향을 미치는 문제의 조사 보고서입니다. 우리의 엔지니어링 팀은 적극적으로 투자하고 더 많은 정보를 제공 할 것입니다. 다음 업데이트는 시간 내에 제공됩니다.

공식 인시던트 업데이트를 자동 번역했습니다.

1. 2026년 5월 11일 PM 3:23 UTCinvestigating

   We are currently investigating reports of performance degradation impacting CI/CD pipelines. During this time, some users may also experience intermittent delays or unexpected validation errors when utilizing the CICD Pipeline, which may also affect the Data Deploy Service. Our engineering teams are actively looking into the situation to identify the root cause. We will provide an update as soon as more information is available.

2. 2026년 5월 11일 PM 3:44 UTCinvestigating

   The majority of the instability being experienced is due to the "frequent rotation of Refresh Tokens" now being enforced by Salesforce, which we are in the process of further mitigating, at scale.

3. 2026년 5월 11일 PM 7:05 UTCmonitoring

   Status update: Monitoring We have observed a complete drop in authentication errors over the last hour and currently consider this issue resolved. Our team will continue to monitor the system to ensure stability. Action required: If you experienced an error, please re-authenticate your credentials and retry your jobs. Please note that the main Credential for the Org where Copado is installed must be validated first; after that is complete, all subsequent Credential records must be validated by their respective record owners in Salesforce.

4. 2026년 5월 11일 PM 8:21 UTCresolved

   We have implemented the necessary backend mitigations to resolve the recent service disruptions, and system stability has been fully restored. Action required: If you experienced an error, please re-authenticate your credentials and retry your jobs. Please note that the main Credential for the Org where Copado is installed must be validated first; after that is complete, all subsequent Credential records must be validated by their respective record owners in Salesforce.

1. 2026년 5월 5일 PM 2:58 UTCinvestigating

   We are currently investigating an issue affecting Copado AI Services, where users may be experiencing consistent "Agentic LLM task failed" errors. Our engineering team has identified that this is stemming from an overloaded state with one of our upstream AI providers (Anthropic) in our primary hosting region, North America. We are treating this with the highest priority and will provide another update as soon as we have more information or a resolution.

2. 2026년 5월 5일 PM 3:59 UTCinvestigating

   We are continuing to investigate this issue.

3. 2026년 5월 5일 PM 4:00 UTCmonitoring

   We have successfully resolved the issue affecting Copado AI Services for our North American servers. Normal service has been fully restored, and users should no longer experience errors when utilizing AI features. We will continue to monitor the service closely to ensure optimal performance. We apologize for any inconvenience this caused and appreciate your patience while our team worked to restore service.

4. 2026년 5월 5일 PM 8:07 UTCresolved

   Update - Resolved Our monitoring period has concluded, and we have confirmed that Copado AI Services are fully operational and performing as expected for all North American servers. This incident is now resolved. Thank you for your patience.

1. 2025년 10월 20일 PM 3:45 UTCmonitoring

   We are currently experiencing a partial outage due to an ongoing issue with Amazon Web Services (AWS), primarily affecting the US-EAST-1 region. This issue has also impacted certain Salesforce-hosted environments, including our US CRT/AI infrastructure. While AWS initially began showing signs of recovery around 5:10 AM PDT, subsequent challenges in their mitigation efforts have resulted in renewed instability. As of now, both AWS and Salesforce continue to report this as an active incident. Due to the dependencies between our systems and these third-party services, a growing number of customers are experiencing service disruptions or inaccessibility across multiple products — including CRT, AI, CI/CD, and Essentials. Our engineering team is actively engaged and continuing to monitor the situation closely. We are working with our partners at AWS and Salesforce to restore full service as quickly as possible. Next update: As soon as new information becomes available.

2. 2025년 10월 20일 PM 4:40 UTCmonitoring

   We are currently seeing a broader impact across US services. Live Testing in CRT US is likely heavily impacted, and we are also receiving reports of document upload failures in AI for US-based users. Our team is actively monitoring and working with AWS and Salesforce to mitigate the issue. Next update: As soon as new information becomes available.

3. 2025년 10월 20일 PM 9:10 UTCmonitoring

   We have seen improvements in the availability and performances of our services, as well as reduced errors connecting to Salesforce. This has been further confirmed by updates from AWS and Salesforce, respectively. We are continuing to monitor the situation and will post further updates, but we believe things are trending positively. Next update: As soon as new information becomes available.

4. 2025년 10월 20일 PM 11:32 UTCidentified

   We have continued to monitor this situation and mostly positive improvements across our products and connectivity with Salesforce. At approximately 23:00 UTC, AWS marked their ongoing incident as resolved. Salesforce continues to consider this incident ongoing, however we are updating our status to reflect the resolution on the AWS side, and now consider this incident to only be degraded performance rather than a partial outage. Customers may still experience issues with any operations that connect to Salesforce.

5. 2025년 10월 21일 AM 1:06 UTCresolved

   Salesforce has updated their status at 00:30 UTC and marked this incident as resolved. We now also will mark this incident as resolved.

1. 2025년 8월 18일 PM 4:32 UTCidentified

   An issue related to projects in Copado Robotic Testing has been identified affecting multiple customers in different regions. Our team is investigating the cause of the error and will provide updates as quickly as possible. We apologize for the inconvenience.

2. 2025년 8월 18일 PM 7:03 UTCresolved

   All regions are operational again. This incident has been resolved.

1. 2025년 6월 12일 PM 6:53 UTCinvestigating

   We have detected a global service disruption as of 13:28 CST, which is impacting, but is not limited to, both Google Cloud and Amazon Web Services worldwide. This disruption is impacting Copado, Salesforce, and many other products and services. During this time you may experience issues such as inability to login, authenticate, and interact with Copado/Salesforce. We will continue to update our status as more information becomes available.

2. 2025년 6월 12일 PM 7:24 UTCinvestigating

   We are seeing a partial recovery in internal Copado Job executions. All the systems have been updated to Degraded Performance.

3. 2025년 6월 12일 PM 8:08 UTCmonitoring

   We have observed a full recovery across Copado Robotic Testing, Copado Explorer, Essentials, and Copado AI services. We are still seeing partial degradation affecting CICD jobs. Our team is actively monitoring the situation to ensure full restoration of all services.

4. 2025년 6월 12일 PM 8:55 UTCresolved

   Google Cloud's mitigation efforts have led to all services showing significant recovery. We will continue to monitor to ensure we don't see signs of the disruption recurring, but at this time, all known impact has subsided. All services are operational.

1. 2025년 3월 3일 PM 10:35 UTCidentified

   It appears that Salesforce has introduced a change to the Partner WSDL in patch 9.9 which is unexpectedly returning JSON for the certain objects in the describe call. This is affecting customers on this version in their ability to get metadata, and to create snapshots. We have temporarily implemented a method to ignore this object in the Salesforce Partner WSDL and have informed Salesforce of the issue. Our engineering and infrastructure teams are engaged and we will share updates here as our investigation continues.

2. 2025년 3월 4일 PM 1:34 UTCidentified

   Our engineering team has reviewed this issue further, and confirmed that customers on Salesforce patch 9.10 are also affected, due to another object not included in the library. They have prepared a hotfix which addresses all cases. This hotfix is being tested now and will be released shortly.

3. 2025년 3월 4일 PM 3:06 UTCmonitoring

   The hotfix has been released. We will monitor the impact for a short period and update this incident to Resolved.

4. 2025년 3월 4일 PM 4:40 UTCresolved

   After the hotfix, our monitoring shows no new instances of this error occurring- this incident is being marked Resolved.

1. 2024년 11월 27일 PM 11:06 UTCresolved

   On 11/18/2024, at 4:00 AM UTC, customers on our APAC backend experienced interruptions resulting job delays and errors during their use of Copado. Our engineering team investigated and identified issues related to the latest backend release. The team implemented the necessary changes on 11/18/2024 at 8:20 AM UTC, and service resumed to normal. We apologize for any inconvenience.

1. 2024년 7월 18일 AM 10:23 UTCinvestigating

   The Copado Success community is currently inaccessible for some users, or loading but with some functionality unavailable. We are currently engaged with Salesforce support to resolve the issue.

2. 2024년 7월 18일 PM 3:01 UTCresolved

   This incident has been resolved.

1. 2024년 7월 18일 AM 1:50 UTCinvestigating

   The Copado Success community is currently inaccessible for some users, or loading but with some functionality unavailable. We are currently engaged with Salesforce support to resolve the issue.

2. 2024년 7월 18일 AM 2:38 UTCresolved

   This issue is now resolved.

1. 2024년 6월 28일 PM 6:49 UTCinvestigating

   We are investigating an issue reported by multiple customers related to GitHub pull requests. At this point, we believe this issue is caused by GitHub's ongoing incident, as indicated on https://www.githubstatus.com/. No changes have been made to Copado's backend or infrastructure that coincide with this issue. We will update as more information becomes available

2. 2024년 6월 29일 AM 1:44 UTCinvestigating

   While GitHub is reporting their incident as resolved, we still have customers experiencing issues. We will continue to investigate.

3. 2024년 7월 1일 PM 2:19 UTCresolved

   Our team has identified the issue, and implemented the necessary changes.

1. 2024년 4월 24일 PM 1:13 UTCidentified

   Copado has identified that Azure DevOps Saas began experiencing service degradation and availability issues starting at approximately 2:30 PM Central Time on April 23, 2024, that could result in failed authentication to Azure DevOps repos via HTTPS. We are currently working with our engineering and infrastructure teams to help identify impacted customers, and are monitoring the ADO status via their site, https://status.dev.azure.com. We have confirmed that this problem does not impact customers utilizing SSH to connect to their repositories. We currently recommend that if possible, customers update their Git Repository record configuration from HTTPS to SSH, to mitigate this outage and return to the full use of Copado and their pipelines with ADO Saas repositories. If you need assistance with this change, please reach out to Copado Support via https://success.copado.com. We will provide updates as more information becomes available.

2. 2024년 4월 24일 PM 2:38 UTCidentified

   We have updated this incident to indicate broader impact; we are seeing examples of customers using Azure DevOps that are experiencing issues beyond authentication. ADO has also broadened the impact of their ongoing incident: https://status.dev.azure.com/_event/496920595 We recommend contacting ADO directly if you are utilizing an ADO repository and are experiencing issues, including but not limited to authentication, performance, and availability.

3. 2024년 4월 24일 PM 6:11 UTCidentified

   We are actively engaged with ADO support and they have provided an estimate of 4 hours to complete the work required to resolve this issue.

4. 2024년 4월 25일 AM 2:53 UTCresolved

   This incident has been resolved.

1. 2024년 1월 23일 PM 5:55 UTCidentified

   We have identified multiple jobs, across various customers, which are being impacted by delays in metadata retrieval from Salesforce. While this apparent performance degradation is currently not listed on status.salesforce.com, we have engaged Salesforce directly for further clarity. In our initial testing, it appears the impact is limited to Hyperforce instances. In the majority of cases, this issue simply impacts the duration of the associated job. In limited examples we have seen jobs fail due to timeouts as a result of this delayed response. We will update this incident as more information is available.

2. 2024년 1월 23일 PM 9:11 UTCidentified

   We are still awaiting a response from the Salesforce team; we continue to see job performance impacted by delays in metadata retrieval.

3. 2024년 1월 24일 AM 1:47 UTCresolved

   We are no longer seeing impacted jobs. This issue appears to have been related to Incident ID 12223 (https://status.salesforce.com/incidents/12223). We are working with Salesforce to confirm.

1. 2023년 11월 2일 PM 3:06 UTCidentified

   We are tracking two separate Salesforce incidents that are impacting our customers’ use of Copado: Record relationship Feature Degradation (Global Issue): https://status.salesforce.com/generalmessages/1220 Instance Outage (Regional Issue) https://status.salesforce.com/incidents/11897 While there is no impact to the Copado Infrastructure itself, we have seen examples where the Salesforce records associated with Copado jobs are not successfully created. We recommend delaying any Copado actions until Salesforce has confirmed these incidents are resolved. We also recommend that our customers review any jobs that were already performed during the incidents to confirm the records were created successfully. Additionally, these incidents are impacting our own Salesforce instance. Support for any cases submitted while the incidents are ongoing may be delayed, and there is a possibility that any cases created during the incidents, or during the mitigation actions that are occurring now, may not have been successfully transmitted. Please follow up to confirm successful transmission of support cases after the Salesforce issues are resolved.

2. 2023년 11월 2일 PM 3:29 UTCidentified

   Salesforce has indicated that one of the two incidents (below) is resolved; however, we still recommend delaying any activities within Copado until the other incident is resolved. Instance Outage (Regional Issue) - Now Resolved https://status.salesforce.com/incidents/11897

3. 2023년 11월 2일 PM 6:44 UTCidentified

   Salesforce is performing a rollback of the latest release to resolve the remaining incident. Initial testing suggests that this is resolving previously reported issues. We will continue to monitor and update this status to resolved once Salesforce indicates the rollback is complete.

4. 2023년 11월 3일 PM 3:47 UTCresolved

   Salesforce has indicated that the second incident impacting the use of Copado is now resolved. We are marking this as resolved, but will continue to monitor for impact from both of the reported incidents.

1. 2023년 10월 16일 PM 9:39 UTCinvestigating

   When committing to version control on any job in Copado Essentials Plus via Deployment, Work Item or CI job will result in error "Error: An error occurred while processing your request. Please contact our customer support if the error persists.". We are aware of the issue and working diligently to resolve the matter as quickly as possible. Work Around: no current work arounds are available.

2. 2023년 10월 17일 AM 12:47 UTCidentified

   Copado Essentials Engineering and Copado Infrastructure Engineering have tentatively identified the issue and have run a number of successful Git deployments. We will provide another update shortly with more details.

3. 2023년 10월 17일 AM 12:52 UTCmonitoring

   We have implemented a fix and the issue with deploying to Git repos has been tentatively resolved. We will continue to update you here. Our Apologies for this inconvenience.

4. 2023년 10월 17일 AM 2:00 UTCresolved

   This incident has been resolved.

1. 2023년 6월 21일 PM 4:48 UTCidentified

   We are aware some clients are experiencing the following deployment error while deploying Apex Classes: ‘INVALID_OPERATION: runTests must not be empty when a testLevel of RunSpecifiedTests is used’ Our team has investigated and identified the issue. We are working to implement the necessary changes now; we will update the status of this incident once the issue is resolved. We apologize for any inconvenience.

2. 2023년 6월 21일 PM 5:09 UTCresolved

   This issue is now resolved. If you experienced this issue, be sure to create a NEW promotion record from the user story where you experienced the issue.