Datarobot 장애 기록

1. 2026년 8월 18일 오후 12:20 UTCidentified

   Generative AI 놀이터의 Codespace 세션은 유효성 오류로 인해 실패합니다. 엔지니어링은 수정을 만들기 위해 노력하고 있습니다.

2. 2026년 8월 18일 오후 1:56 UTCidentified

   엔지니어링은 수정을 구현했으며 현재 생산 클러스터의 배포에 종사하고 있습니다.

3. 2026년 8월 18일 오후 2:44 UTCresolved

   수정은 이제 모든 Prod 환경과 Generative AI 놀이터에 배포되었습니다.

공식 인시던트 업데이트를 자동 번역했습니다.

1. 2026년 7월 24일 오후 5:32 UTCinvestigating

   DataRobot Engineering 팀은 DataRobot이 Vector Database를 호스팅하는 동안 문제를 관찰했습니다. 현재, 이것은 모든 DataRobot MTS 환경에 영향을 미치고 엔지니어링은 뿌리 원인을 조사하고 있습니다.

2. 2026년 7월 24일 오후 5:44 UTCmonitoring

   엔지니어링 팀은 수정 및 벡터 데이터베이스 생성을 적용했습니다. 팀은 상황을 감시하고 있습니다.

3. 2026년 7월 24일 오후 6:42 UTCresolved

   엔지니어링 팀은 모든 DataRobot MTS 환경에서 Vector Database 생성이 운영되는 것을 관찰했습니다. 사건은 해결됩니다.

공식 인시던트 업데이트를 자동 번역했습니다.

1. 2026년 7월 2일 오후 1:03 UTCinvestigating

   We are investigating issues in MTSaaS within EU region impacting prediction requests against Custom Models deployments.

2. 2026년 7월 2일 오후 1:20 UTCidentified

   The team root caused the issue causing HTTP 403 responses for Custom Model deployments in EU region.  Mitigations were applied and the team is verifying the fix stabilized the service.

3. 2026년 7월 2일 오후 1:37 UTCresolved

   The issue is contained and the team has verified that predictions against Custom Models in all MTSaaS environments are healthy.

1. 2026년 5월 29일 오전 8:37 UTCidentified

   Batch jobs that are currently in the queue are completed successfully; however, their completion status is not being updated correctly. As a temporary workaround, we are manually marking these jobs as completed while we work on implementing a solution.

2. 2026년 5월 29일 오전 10:22 UTCidentified

   The affected jobs have been resolved and the issue is mitigated. Engineering is actively working on a permanent fix and testing is currently underway

3. 2026년 5월 29일 오후 2:38 UTCmonitoring

   The remediation script has been deployed and Engineering is actively monitoring the situation. Batch jobs may take longer than usual to show as completed until a permanent fix is rolled out in the next production deployment.

4. 2026년 5월 29일 오후 3:20 UTCmonitoring

   We are continuing to monitor for any further issues.

5. 2026년 6월 3일 오전 9:29 UTCresolved

   A fix has been deployed to production and the issue is now resolved. All systems are operating normally.

1. 2026년 5월 28일 오전 9:09 UTCidentified

   Feature drift statistics are currently experiencing a processing delay of approximately 1 hour. No data has been lost, and all metrics will reflect accurate values once processing catches up. Our team is actively working to resolve this.

2. 2026년 5월 28일 오전 9:47 UTCresolved

   Feature drift statistics processing has been restored to normal. All metrics are now up to date. No data was lost.

1. 2026년 5월 26일 오전 5:09 UTCinvestigating

   We are experiencing a service interruption with Custom Models functionality in US SAAS environment. Predictions to existing deployments are working fine, but users cannot create new custom models. Engineering is investigating the issue and will provide updates as we make further progress.

2. 2026년 5월 26일 오전 6:01 UTCmonitoring

   Engineering has applied a fix in the US SAAS environment which resolved the issue. At the time of issue, some users might have experienced issues with Custom Apps, Data upload and custom model creation. The issue is contained.

3. 2026년 5월 26일 오전 6:04 UTCmonitoring

   We are continuing to monitor for any further issues.

4. 2026년 5월 26일 오전 6:18 UTCresolved

   Custom Models, Custom Applications & Data upload services are back to Operational state in US SAAS Environment. Issue is Resolved.

1. 2026년 5월 8일 오전 3:53 UTCmonitoring

   We are currently experiencing intermittent service issues in US Production, which are primarily affecting the launch of new workloads for Notebooks, Custom models, and Custom Applications. This issue does not impact existing workloads. This disruption is strongly correlated with an ongoing AWS Availability Zone outage (https://health.aws.amazon.com/health/status), causing resource allocation failures. The team is actively monitoring the situation and tracking updates from AWS.

2. 2026년 5월 8일 오전 6:13 UTCidentified

   We are continuing to experience issues launching new workloads for Custom Models and Custom Applications in US Production. This is connected to an ongoing AWS outage. Our team is exploring multiple mitigation options.

3. 2026년 5월 8일 오후 1:40 UTCmonitoring

   Engineering resolved the underlying issue with workload scheduling and is monitoring the cluster.

4. 2026년 5월 8일 오후 8:21 UTCresolved

   Engineering confirmed the issue is resolved and all services are restored.

1. 2026년 4월 10일 오전 10:24 UTCmonitoring

   Processing actual messages on JP MTS is delayed due to autoscaling malfunction. Engineering scaled up the deployment to alleviate the issue. Root cause mitigation in progress

2. 2026년 4월 10일 오전 11:01 UTCresolved

   Engineering has applied the required infrastructure configuration changes. The service is operating normally and no further user impact is observed. Engineering will continue monitoring cluster health to ensure stability. The incident is now marked as Contained.

1. 2026년 4월 9일 오후 9:56 UTCinvestigating

   We're experiencing an elevated level of errors and are currently looking into the issue.

2. 2026년 4월 9일 오후 10:56 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

3. 2026년 4월 10일 오전 11:31 UTCmonitoring

   Engineering has applied changes to mitigate the elevated error rates. Services are now operating normally. We are continuing to monitor the system while investigating the cause of the issue.

4. 2026년 4월 10일 오후 1:04 UTCresolved

   Engineering has implemented the required fixes to resolve the elevated error rates. Services are now operating normally, and no further user impact has been observed. The team will continue to monitor the system to ensure stability. The incident is now considered contained.

1. 2026년 3월 30일 오후 8:43 UTCidentified

   Our engineering team has found the the Quay outage currently happening is causing degraded performance across the DataRobot platform. Engineering is currently monitoring the situation.

2. 2026년 3월 30일 오후 8:44 UTCidentified

   We are continuing to work on a fix for this issue.

3. 2026년 3월 31일 오전 8:53 UTCresolved

   Quay.io functionality has been restored and DataRobot environments are fully stabilized.

1. 2026년 3월 13일 오후 5:49 UTCinvestigating

   We are experiencing performance degradation on Managed AI Cloud.

2. 2026년 3월 13일 오후 6:31 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

3. 2026년 3월 13일 오후 6:43 UTCresolved

   This incident has been resolved.

1. 2026년 3월 11일 오후 8:05 UTCinvestigating

   We are currently investigating this issue.

2. 2026년 3월 11일 오후 8:22 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

3. 2026년 3월 17일 오후 6:31 UTCmonitoring

   We are continuing to monitor for any further issues.

4. 2026년 3월 17일 오후 6:32 UTCresolved

   This incident has been resolved.

1. 2026년 3월 11일 오후 2:36 UTCinvestigating

   DataRobot is experiencing network issue related to Kubernetes in US Cluster. This will have impact on model deployment and predictions. Engineering is investigating the root cause.

2. 2026년 3월 11일 오후 3:06 UTCidentified

   Engineering has identified the root cause of the problem and a mitigation is put in place.

3. 2026년 3월 11일 오후 3:19 UTCmonitoring

   The mitigation implemented by Engineering has improved the network issue. The team is continuing to monitor the environment to ensure full recovery.

4. 2026년 3월 11일 오후 4:41 UTCresolved

   The mitigation implemented by Engineering has resolved the Kubernetes network issue, and the incident is now contained.

1. 2026년 2월 18일 오후 9:04 UTCinvestigating

   Our engineering team has found the the Quay outage currently happening is causing degraded performance across the DataRobot platform.

2. 2026년 2월 18일 오후 9:19 UTCresolved

   This incident is now resolved.

1. 2026년 2월 17일 오후 3:48 UTCidentified

   LLM blueprints deployments can not be created in JP MTS environment. Engineering is rolling back JP cluster to previous version to mitigate the issue.

2. 2026년 2월 17일 오후 3:55 UTCresolved

   Rollback of the JP cluster to the previous version is complete and the problem has been mitigated.

1. 2026년 2월 16일 오전 11:35 UTCidentified

   Agent application template is affected with the recent moderations library upgrade, fix is identified and mitigation is in progress.

2. 2026년 2월 16일 오후 12:49 UTCresolved

   New version of Agentic application template is released, the issue is resolved

1. 2026년 2월 13일 오후 8:39 UTCinvestigating

   We are observing issues on DataRobot US MTS environment. Users may experience degraded performance using APIs and data ingest services. The engineering team is currently investigating the root cause.

2. 2026년 2월 13일 오후 9:31 UTCresolved

   The incident has now been resolved. All services are now operational.

1. 2026년 1월 8일 오후 6:15 UTCmonitoring

   Some customers have reported issue connecting to DataRobot. Please do a hard refresh of your browser by clearing the cache and this should fix the problem. As always let us know if it continue to have issue connecting to DataRobot after clearing the cache.

2. 2026년 1월 8일 오후 6:16 UTCmonitoring

   We are continuing to monitor for any further issues.

3. 2026년 1월 22일 오후 2:19 UTCresolved

   This incident has been resolved.

1. 2025년 12월 17일 오후 5:41 UTCresolved

   Between 16:50 UTC and 17:09 UTC, one of external providers(DockerHub) had an outage that might have caused temporary delays in starting platform workloads. The issue has been resolved and normal operations have resumed. Engineering is continuing to monitor the system.

1. 2025년 12월 5일 오전 11:09 UTCmonitoring

   Between 09:04 UTC and 09:17 UTC, one of external providers(DockerHub) had an outage that might have caused temporary delays in starting platform workloads. The issue has been resolved and normal operations have resumed. Engineering is continuing to monitor the system.

2. 2025년 12월 5일 오전 11:35 UTCresolved

   The engineering team has confirmed recovery across all MTS and STS environments. No further updates are expected.