Datarobot 中断历史

1. 2026年8月18日 UTC 12:20identified

   Generative AI Playground中的代码空间会话由于验证错误而失败. 工程正致力于建立固定装置.

2. 2026年8月18日 UTC 13:56identified

   工程已实施一个固定装置,目前正在努力将其部署在生产集群.

3. 2026年8月18日 UTC 14:44resolved

   修复装置现在部署在普罗德的所有环境中,Generative AI Playground现在正在按预期工作.

自动翻译自官方事件更新。

1. 2026年7月24日 UTC 17:32investigating

   DataRobot工程组在创建DataRobot托管矢量数据库的同时观察到了问题. 目前,这正在影响所有DataRobot MTS环境,工程正在调查其根源.

2. 2026年7月24日 UTC 17:44monitoring

   工程组已应用了固定器,矢量数据库的创建现已投入使用. 该小组正在监测局势.

3. 2026年7月24日 UTC 18:42resolved

   工程小组观察到,在DataRobot MTS的所有环境中,病媒数据库的创建都能够运行。 此事已决.

自动翻译自官方事件更新。

1. 2026年7月2日 UTC 13:03investigating

   We are investigating issues in MTSaaS within EU region impacting prediction requests against Custom Models deployments.

2. 2026年7月2日 UTC 13:20identified

   The team root caused the issue causing HTTP 403 responses for Custom Model deployments in EU region.  Mitigations were applied and the team is verifying the fix stabilized the service.

3. 2026年7月2日 UTC 13:37resolved

   The issue is contained and the team has verified that predictions against Custom Models in all MTSaaS environments are healthy.

1. 2026年5月29日 UTC 08:37identified

   Batch jobs that are currently in the queue are completed successfully; however, their completion status is not being updated correctly. As a temporary workaround, we are manually marking these jobs as completed while we work on implementing a solution.

2. 2026年5月29日 UTC 10:22identified

   The affected jobs have been resolved and the issue is mitigated. Engineering is actively working on a permanent fix and testing is currently underway

3. 2026年5月29日 UTC 14:38monitoring

   The remediation script has been deployed and Engineering is actively monitoring the situation. Batch jobs may take longer than usual to show as completed until a permanent fix is rolled out in the next production deployment.

4. 2026年5月29日 UTC 15:20monitoring

   We are continuing to monitor for any further issues.

5. 2026年6月3日 UTC 09:29resolved

   A fix has been deployed to production and the issue is now resolved. All systems are operating normally.

1. 2026年5月28日 UTC 09:09identified

   Feature drift statistics are currently experiencing a processing delay of approximately 1 hour. No data has been lost, and all metrics will reflect accurate values once processing catches up. Our team is actively working to resolve this.

2. 2026年5月28日 UTC 09:47resolved

   Feature drift statistics processing has been restored to normal. All metrics are now up to date. No data was lost.

1. 2026年5月26日 UTC 05:09investigating

   We are experiencing a service interruption with Custom Models functionality in US SAAS environment. Predictions to existing deployments are working fine, but users cannot create new custom models. Engineering is investigating the issue and will provide updates as we make further progress.

2. 2026年5月26日 UTC 06:01monitoring

   Engineering has applied a fix in the US SAAS environment which resolved the issue. At the time of issue, some users might have experienced issues with Custom Apps, Data upload and custom model creation. The issue is contained.

3. 2026年5月26日 UTC 06:04monitoring

   We are continuing to monitor for any further issues.

4. 2026年5月26日 UTC 06:18resolved

   Custom Models, Custom Applications & Data upload services are back to Operational state in US SAAS Environment. Issue is Resolved.

1. 2026年5月8日 UTC 03:53monitoring

   We are currently experiencing intermittent service issues in US Production, which are primarily affecting the launch of new workloads for Notebooks, Custom models, and Custom Applications. This issue does not impact existing workloads. This disruption is strongly correlated with an ongoing AWS Availability Zone outage (https://health.aws.amazon.com/health/status), causing resource allocation failures. The team is actively monitoring the situation and tracking updates from AWS.

2. 2026年5月8日 UTC 06:13identified

   We are continuing to experience issues launching new workloads for Custom Models and Custom Applications in US Production. This is connected to an ongoing AWS outage. Our team is exploring multiple mitigation options.

3. 2026年5月8日 UTC 13:40monitoring

   Engineering resolved the underlying issue with workload scheduling and is monitoring the cluster.

4. 2026年5月8日 UTC 20:21resolved

   Engineering confirmed the issue is resolved and all services are restored.

1. 2026年4月10日 UTC 10:24monitoring

   Processing actual messages on JP MTS is delayed due to autoscaling malfunction. Engineering scaled up the deployment to alleviate the issue. Root cause mitigation in progress

2. 2026年4月10日 UTC 11:01resolved

   Engineering has applied the required infrastructure configuration changes. The service is operating normally and no further user impact is observed. Engineering will continue monitoring cluster health to ensure stability. The incident is now marked as Contained.

1. 2026年4月9日 UTC 21:56investigating

   We're experiencing an elevated level of errors and are currently looking into the issue.

2. 2026年4月9日 UTC 22:56monitoring

   A fix has been implemented and we are monitoring the results.

3. 2026年4月10日 UTC 11:31monitoring

   Engineering has applied changes to mitigate the elevated error rates. Services are now operating normally. We are continuing to monitor the system while investigating the cause of the issue.

4. 2026年4月10日 UTC 13:04resolved

   Engineering has implemented the required fixes to resolve the elevated error rates. Services are now operating normally, and no further user impact has been observed. The team will continue to monitor the system to ensure stability. The incident is now considered contained.

1. 2026年3月30日 UTC 20:43identified

   Our engineering team has found the the Quay outage currently happening is causing degraded performance across the DataRobot platform. Engineering is currently monitoring the situation.

2. 2026年3月30日 UTC 20:44identified

   We are continuing to work on a fix for this issue.

3. 2026年3月31日 UTC 08:53resolved

   Quay.io functionality has been restored and DataRobot environments are fully stabilized.

1. 2026年3月13日 UTC 17:49investigating

   We are experiencing performance degradation on Managed AI Cloud.

2. 2026年3月13日 UTC 18:31monitoring

   A fix has been implemented and we are monitoring the results.

3. 2026年3月13日 UTC 18:43resolved

   This incident has been resolved.

1. 2026年3月11日 UTC 20:05investigating

   We are currently investigating this issue.

2. 2026年3月11日 UTC 20:22monitoring

   A fix has been implemented and we are monitoring the results.

3. 2026年3月17日 UTC 18:31monitoring

   We are continuing to monitor for any further issues.

4. 2026年3月17日 UTC 18:32resolved

   This incident has been resolved.

1. 2026年3月11日 UTC 14:36investigating

   DataRobot is experiencing network issue related to Kubernetes in US Cluster. This will have impact on model deployment and predictions. Engineering is investigating the root cause.

2. 2026年3月11日 UTC 15:06identified

   Engineering has identified the root cause of the problem and a mitigation is put in place.

3. 2026年3月11日 UTC 15:19monitoring

   The mitigation implemented by Engineering has improved the network issue. The team is continuing to monitor the environment to ensure full recovery.

4. 2026年3月11日 UTC 16:41resolved

   The mitigation implemented by Engineering has resolved the Kubernetes network issue, and the incident is now contained.

1. 2026年2月18日 UTC 21:04investigating

   Our engineering team has found the the Quay outage currently happening is causing degraded performance across the DataRobot platform.

2. 2026年2月18日 UTC 21:19resolved

   This incident is now resolved.

1. 2026年2月17日 UTC 15:48identified

   LLM blueprints deployments can not be created in JP MTS environment. Engineering is rolling back JP cluster to previous version to mitigate the issue.

2. 2026年2月17日 UTC 15:55resolved

   Rollback of the JP cluster to the previous version is complete and the problem has been mitigated.

1. 2026年2月16日 UTC 11:35identified

   Agent application template is affected with the recent moderations library upgrade, fix is identified and mitigation is in progress.

2. 2026年2月16日 UTC 12:49resolved

   New version of Agentic application template is released, the issue is resolved

1. 2026年2月13日 UTC 20:39investigating

   We are observing issues on DataRobot US MTS environment. Users may experience degraded performance using APIs and data ingest services. The engineering team is currently investigating the root cause.

2. 2026年2月13日 UTC 21:31resolved

   The incident has now been resolved. All services are now operational.

1. 2026年1月8日 UTC 18:15monitoring

   Some customers have reported issue connecting to DataRobot. Please do a hard refresh of your browser by clearing the cache and this should fix the problem. As always let us know if it continue to have issue connecting to DataRobot after clearing the cache.

2. 2026年1月8日 UTC 18:16monitoring

   We are continuing to monitor for any further issues.

3. 2026年1月22日 UTC 14:19resolved

   This incident has been resolved.

1. 2025年12月17日 UTC 17:41resolved

   Between 16:50 UTC and 17:09 UTC, one of external providers(DockerHub) had an outage that might have caused temporary delays in starting platform workloads. The issue has been resolved and normal operations have resumed. Engineering is continuing to monitor the system.

1. 2025年12月5日 UTC 11:09monitoring

   Between 09:04 UTC and 09:17 UTC, one of external providers(DockerHub) had an outage that might have caused temporary delays in starting platform workloads. The issue has been resolved and normal operations have resumed. Engineering is continuing to monitor the system.

2. 2025年12月5日 UTC 11:35resolved

   The engineering team has confirmed recovery across all MTS and STS environments. No further updates are expected.