Störungsverlauf von Scale Ai

Degraded performance

Beginn 6. Januar 2025 um 19:34 UTC 2h 37m

OutageSchwerwiegender Vorfall

Betroffene Komponenten

APIPlatformWeb Application

6. Januar 2025 um 19:34 UTCinvestigating
We are currently investigating this issue.
6. Januar 2025 um 19:40 UTCinvestigating
We are continuing to investigate this issue.
6. Januar 2025 um 22:11 UTCresolved
This incident has been resolved.

Site Outage

Beginn 26. April 2024 um 19:24 UTC 36m

IssuesGeringfügiger Vorfall

Betroffene Komponenten

APIPlatformWeb Application

26. April 2024 um 19:24 UTCinvestigating
We are currently investigating this issue.
26. April 2024 um 20:01 UTCresolved
This incident has been resolved.

Nucleus Degraded Performance

Beginn 19. März 2024 um 10:08 UTC 3h 14m

IssuesGeringfügiger Vorfall

Betroffene Komponenten

Nucleus

19. März 2024 um 10:08 UTCinvestigating
Nucleus and Data Engine is currently experiencing degraded performance. Pages and queries will take longer to respond than usual.
19. März 2024 um 12:37 UTCmonitoring
A fix has been implemented and we are monitoring the results.
19. März 2024 um 13:22 UTCresolved
This incident has been resolved.

Cloudflare Outage

Beginn 15. Januar 2024 um 19:54 UTC 6h 19m

OutageKritischer Vorfall

Betroffene Komponenten

Commercial Web Application

15. Januar 2024 um 19:54 UTCinvestigating
We are currently investigating this issue.
15. Januar 2024 um 19:55 UTCinvestigating
Cloudflare has implemented a fix and seems to be resolving any certificate issues.
15. Januar 2024 um 19:55 UTCinvestigating
Confirmed that the issue has been resolved.
16. Januar 2024 um 02:13 UTCresolved
This incident has been resolved.

Donovan Web Application Outage

Beginn 20. November 2023 um 19:59 UTC 32m

OutageKritischer Vorfall

Betroffene Komponenten

Commercial Web Application

20. November 2023 um 19:59 UTCinvestigating
We are currently investigating an application issue. We will post updates shortly.
20. November 2023 um 20:27 UTCmonitoring
A fix has been implemented and we are monitoring the results.
20. November 2023 um 20:31 UTCresolved
This incident has been resolved.

Site Outage

Beginn 23. Oktober 2023 um 19:29 UTC 1h 9m

Pending

Betroffene Komponenten

APIDocument AISpellbookQuality AssurancePlatformWeb ApplicationNucleus

23. Oktober 2023 um 19:29 UTCinvestigating
We are currently investigating an issue, some operations are experiencing high latency or are failing.
23. Oktober 2023 um 20:25 UTCmonitoring
A fix has been implemented and we are monitoring the results.
23. Oktober 2023 um 20:38 UTCresolved
This incident has been resolved.

Elevated Error Rate

Beginn 18. September 2023 um 18:35 UTC 23h 15m

OutageKritischer Vorfall

Betroffene Komponenten

APIDocument AIQuality AssurancePlatformWeb ApplicationNucleus

18. September 2023 um 18:35 UTCinvestigating
We are currently investigating this issue.
18. September 2023 um 19:09 UTCidentified
We are seeing elevated error rates with our cloud provider. We are currently investigating the issue.
18. September 2023 um 23:25 UTCidentified
We are continuing to work on mitigating the issue. Our website is experiencing a technical issue with the hosting provider. Current ETA for resolution is ~2 hours, or around 6:30 PT.
19. September 2023 um 00:30 UTCmonitoring
A temporary fix has been implemented and we are monitoring the status of the platform
19. September 2023 um 00:52 UTCmonitoring
We are continuing to monitor for any further issues.
19. September 2023 um 17:50 UTCresolved
This incident has been resolved.

Spellbook Application Error

Beginn 16. Mai 2023 um 23:12 UTC 1h 16m

OutageKritischer Vorfall

Betroffene Komponenten

Spellbook

16. Mai 2023 um 23:12 UTCidentified
We have identified a platform issue impacting all Spellbook applications. Applications will not appear in the UI or be available over the API. We are deploying a fix for this issue.
16. Mai 2023 um 23:48 UTCidentified
The fix for the deployment is in progress now.
17. Mai 2023 um 00:27 UTCresolved
The Spellbook application is fully back up and operational.

Catalog Forge Platform Unresponsive

Beginn 28. April 2023 um 14:26 UTC 3h 14m

OutageKritischer Vorfall

Betroffene Komponenten

Catalog Forge

28. April 2023 um 14:26 UTCinvestigating
We are currently investigating this issue.
28. April 2023 um 15:59 UTCmonitoring
A fix has been implemented and we are monitoring the results.
28. April 2023 um 17:40 UTCresolved
This incident has been resolved.

ML Pod Startup Failures

Beginn 30. März 2023 um 14:30 UTC 0m

Pending

30. März 2023 um 17:13 UTCresolved
There was an outage with a service that manages the metadata for models from 7:06am PT to 7:50am PT. It prevented model services from starting up new pods. ML services that received security updates, deployed a new version, or attempted to scale up were impacted. There was an additional 8 to 12 minutes of service degradation after restoring the impacted ML services to catch up on missed requests.

Site Outage

Beginn 11. März 2023 um 10:58 UTC 1h 36m

OutageKritischer Vorfall

Betroffene Komponenten

APIDocument AIQuality AssurancePlatformWeb ApplicationNucleus

11. März 2023 um 10:58 UTCinvestigating
We are currently investigating this issue.
11. März 2023 um 12:20 UTCmonitoring
A fix has been implemented and we are monitoring the results.
11. März 2023 um 12:34 UTCresolved
This incident has been resolved.

Elevated latencies

Beginn 3. März 2023 um 23:59 UTC 1h 42m

IssuesGeringfügiger Vorfall

Betroffene Komponenten

APICatalog ExplorerDocument AISpellbookQuality AssurancePlatformWeb ApplicationNucleusCatalog Forge

3. März 2023 um 23:59 UTCinvestigating
We are currently investigating this issue.
4. März 2023 um 00:15 UTCmonitoring
A fix has been implemented and we are monitoring the results.
4. März 2023 um 01:41 UTCresolved
This incident has been resolved.

Elevated API Error Rates

Beginn 3. März 2023 um 01:30 UTC 1h 15m

OutageSchwerwiegender Vorfall

Betroffene Komponenten

APIDocument AIQuality AssurancePlatformWeb ApplicationNucleus

3. März 2023 um 02:19 UTCinvestigating
We're experiencing an elevated level of API errors and are currently looking into the issue.
3. März 2023 um 02:45 UTCresolved
This incident has been resolved.

Catalog Forge slow or unresponsive

Beginn 27. Januar 2023 um 19:31 UTC 1h 26m

OutageSchwerwiegender Vorfall

Betroffene Komponenten

Catalog Forge

27. Januar 2023 um 19:31 UTCinvestigating
We are currently investigating this issue.
27. Januar 2023 um 20:42 UTCmonitoring
A fix has been implemented and we are monitoring the results.
27. Januar 2023 um 20:57 UTCresolved
This incident has been resolved.

ML Inference Failures

Beginn 29. Dezember 2022 um 19:00 UTC 0m

Pending

29. Dezember 2022 um 21:03 UTCresolved
Networking issue with ML infra.

Degraded Site Performance

Beginn 29. Dezember 2022 um 07:08 UTC 17h 25m

OutageSchwerwiegender Vorfall

Betroffene Komponenten

APIDocument AIPlatformWeb ApplicationNucleus

29. Dezember 2022 um 07:08 UTCinvestigating
We are currently investigating this issue.
29. Dezember 2022 um 11:57 UTCidentified
Many users may still be experiencing slow application loading times and less responsiveness. We have identified that our database has been experiencing internal issues and are continuing to work to resolve them. We have made efforts to mitigate the impact caused and have tuned our database to optimize for the current situation.
29. Dezember 2022 um 22:36 UTCmonitoring
A fix has been implemented and we are monitoring the results.
30. Dezember 2022 um 23:32 UTCresolved
This incident has been resolved.

Platform slow or unresponsive

Beginn 1. Dezember 2022 um 08:42 UTC 3h 45m

OutageKritischer Vorfall

Betroffene Komponenten

Platform

1. Dezember 2022 um 08:42 UTCinvestigating
We are currently investigating this issue.
1. Dezember 2022 um 11:41 UTCinvestigating
We are continuing to investigate this issue.
1. Dezember 2022 um 11:52 UTCmonitoring
A fix has been implemented and we are monitoring the results.
1. Dezember 2022 um 12:13 UTCmonitoring
We are continuing to monitor for any further issues.
1. Dezember 2022 um 12:27 UTCresolved
This incident has been resolved.

Platform Slow or Unresponsive

Beginn 27. November 2022 um 13:24 UTC 45m

OutageKritischer Vorfall

Betroffene Komponenten

APIDocument AIQuality AssurancePlatformWeb ApplicationNucleus

27. November 2022 um 13:24 UTCinvestigating
We are seeing elevated error rates that may indicate a slow or unresponsive platform. We are currently investigating the issue.
27. November 2022 um 14:09 UTCresolved
This incident has been resolved.

Platform slow or unresponsive

Beginn 23. November 2022 um 17:04 UTC 1h 10m

OutageSchwerwiegender Vorfall

Betroffene Komponenten

APIDocument AIQuality AssurancePlatformWeb ApplicationNucleus

23. November 2022 um 17:04 UTCinvestigating
We are currently investigating this issue.
23. November 2022 um 17:06 UTCinvestigating
We are continuing to investigate this issue.
23. November 2022 um 18:15 UTCresolved
Incident has been resolved.

Degraded performance

Beginn 10. November 2022 um 13:38 UTC 5h 23m

IssuesGeringfügiger Vorfall

Betroffene Komponenten

APIDocument AIQuality AssurancePlatformWeb ApplicationNucleus

10. November 2022 um 13:38 UTCinvestigating
We are currently investigating an issue with our asynchronous jobs.
10. November 2022 um 18:45 UTCmonitoring
A fix has been implemented and we are monitoring the results.
10. November 2022 um 19:01 UTCresolved
This incident has been resolved.