50 Scale Ai incidents · August 2021 — official updates, affected components, duration and resolution details.
We are currently investigating this issue.
We are continuing to investigate this issue.
This incident has been resolved.
We are currently investigating this issue.
This incident has been resolved.
Nucleus and Data Engine is currently experiencing degraded performance. Pages and queries will take longer to respond than usual.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We are currently investigating this issue.
Cloudflare has implemented a fix and seems to be resolving any certificate issues.
Confirmed that the issue has been resolved.
This incident has been resolved.
We are currently investigating an application issue. We will post updates shortly.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We are currently investigating an issue, some operations are experiencing high latency or are failing.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We are currently investigating this issue.
We are seeing elevated error rates with our cloud provider. We are currently investigating the issue.
We are continuing to work on mitigating the issue. Our website is experiencing a technical issue with the hosting provider. Current ETA for resolution is ~2 hours, or around 6:30 PT.
A temporary fix has been implemented and we are monitoring the status of the platform
We are continuing to monitor for any further issues.
This incident has been resolved.
We have identified a platform issue impacting all Spellbook applications. Applications will not appear in the UI or be available over the API. We are deploying a fix for this issue.
The fix for the deployment is in progress now.
The Spellbook application is fully back up and operational.
We are currently investigating this issue.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
There was an outage with a service that manages the metadata for models from 7:06am PT to 7:50am PT. It prevented model services from starting up new pods. ML services that received security updates, deployed a new version, or attempted to scale up were impacted. There was an additional 8 to 12 minutes of service degradation after restoring the impacted ML services to catch up on missed requests.
We are currently investigating this issue.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We are currently investigating this issue.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We're experiencing an elevated level of API errors and are currently looking into the issue.
This incident has been resolved.
We are currently investigating this issue.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
Networking issue with ML infra.
We are currently investigating this issue.
Many users may still be experiencing slow application loading times and less responsiveness. We have identified that our database has been experiencing internal issues and are continuing to work to resolve them. We have made efforts to mitigate the impact caused and have tuned our database to optimize for the current situation.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We are currently investigating this issue.
We are continuing to investigate this issue.
A fix has been implemented and we are monitoring the results.
We are continuing to monitor for any further issues.
This incident has been resolved.
We are seeing elevated error rates that may indicate a slow or unresponsive platform. We are currently investigating the issue.
This incident has been resolved.
We are currently investigating this issue.
We are continuing to investigate this issue.
Incident has been resolved.
We are currently investigating an issue with our asynchronous jobs.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
