Historia awarii Phantombuster

At approximately 16:30 UTC, an unusually high volume of specific write queries from our application began to strain our main database. This load overwhelmed our internal data replication system. As a result, a backlog of data awaiting replication quickly grew, consuming all available space on the main database's system disk and forcing it offline. We are currently restoring the database from a snapshot; however, this process is taking longer than expected because the snapshot includes this large volume of non-replicated data, which is inherently slower to process during recovery.

The data restoration process is nearly complete. We estimate that the system will be fully back to normal operation within the next hour. Thank you for your patience as we bring services back online.

Our systems have fully recovered and are back to normal operation. The data restoration process has completed successfully, and all services—API and Phantoms—are now functioning as expected. We’ve addressed the issue caused by an unusually high volume of write queries that led to a backlog in our internal replication system. We sincerely thank you for your patience throughout the resolution process. If you continue to experience any issues, please don’t hesitate to reach out to our support team.

Due to major network connectivity problems at one of our infrastructure providers (OVH), we saw very high Phantom launch failure rates (>60%) for a duration of around 1 hour. The incident quickly resolved itself when our provider corrected the connectivity problem. https://twitter.com/ovh_status/status/1448204876232728579

Our support ticket system, provided by Zendesk, is currently experiencing issues which might prevent customers from contacting our support team as well as prevent our support team from solving tickets. We're monitoring the situation. Our help center is still available at https://support.phantombuster.com/hc/en-001. Zendesk provides information on this page: https://status.zendesk.com/

Zendesk is actively working on resolving this outage. Things are progressively getting back to normal. We're still monitoring.

This incident has been resolved.

We saw very high Phantom launch error rates. It was due to major network connectivity problems on the NA<>EU link of OVH, one of our infrastructure providers. After prompt intervention of their service reliability engineers, everything went back to normal.