Ghost Inspector Status

Early this morning, we experienced degraded performance affecting Chrome and Firefox test execution. Customers may have noticed:    - Delayed test runs - Chrome and Firefox tests were queued longer than usual.  - Failed results on our end; during a short window (~14:45 –15:10 UTC), some Chrome and Firefox test runs returned a failed result due to a temporary issue in our system. This is not a problem with your tests. The cause has been resolved, and we've added further system capacity to work through the delay. Test execution is catching up now, and we expect normal performance shortly. Any such failed test runs will re-run normally on their next scheduled execution and can be disregarded. All tests stuck in a queued state should be starting to run over the next 2 hours.   All other Ghost Inspector functionality was not affected. We greatly apologize for this system disruption. We take system reliability seriously and will do our best to ensure this does not happen again.

This incident has been resolved.

This incident has been resolved.

We experienced two outages today due to internal database issues. These incidents have now been fully resolved, and services are back to normal. Please note that we may perform additional maintenance work throughout the day; however, we will make every effort to avoid further disruptions. Thank you for your patience and understanding.

One of our test runner instances began returning intermittent Internal Server Error responses, causing some test runs to fail unexpectedly. The issue was traced to an internal fault on the runner, which we promptly identified and removed from the rotation. We have taken steps to ensure the faulty runner is no longer in use and are continuing to monitor for stability across all test executions. We sincerely apologize for the disruption and appreciate your patience.

Recently, our platform experienced an intermittent error due to one Chrome test runner server among many, which had entered a faulty state and was consistently failing. This issue has now been identified and resolved, and we apologize for any inconvenience this may have caused. To confirm the fix, please restart any tests that previously encountered errors. If you continue to experience any problems, please let us know. Our monitoring systems didn’t detect this sooner because the error rate across all tests stayed within limits. We take this seriously and are working to improve our monitoring to catch similar cases more effectively in the future. Thank you again for your assistance and patience.

We experienced roughly 65 minutes of downtime on Tuesday around 5pm PT. While rotating instances in our load balancers, two instances were shut down prior to the newly-provisioned instances being healthy and available. Internal alarms notified the team and the issue was remediated.

Queue depths are back to normal and test runners should be operating as expected. any customers still experiencing issues are asked to contact support.

We've observed almost 100% reduction in timeouts over the last 24 hours and are closing this incident. Any customers still experiencing timeouts within their tests are asked to contact support.

We are rolling back our UI changes out of an abundance of caution and will engage with any impacted customers again prior to the redeployment with fixes. Thank you all for your patience.

The scheduled maintenance has been completed.