Historia awarii Ghost Inspector

Early this morning, we experienced degraded performance affecting Chrome and Firefox test execution. Customers may have noticed:    - Delayed test runs - Chrome and Firefox tests were queued longer than usual.  - Failed results on our end; during a short window (~14:45 –15:10 UTC), some Chrome and Firefox test runs returned a failed result due to a temporary issue in our system. This is not a problem with your tests. The cause has been resolved, and we've added further system capacity to work through the delay. Test execution is catching up now, and we expect normal performance shortly. Any such failed test runs will re-run normally on their next scheduled execution and can be disregarded. All tests stuck in a queued state should be starting to run over the next 2 hours.   All other Ghost Inspector functionality was not affected. We greatly apologize for this system disruption. We take system reliability seriously and will do our best to ensure this does not happen again.

Due to a global AWS outage, our services may be temporarily unavailable. We’ll keep you updated. https://health.aws.amazon.com/health/status

The issue has been identified. Partial functionality of the test runner has been restored, though some disruptions may still occur. We’re closely monitoring the recovery process with our provider, which is currently experiencing a global outage.

We’re seeing steady improvements and will keep monitoring the situation until everything is fully stable.

Service has stabilized and the test runner is operating normally. We’re continuing to monitor the situation and remain alert for any potential disruptions.

We are continuing to monitor for any further issues.

Earlier today, a global AWS outage affected parts of our infrastructure, which caused interruptions in test execution and related services. https://health.aws.amazon.com/health/status The issue has been identified, and partial functionality was restored. While our systems have shown steady improvement, some regions are still experiencing degraded performance, which may cause certain tests to fail or time out. We recommend holding off on running additional tests for now, as performance and stability may continue to fluctuate until AWS fully resolves the issue. Our team is monitoring the situation closely and will ensure everything stabilizes before resuming normal operations. We truly appreciate your patience and sincerely apologize for the inconvenience.

We’re still seeing instability from the ongoing AWS outage affecting parts of our infrastructure. Some tests may fail or time out. Our team is keeping a close eye on things and will restore full stability as soon as AWS resolves the issue. Thanks for bearing with us — we know this is frustrating and appreciate your patience. https://health.aws.amazon.com/health/status

We’re seeing gradual recovery as AWS continues resolving the global outage. You can try restarting tests carefully — some runs may still fail. We’re watching things closely and will update once stability is confirmed.

This incident has been resolved.

We are currently experiencing an outage due to internal database issues. Please note that we may perform additional maintenance work throughout the day; however, we will make every effort to avoid further disruptions. Thank you for your patience and understanding.

This incident has been resolved.

We experienced two outages today due to internal database issues. These incidents have now been fully resolved, and services are back to normal. Please note that we may perform additional maintenance work throughout the day; however, we will make every effort to avoid further disruptions. Thank you for your patience and understanding.

One of our test runner instances began returning intermittent Internal Server Error responses, causing some test runs to fail unexpectedly. The issue was traced to an internal fault on the runner, which we promptly identified and removed from the rotation. We have taken steps to ensure the faulty runner is no longer in use and are continuing to monitor for stability across all test executions. We sincerely apologize for the disruption and appreciate your patience.

Recently, our platform experienced an intermittent error due to one Chrome test runner server among many, which had entered a faulty state and was consistently failing. This issue has now been identified and resolved, and we apologize for any inconvenience this may have caused. To confirm the fix, please restart any tests that previously encountered errors. If you continue to experience any problems, please let us know. Our monitoring systems didn’t detect this sooner because the error rate across all tests stayed within limits. We take this seriously and are working to improve our monitoring to catch similar cases more effectively in the future. Thank you again for your assistance and patience.

We experienced roughly 65 minutes of downtime on Tuesday around 5pm PT. While rotating instances in our load balancers, two instances were shut down prior to the newly-provisioned instances being healthy and available. Internal alarms notified the team and the issue was remediated.

We are currently experiencing a surge in both the Chrome and Firefox queues and are investigating the cause, customers may experience some delays in test runs with both browsers. We will follow up with an update as soon as we have more information

The Firefox capacity issue has been addressed and we are seeing queue levels return to normal, customers should see their enqueued tests complete shortly. Chrome queue volume is still high and we are provisioning additional capacity to help bring levels back to normal. We will update with more information again shortly.

Queue depths are back to normal and test runners should be operating as expected. any customers still experiencing issues are asked to contact support.

We are currently observing an higher than normal level of 10-minute test timeouts across our test infrastructure, beginning around 9pm PT Sunday, May 7th. The root cause is still being investigated, but we suspect the issue lies with Chrome 107. Affected customers are asked to use Chrome 106 for the time being and to reach out to support if the problem persists.

We apologize for the long-running issue, we are currently experiencing an issue with Chrome (Traditional) centred around version 107. We are asking customers still facing the timeout issue to try the Headless version of the browser for the time being to work around the timeouts. If you are still experiencing issues, please reach out to support. We thank you for your patience.

Fixes have been pushed into production and we are seeing a significant reduction in timeouts when running tests with Chrome Traditional. Any customers still experiencing timeouts are asked to contact support so we can investigate further. We want to thank you again for your patience as we worked through this issue.

We've observed almost 100% reduction in timeouts over the last 24 hours and are closing this incident. Any customers still experiencing timeouts within their tests are asked to contact support.

This is an update to let our customers know that we rolled out a major UI update in the last 24 hours as part of our efforts to continually improve Ghost Inspector. This update included a rewrite of our application to a newer framework, and while the deployment has largely been a success, some customers have noticed a few issues with the new UI, including missing some settings, some statuses not updating correctly, etc. We are currently working on these issues and ask you to reach out to support if you notice anything out of the ordinary, and we'll follow up with updates as the day goes on. Thank you from the Ghost Inspector team!

We are rolling back our UI changes out of an abundance of caution and will engage with any impacted customers again prior to the redeployment with fixes. Thank you all for your patience.

A configuration change was put in place this afternoon that resulted in a few minutes of disruption to our email service. The impact was recognized immediately and the change was reverted. We are continuing to monitor our systems, but emails are being received as expected and we anticipate no further disruptions.

As our team internally was testing our PagerDuty integration we accidentally created a false-positive incident that may have notified our customers. We apologize for the additional noise in your day, we've addressed the issue so it cannot reoccur.

At approximately 9:45am PST (GMT-7) we attempted to roll out additional capacity to our Chrome test runner infrastructure. A change was in place that caused a large number of Chrome executions to fail that has impacted many customers. The issue was identified and corrected at 10am PST (GMT-7). Customers that have "Unknown Error" in their results at this time should be able to re-execute those tests & suites normally. We are continuing to monitor the situation and customers are encouraged to reach out to support if you are still experiencing issues.

This incident has been resolved.

We are seeing a higher than normal volume of test executions in our Chrome queue and are working towards provisioning more capacity. Customers may see their test and suite executions taking longer than normal. We will follow up with more details as capacity issues are resolving.

The Chrome queue is now back at full capacity and dequeueing normally, we still have a significant volume to process over the next little bit but we should see the backlog cleared up shortly.

This incident has been resolved.

We are currently experiencing a reduction in capacity for our Chrome test runners and are rolling out remediation. Some Chrome test runs may be taking longer than normal. We will update shortly with more further details.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

We have identified an issue with the search functionality of our application where no results are being returned for a search. We have isolated the cause to an issue with our search index and are working to address the issue. We will follow up with more details when available.

The search index has been restored.

At approximately 11:30am PDT a code change was deployed to our test running service that updated the way we handle JavaScript processing in our step execution. This code had been fully tested in our staging environment and had the green light for production. Shortly after the deployment around noon PDT internal alarms regarding the queue backlog and some other test runner alerts were received internally that indicated diminished test runner capacity. Immediate steps were taken to restore capacity however it was not identified that the code change was causing an ongoing problem until approximately 10pm PDT and test runner capacity gradually diminished over time throughout the day as browser processes ran beyond the 10-minute test run limit due to high resource consumption. Once the team realized capacity was diminishing again we identified the recent code change as the culprit and immediately rolled back the code. Service was restore to normal operating capacity around approximately 11pm PDT. Any customers having received a warning of "10 minute timeout" are encouraged to re-execute the test and contact support if any issues remain.

We're investigating a slow down with Firefox test running.

The test running slow down has been identified and addressed. The Firefox queue is catching up and should be back to 100% capacity shortly.

This issue has been resolved. We will continue to monitor the situation.

The AWS Northern California region (us-west-1) is experiencing network connectivity issues this morning which may affect Ghost Inspector tests running in that geolocation.

Network connectivity is being restored to the AWS Northern California region (us-west-1) and Ghost Inspector tests using that geolocation should begin to operate properly again. We'll continue to monitor this situation as the AWS region recovers.

Network connectivity has been restored to the AWS Northern California region (us-west-1). Ghost Inspector tests using that geolocation should be running without issue again.

We are currently investigating a slow down in running Chrome tests on our system. Tests are still completed but may take slightly longer to begin executing.

We've identified the source of the slow down. Test running is coming back up to speed and working through the queue of Chrome tests. We'll follow up with some additional detail on the issue and the fixes we'll be planning to address it in the future. We expect the queue to be 100% caught up by 12pm PT today.

The Chrome queue is 100% caught up and test runs should be completing in their normal timeframe again. We are in the process of implementing fixes and creating additional monitors to prevent this type of slow down in the future.

Test running capacity is stable. We'll continue to monitor test running capacity and will update this ticket with additional details.