Rainforest Qa outage history

We have discovered and issue with our Windows 11 VMs that is causing them to fail to boot and function. The fix has been identified and we are working to roll it out, but this may take a few hours. In the meantime runs that utilize these VM types will be delayed or may error. We are working to resolve the situation.

We have fixed the issue affecting our VMs and runs appear to be progressing normally now. There may still be some delays as we work through a small backlog of pending runs, and we will continue to monitor the situation.

This incident has been resolved.

We are currently experiencing an issue with one of our providers that is causing the application to be intermittently available.

Our upstream provider is aware of the issue and is continuing to investigate.

Our upstream provider has identified the issue and is currently working on restoring service

Our upstream provider has implemented a fix and services are returning to normal operation. We're continuing to monitor the situation to ensure everything has recovered.

Our provider is still reporting intermittent errors which may have an impact on some clients being able to access Rainforest services. We are continuing to monitor the situation.

This incident has been resolved.

We are investigating an issue with our background job processing which is causing tasks to be delayed. This will impact starting new tests, execution of current tests, and many other pieces of functionality within the application. We are currently working to resolve this.

We have implemented a fix and background job processing appears to be returning to normal. We will continue to investigate to ensure everything is working as it should be. There is a backlog of tests awaiting execution, so test results may continue to be delayed as we work our way through executing them all.

All systems are back to normal.

We're currently investigating an issue affecting user logins. Some users may be having trouble logging in.

We've addressed the issue and logins should be functional for all users.

One of our upstream providers is currently experiencing an outage. We are monitoring the impact of this on our application.

Our upstream provider appears to have addressed the issue and we are no longer seeing any issues, but will continue to monitor the situation.

This incident has been resolved.

We have identified an issue that is affecting step generation, self-healing snippets, and AI test assertions. Those service will be unavailable until such time as the incident is fixed.

A fix has been applied and will be rolling out over the next hour. AI services should now be broadly available for most platforms but some older runs / previews may still be affected. We'll continue to monitor the situation until the rollout is complete.

The fix has been fully applied and all AI services should be back to normal.

We are investigating an issue related to background job processing. Some results or runs may be delayed during this time.

We've implemented a fix and all delayed background jobs have been processed. We are monitoring to make sure everything continues working properly.

This incident has been resolved.

The Rainforest application is currently offline. This affects access to the application, as well as any tests or runs that were in progress. We're looking into it and will provide an update shortly.

The application is back online, and we're monitoring to make sure everything is working properly again.

This incident has been resolved.

We've identified an issue which is causing problems when trying to execute tests. Test runs are not being processed. Virtual machines in the Visual Editor are not available. We're working on a fix.

We've applied a fix and everything should be operational again. We're processing the backlog of runs created in the last 3 hours which will take some time. We'll update the status to "Resolved" once the backlog is processed.

The backlog of runs has been processed and everything should be back to normal.

We are investigating an issue that is delaying the availability of test result videos

We have resolved the issue affecting test video recording. Test runs from approximately 06:30 UTC to 17:35 UTC may be missing their video recordings.

We are working on an issue that is delaying the allocation of virtual machines when starting a run. We have identified the issue and are working to resolve it.

We have applied a temporary fix to mitigate the issue and are continuing to work on a more permanent fix.

A permanent fix has been implemented and performance appears to have returned to normal. No further delays are expected.

This incident has been resolved.

We've identified an issue that may have caused some runs to be delayed.

We've fixed the issue and runs are now starting and progressing as usual.

We are currently investigating the issue.

A fix has been implemented, we are now monitoring.

The incident has been resolved.

We've noticed that some test executions are not being processed correctly, which is holding up the completion of runs. We're currently investigating it.

We've identified the component we believe is causing the problem and are investigating why it's behaving badly.

We've shipped a fix and are monitoring the situation.

This incident has been resolved.

Operations involving AI are taking some time to complete. We're investigating the issue.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

We realized that our provider for AI infrastructure was having a soft outage which was causing some requests to take a very long time \(10 minutes\) to error. We worked around this by decreasing the timeout for the requests so that they would error faster and we could then retry them.

We’ve identified another problem with action generation and are working on a fix

We implemented a fix and are monitoring the results.

This incident has been resolved.

We’ve identified a problem with action generation and are working on a fix”

We've shipped a fix and will continue to monitor the situation.

The issue has been resolved

We're experiencing issues with failure summarization. The fix is being deployed.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

We are currently experiencing an issue with runs being delayed and currently working on a fix.

We are continuing to investigate this issue.

We've identified the cause of the delays and are working on a fix.

We've shipped a fix and will continue to monitor the situation.

This incident has been resolved.

We've noticed a problem with our fleet of VMs, which are used for test execution and test authoring. This may mean that you can't edit tests, and that test execution will be delayed.

The problem is somewhat improved, but we're continuing to investigate.

This incident has been resolved.