Rainforest Qa 장애 기록

1. 2026년 6월 24일 오전 6:22 UTCidentified

   우리는 부팅 및 기능에 실패하기 위해 Windows 11 VMs와 함께 발견하고 문제가 있습니다. 수정이 확인되었습니다. 우리는 그것을 밖으로 구하기 위해 노력하고 있습니다. 그러나 이것은 몇 시간 걸릴 수 있습니다. 이러한 VM 유형의 사용은 지연되거나 오류가 발생할 수 있음을 의미합니다. 우리는 상황을 해결하기 위해 노력하고 있습니다.

2. 2026년 6월 24일 오전 10:29 UTCmonitoring

   우리는 우리의 VM에 영향을 미치는 문제를 해결하고 일반적으로 진행 될 것으로 보인다. 우리는 작은 백그라운드 실행을 통해 일하면서 여전히 몇 가지 지연이있을 수 있으며, 우리는 상황을 모니터링 할 것입니다.

3. 2026년 6월 24일 오후 1:55 UTCresolved

   이 사건은 해결되었습니다.

공식 인시던트 업데이트를 자동 번역했습니다.

1. 2025년 11월 18일 오전 11:51 UTCidentified

   We are currently experiencing an issue with one of our providers that is causing the application to be intermittently available.

2. 2025년 11월 18일 오후 12:25 UTCidentified

   Our upstream provider is aware of the issue and is continuing to investigate.

3. 2025년 11월 18일 오후 2:05 UTCidentified

   Our upstream provider has identified the issue and is currently working on restoring service

4. 2025년 11월 18일 오후 2:59 UTCmonitoring

   Our upstream provider has implemented a fix and services are returning to normal operation. We're continuing to monitor the situation to ensure everything has recovered.

5. 2025년 11월 18일 오후 4:40 UTCmonitoring

   Our provider is still reporting intermittent errors which may have an impact on some clients being able to access Rainforest services. We are continuing to monitor the situation.

6. 2025년 11월 18일 오후 6:19 UTCresolved

   This incident has been resolved.

1. 2025년 9월 8일 오후 5:02 UTCinvestigating

   We are investigating an issue with our background job processing which is causing tasks to be delayed. This will impact starting new tests, execution of current tests, and many other pieces of functionality within the application. We are currently working to resolve this.

2. 2025년 9월 8일 오후 5:16 UTCmonitoring

   We have implemented a fix and background job processing appears to be returning to normal. We will continue to investigate to ensure everything is working as it should be. There is a backlog of tests awaiting execution, so test results may continue to be delayed as we work our way through executing them all.

3. 2025년 9월 8일 오후 5:39 UTCresolved

   All systems are back to normal.

1. 2025년 7월 28일 오후 1:01 UTCinvestigating

   We're currently investigating an issue affecting user logins. Some users may be having trouble logging in.

2. 2025년 7월 28일 오후 1:16 UTCresolved

   We've addressed the issue and logins should be functional for all users.

1. 2025년 6월 12일 오후 6:50 UTCinvestigating

   One of our upstream providers is currently experiencing an outage. We are monitoring the impact of this on our application.

2. 2025년 6월 12일 오후 8:47 UTCmonitoring

   Our upstream provider appears to have addressed the issue and we are no longer seeing any issues, but will continue to monitor the situation.

3. 2025년 6월 12일 오후 9:13 UTCresolved

   This incident has been resolved.

1. 2025년 4월 22일 오후 3:06 UTCidentified

   We have identified an issue that is affecting step generation, self-healing snippets, and AI test assertions. Those service will be unavailable until such time as the incident is fixed.

2. 2025년 4월 22일 오후 3:39 UTCmonitoring

   A fix has been applied and will be rolling out over the next hour. AI services should now be broadly available for most platforms but some older runs / previews may still be affected. We'll continue to monitor the situation until the rollout is complete.

3. 2025년 4월 22일 오후 4:03 UTCresolved

   The fix has been fully applied and all AI services should be back to normal.

1. 2025년 4월 7일 오후 5:30 UTCinvestigating

   We are investigating an issue related to background job processing. Some results or runs may be delayed during this time.

2. 2025년 4월 7일 오후 5:37 UTCmonitoring

   We've implemented a fix and all delayed background jobs have been processed. We are monitoring to make sure everything continues working properly.

3. 2025년 4월 7일 오후 5:52 UTCresolved

   This incident has been resolved.

1. 2025년 3월 28일 오후 4:29 UTCidentified

   The Rainforest application is currently offline. This affects access to the application, as well as any tests or runs that were in progress. We're looking into it and will provide an update shortly.

2. 2025년 3월 28일 오후 4:35 UTCmonitoring

   The application is back online, and we're monitoring to make sure everything is working properly again.

3. 2025년 3월 28일 오후 4:48 UTCresolved

   This incident has been resolved.

1. 2025년 1월 7일 오후 12:33 UTCidentified

   We've identified an issue which is causing problems when trying to execute tests. Test runs are not being processed. Virtual machines in the Visual Editor are not available. We're working on a fix.

2. 2025년 1월 7일 오후 1:40 UTCmonitoring

   We've applied a fix and everything should be operational again. We're processing the backlog of runs created in the last 3 hours which will take some time. We'll update the status to "Resolved" once the backlog is processed.

3. 2025년 1월 7일 오후 2:01 UTCresolved

   The backlog of runs has been processed and everything should be back to normal.

1. 2024년 10월 10일 오후 4:30 UTCinvestigating

   We are investigating an issue that is delaying the availability of test result videos

2. 2024년 10월 10일 오후 5:52 UTCresolved

   We have resolved the issue affecting test video recording. Test runs from approximately 06:30 UTC to 17:35 UTC may be missing their video recordings.

1. 2024년 10월 2일 오후 12:48 UTCidentified

   We are working on an issue that is delaying the allocation of virtual machines when starting a run. We have identified the issue and are working to resolve it.

2. 2024년 10월 2일 오후 1:42 UTCidentified

   We have applied a temporary fix to mitigate the issue and are continuing to work on a more permanent fix.

3. 2024년 10월 2일 오후 7:28 UTCmonitoring

   A permanent fix has been implemented and performance appears to have returned to normal. No further delays are expected.

4. 2024년 10월 2일 오후 8:15 UTCresolved

   This incident has been resolved.

1. 2024년 10월 1일 오후 7:41 UTCidentified

   We've identified an issue that may have caused some runs to be delayed.

2. 2024년 10월 1일 오후 7:54 UTCresolved

   We've fixed the issue and runs are now starting and progressing as usual.

1. 2024년 9월 18일 오후 4:43 UTCinvestigating

   We are currently investigating the issue.

2. 2024년 9월 18일 오후 5:52 UTCmonitoring

   A fix has been implemented, we are now monitoring.

3. 2024년 9월 18일 오후 6:19 UTCresolved

   The incident has been resolved.

1. 2024년 6월 12일 오후 11:24 UTCinvestigating

   We've noticed that some test executions are not being processed correctly, which is holding up the completion of runs. We're currently investigating it.

2. 2024년 6월 13일 오전 12:02 UTCinvestigating

   We've identified the component we believe is causing the problem and are investigating why it's behaving badly.

3. 2024년 6월 13일 오전 2:15 UTCmonitoring

   We've shipped a fix and are monitoring the situation.

4. 2024년 6월 13일 오전 6:05 UTCresolved

   This incident has been resolved.

1. 2024년 6월 7일 오후 12:11 UTCinvestigating

   Operations involving AI are taking some time to complete. We're investigating the issue.

2. 2024년 6월 7일 오후 12:57 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

3. 2024년 6월 7일 오후 2:46 UTCresolved

   This incident has been resolved.

4. 2024년 6월 11일 오후 1:16 UTCpostmortem

   We realized that our provider for AI infrastructure was having a soft outage which was causing some requests to take a very long time \(10 minutes\) to error. We worked around this by decreasing the timeout for the requests so that they would error faster and we could then retry them.

1. 2024년 6월 6일 오후 3:39 UTCidentified

   We’ve identified another problem with action generation and are working on a fix

2. 2024년 6월 6일 오후 4:23 UTCmonitoring

   We implemented a fix and are monitoring the results.

3. 2024년 6월 6일 오후 4:36 UTCresolved

   This incident has been resolved.

1. 2024년 6월 6일 오후 2:53 UTCidentified

   We’ve identified a problem with action generation and are working on a fix”

2. 2024년 6월 6일 오후 2:58 UTCmonitoring

   We've shipped a fix and will continue to monitor the situation.

3. 2024년 6월 6일 오후 3:07 UTCresolved

   The issue has been resolved

1. 2024년 5월 8일 오전 10:04 UTCidentified

   We're experiencing issues with failure summarization. The fix is being deployed.

2. 2024년 5월 8일 오전 10:13 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

3. 2024년 5월 8일 오전 10:20 UTCresolved

   This incident has been resolved.

1. 2024년 4월 15일 오전 10:38 UTCinvestigating

   We are currently experiencing an issue with runs being delayed and currently working on a fix.

2. 2024년 4월 15일 오전 10:40 UTCinvestigating

   We are continuing to investigate this issue.

3. 2024년 4월 15일 오전 10:56 UTCidentified

   We've identified the cause of the delays and are working on a fix.

4. 2024년 4월 15일 오후 12:49 UTCmonitoring

   We've shipped a fix and will continue to monitor the situation.

5. 2024년 4월 15일 오후 6:24 UTCresolved

   This incident has been resolved.

1. 2024년 4월 12일 오후 2:59 UTCinvestigating

   We've noticed a problem with our fleet of VMs, which are used for test execution and test authoring. This may mean that you can't edit tests, and that test execution will be delayed.

2. 2024년 4월 12일 오후 4:59 UTCinvestigating

   The problem is somewhat improved, but we're continuing to investigate.

3. 2024년 4월 12일 오후 6:23 UTCresolved

   This incident has been resolved.