Labflow 장애 기록

At 4:47PM (CST) on 12/9/2025, Students started experiencing failures of file uploads. This impacts report assignments that required file uploads such as images or PDFs. We have identified the problem and our engineering team is working with the cloud file upload provider to resolved this issue.

We are continuing to work on a fix for this issue.

This incident has been resolved.

Google Cloud Platform is experiencing degraded networking performance and service outages. Labflow is hosted on Google Cloud Platform and is also experiencing degraded performance. This is also affecting other services hosted on Google's cloud including Cloudflare, Spotify, Snapchat, and many others. We are diligently monitoring the situation and will report back with any updates from Google. At this time you may experience "red banner" errors. To avoid any data integrity issues, we recommend waiting to submit your work until the outage has been resolved.

Google Cloud is still experiencing widespread service outages. Google engineers are beginning to apply fixes and workarounds but we are still awaiting confirmation that Google's services that Labflow relies on are fully operational. We will give another update as soon as we have more information from Google, see improvements in Labflow performance, or in 30 minutes.

Google is reporting that services are continuing to slowly come online. The Catalyst Education team is still observing intermittent connectivity issues with Google Cloud services. We recommend continuing to wait to submit your work until the outage has been resolved. We will give another update as soon as we have more information from Google, see improvements in Labflow performance, or in 30 minutes.

We are observing a recovery in the Google Cloud services that Labflow relies on. At this time you may resume normal activity in Labflow. We will send a final update after Google confirms that the outage has been fully resolved.

This incident has been resolved. At this time please feel free to resume using Labflow. The Catalyst Education team will append a full incident report in the coming days as Google provides more information on the root cause of this incident. If you are still experiencing errors, please reach out to [email protected] with any concerns. We sincerely apologize for any inconvenience that this may have caused.

We are currently investigating this issue.

We are continuing to investigate this issue.

Labflow's main database service had incurred a critical error. The system switched over to its fail-over and came back online. We are continuing to monitor.

This incident has been resolved.

The Labflow site is currently unavailable to users. The issue started around 3:10 pm Central. We are diagnosing the issue and will update when we have more information. We will update status in 30 mins.

Labflow site is currently running. We are continuing to monitor the situation.

The incident has been resolved. We will continue to monitor the system to ensure that service remains available.

**Labflow Service Disruption Incident** **Severity:** COMPLETE DISRUPTION **Impacted:** ALL USERS **Status:** Resolved **Time Frame:** Starting at: At 3:10PM \(CDT\) on 10/8/24 Ending at: At 3:36PM \(CDT\) on 10/8/24 Duration: 26 minutes **Incident Overview:** At 3:10PM \(CDT\) on 10/8/24, the entire Labflow platform became unresponsive, impacting all users attempting to access [Labflow.com](http://Labflow.com) or [Labflow.ca](http://Labflow.ca). The Catalyst engineering team determined that the cause was a primary database failure, which did not trigger the expected failover to the secondary server. We immediately engaged our database infrastructure vendor to resolve the issue. Labflow services were fully restored at 3:36PM \(CDT\). **Root Cause:**  Catalyst is awaiting the final root cause analysis from our database infrastructure vendor. We will provide further updates on the Labflow status page as soon as they become available: [http://status.labflow.com](http://status.labflow.com)  **Mitigations:** While there were no immediate mitigations possible during the outage, our engineering team will continue to monitor the system closely for stability. We are also working with our vendor to prevent similar incidents from occurring in the future. **Student Impact:** Students were unable to access Labflow for 26 minutes, from 3:10PM to 3:36PM \(CDT\). -------------- We understand the disruption this outage may have caused and sincerely apologize for the inconvenience. If you have any questions or concerns, please don't hesitate to reach out to your faculty success manager.

Labflow Service Disruption Incident Severity: Complete Disruption Impacted: All Labflow.ca users. (CANADA/ALL USERS) Status: Resolved Incident Impact: At 12:30pm (CST) on 6/25/24, All Labflow.ca users became unable to access the Labflow.ca site. Time Frame: Starting at: First Observed At 12:30pm (CST) on 6/25/24 Ending at: 1:11PM (CST) Duration: 41 MINUTES Incident Overview: All users logging into Labflow.ca were unable to access the site. Root Cause: The cause of this issue is due to an invalid SSL certificate for the .ca domain. In the most recent software version of Labflow that was deployed on 6/24/24, the retained valid SSL domain certificate for Labflow.ca was not included in its proper iteration. Mitigations: The Labflow support team deployed a fix in the latest version of the Labflow platform software, which has resolved all issues related to the incident.

We are continuing to monitor the status of [Labflow.ca](http://Labflow.ca) and will be adding additional system health checks.

Labflow's file upload service (Filestack) is responding with errors during the upload process. We are working with the vendor to resolve this issue as quickly as possible

We are continuing to work on a fix for this issue.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

Labflow Outage Incident (Complete) Status: Resolved At 5:3OPM (CDT) on 3/22/23, students started experiencing intermittent slowness while in Labflow. At 7:00PM (CDT) on 3/22/23, the Labflow site became unavailable. Time Frame: Starting at: 5:30PM (CDT) on 3/23/23 Ending at: 7:45PM (CDT) on 3/23/23 Duration: 2 hours 45 minutes Root Cause: An update to the data recording processes in Labflow caused too many read requests to the Labflow database resulting in the site to eventually become unresponsive. Remediation: Labflow reverted to the previous version of the software. Student Impact: Between 5:30 pm (CDT) and 7:00 pm (CDT), students would have experienced intermittent slowdowns. If they were taking a quiz, they may have been kicked out of the quiz if the pages were taking too long to load. Between 7:00 pm (CDT) and 7:45 pm (CDT), students were not able to access Labflow at all. Starting at 7:45 pm (CDT) students were able to log into Labflow again. We reached out to students who contacted Labflow support to let them know the site is working again. Again, we apologize for any inconvenience and thank you for your patience!

The issue has been identified. The external service used for file uploads and viewing, Filestack, is experiencing degraded service. Labflow users may experience issues with file previews and uploading files within reports.

Filestack has identified the issue and has implemented a fix. We expect service to return to normal by 11am (CST).

A fix has been implemented we are continuing to monitor user experience.

This incident has been resolved.

Labflow Outage Incident (Partial) Status: In-Progress At 8:30AM (CST) on 1/20/23, Students started experiencing intermittent failure of file uploads and file viewing. Time Frame: Starting at: 8:30AM (CST) on 1/20/23 Ending at: (In-Progress) Duration: (In-Progress) Root Cause: (TBD) Labflow’s external service that processes file uploads and file viewing, FileStack, is experiencing intermittent issues. Remedies: Labflow is working with the vendor to remediate this issue as quickly as possible. Labflow needs to temporarily disable the ability to upload files within reports. Student Impact: The intermittent failure of file uploads and file viewing is creating a frustrating experience for students where they may encounter an error preventing them from uploading a file one time, but are then successful on other attempts, or situations where 1 or 2 of their files do not appear to be uploaded but others appear successful.

We are continuing to investigate this issue.

The root cause of the issue has been identified. The external service used for file uploads and viewing, Filestack, is now testing solutions.

We are continuing to monitor the external service's progress. No additional information is available at this time.

Status: Resolved At 8:30AM (CST) on 1/20/23, Students started experiencing intermittent failure of file uploads and file viewing. This impacted report assignments that required file uploads such as images or PDFs. Time Frame: Starting at: 8:30AM (CST) on 1/20/23 Ending at: 3:20PM (CST) on 1/20/23 Duration: 6 hours 50 minutes Root Cause: Labflow’s vendor that processes file uploads and file viewing, FileStack, was experiencing intermittent issues. Labflow is expecting a follow-up root cause analysis report to understand their future remediation strategies. Remediation: Labflow has re-enabled the file uploads and file viewing functionality as of 3:20PM (CST). Student Impact: The intermittent failure of file uploads and file viewing created a frustrating experience for students where they may encounter an error preventing them from uploading a file one time, but are then successful on other attempts, or situations where 1 or 2 of their files do not appear to be uploaded but others appear successful.

The Labflow engineering team has successfully completed the previously announced scheduled maintenance. ***When*** Scheduled time: Saturday, May 14th, 6:00 am - 9:00 am (Central Time) Actual time: Saturday, May 14th, 6:00 am - 7:40 am (Central Time) ***Impact*** Labflow was offline and completely unavailable to all users (instructors, TAs and students). Only Labflow's maintenance message page was displayed during this time. ***Important Updates*** We have updated how the quiz question list is viewed with this release. The new QUESTIONS tab replaces the old EDIT tab within a quiz. The questions tab will show you the list of questions in the quiz and allow you to switch to a preview mode so you can see the question stem, solution, and explanation for the question. When you first open a quiz now, you will see the new questions tab. The first time you open a quiz with a question pool, you will get a notice asking if you want to update the format of the question pool. Go ahead and change the format of the pool. The new question pool format allows us to more easily manage the questions included. It is important to note this new page will not impact any existing student attempts at the quiz. Initially, we are limiting the functionality available to instructors on this tab. In future releases, we will add more functionality available to you. If you need any help editing your quiz, contact your faculty success manager. We want to thank the instructors who helped us in the design and review of this new page. We hope that this page will help you to review the questions in the quizzes in a more efficient manner.

- Labflow App Outage Time Frame: Start: 02/10/2022 11:18:11 AM End: 02/10/2022 11:45:11 AM Duration: 27 minutes Root Cause: Labflow's underlying database service failed. Labflow's engineering team is working with its database vendor to understand why the database service failed. We will update this incident when more information is available. Remedies: The database service was restarted allowing all Labflow services to become operational again. We do not expect future incidents at this point. Student Impact: Students were unable to access Labflow at this time.

After further investigation with our database vendor, it was concluded that 2 major factors contributed to this outage. 1. Database queries were executed with an unexpected consumption of RAM that resulted in cluster’s primary node restarting. 2. When the primary node failed, the cluster did not fail-over to the secondary node due to a bug in the vendor's cluster software resulting in the cluster disconnecting from Labflow.