Historia awarii Respondus

We are currently investigating an intermittent issue with some users being unable to start a Respondus Monitor session. Access to the LockDown Browser Dashboard also appears to be affected. Updates to follow.

The Respondus Monitor service has been restored and should be functioning normally again. We will continue to monitor the service to ensure stability.

Respondus Monitor continues to function normally. No further updates will be provided.

We are investigating an issue relating to Canvas New Quizzes that causes LockDown Browser for Mac and iPad to fail at startup. Impact: - LockDown Browser for iPad: Displays a blank screen after the begin button is pressed - LockDown Browser for Mac: Displays error code -379 after the begin button is pressed This issue does NOT affect LockDown Browser for Windows or Chromebook. Those platforms may be used as a temporary workaround while Instructure and Respondus investigate.

We continue to investigate this issue with Instructure.

The issue has been identified and Instructure plans to issue a hotfix shortly.

Instructure has deployed a hotfix that resolves the issue causing LockDown Browser for Mac and iPad to fail when launching Canvas New Quizzes.

Respondus Monitor is experiencing intermittent issues related to an ongoing AWS incident. The impact is heightened because many users are also unable to launch exams from their learning systems, which are affected by the same AWS disruption. Our development team is actively monitoring the situation as a top priority.

We continue to monitor the widespread AWS outage that has affected many vendors’ ability to scale their systems during periods of high usage. The impact on Respondus Monitor has been limited, as most users were also unable to launch exams from their learning systems due to the same AWS incident. Respondus Monitor remains fully operational, and our development team continues to monitor the situation as a top priority.

Resolved – Respondus Monitor is fully operational and has experienced no service degradation for several hours. Some assessment systems that integrate with Respondus Monitor continue to experience issues related to the AWS outage, which may give the appearance of instability in our service. On our end, all systems are operating normally. We consider this issue resolved but will open a new incident report if new issues arise.

Exams using Respondus Monitor are currently experiencing slow response times. We are investigating the issue and will provide updates as we have them.

Performance has now returned to normal. The slowdown has been resolved.

This incident has been resolved.

We are currently investigating an outage of StudyMate Campus.

The service has resumed and we are monitoring the system.

The service is functioning normally.

The Respondus Monitor service is currently experiencing a slowdown. We are investigating the source of the problem.

We continue to experience problems with users being able to start new Respondus Monitor sessions. We will provide another update shortly.

We continue to work with AWS to troubleshoot this issue. Service continues to be degraded for Respondus Monitor. The same issue is impacting Chromebook users of LockDown Browser.

This issue is ongoing and we continue to work with AWS on a resolution. Respondus Monitor users are experiencing very slow response times during the pre-exam steps. Chromebook users are experiencing an outage.

We are seeing improvements across the system. We will continue to monitor it closely and will provide details later about our investigation.

We are continuing to investigate this issue.

All systems are operational and working properly. We will provide details at a later time about the investigation.

This incident has been resolved.

Root Cause Analysis for Service Slowdown and Interruption March 5, 2025 We sincerely apologize for the service issues on March 5th that affected some of our applications. We understand how much your institution relies on our services, and we regret any inconvenience this may have caused. Below, we’d like to provide details on what happened and the steps we are taking to prevent future occurrences. At approximately 10:00am PST on March 5, 2025, we started receiving alarms indicating a rapid increase in the failure rate of students starting and completing the pre-exam steps in Respondus Monitor. However, everything appeared normal in the application servers in terms of processing requests, server load, memory usage, and thread and connection counts. Requests were serviced without errors and within our maximum target of 250ms response time. But while all health checks within the AWS environment indicated the servers were healthy and operating normally, we began receiving alerts from monitoring services that run external to AWS showing elevated response times and intermittent failures. We then focused on the AWS load balancer service \(ALB\) which distributes incoming traffic to the application servers. This service is fully managed by AWS, and hence, we don’t have good insight into the health of the appliance, nor can we restart the nodes, etc. After examining the access logs for the load balancer, we saw a very high number of Error 460 entries, which according to the AWS documentation means: "Client errors are generated when requests are malformed or incomplete. These requests were not received by the target, other than in the case where the load balancer returns an HTTP 460 error code. This count does not include any response codes generated by the targets." Our initial thought was that a DDoS attack was flooding the load balancer with malformed packets. But this was quickly ruled out because the requests associated with the errors looked normal in the access logs. Additionally, our Global Accelerator endpoints are protected by AWS Shield, which should stop network layer attacks before they reach the load balancer. However, this explained why the application servers appeared to be operating normally – because a large percentage of requests were not reaching them. Engineers at AWS said the 460 errors would also occur if the client \(LockDown Browser\) closes the connection before the load balancer sends the response. This seemed implausible because there hadn’t been recent updates to the client applications. Moreover, such an issue would emerge gradually – over days or weeks – based on how we introduce new releases. This event escalated in minutes. Concurrently with our investigation, we performed two rolling restarts of the application servers which didn't result in much improvement. Given these symptoms, we suspected the issue might be with the load balancer service itself. We decided to terminate all application servers at once \(to fully scale down the load balancer\) and then launch new application servers \(to scale up the load balancer, but on different nodes\). This had the immediate effect of restoring the service, and our initial theory was that the load balancer had gotten into a bad state. After further investigation, however, we determined the problem wasn’t with the load balancer. The root cause was that a bandwidth limit had been reached on the elastic network interfaces attached to the application server instances, resulting in a bottleneck that continued to grow in the early stages as failed requests were retried. This service event primarily entailed slowdowns in request responses and intermittent failures. Once students entered an exam, this event would not have affected them until after the exam was submitted on the learning system. At that point, students may have experienced delays or failures as they attempted to exit the Respondus Monitor system. The only time there was a complete outage was when everything was shut down for a few minutes to restart the entire service. We have since performed a detailed analysis of the event and have configured new alarms that trigger auto-scaling of application servers before the network bandwidth limit is reached. We have also increased the minimum number of application servers which will smooth the opening minutes of a massive autoscaling event. Finally, we also want to note that the StudyMate Campus service was similarly affected during this event, as were users trying to start exams using the Chromebook version of LockDown Browser. In the latter case, the impact was due to how the Chromebook extension retrieves settings at startup.

We are investigating an issue where some Respondus Monitor users are unable to complete the steps that occur prior to the start of an exam.

We are continuing to investigate this issue.

The service is now operating normally. The earlier slowdown is being investigated.

On October 16, 17 and 18th there were single incidents that resulted in a disruption of the Respondus Monitor service lasting approximately 2 minutes. Application servers failed a health check, and were automatically replaced by new ones to resolve the issue. We are investigating the root cause of the health check failure. During these brief disruptions, test takers who were underway with an exam were unaffected. It primarily affected students in the pre-exam and post-exam stages by presenting a “gateway error” message. We are investigating the issue and will update this notice once the problem has been resolved.

The source of the problem has been identified and fixed. We do not anticipate further issues relating to it.

The StudyMate Campus service is currently experiencing an outage that we are investigating. This is not related to LockDown Browser or Respondus Monitor.

We are continuing to investigate this issue.

The StudyMate Campus service has been restored. We are investigating the cause of the brief outage.

The Respondus Monitor LiveChat service is experiencing an outage. This is not affecting the Respondus Monitor application itself, only the LiveChat help system built into it. The underlying LiveChat technology is from a third party; status and updates on this issue can be followed here: https://status.livechat.com/.

Resolved: The Respondus Monitor LiveChat service is functioning normal now.

The Respondus Monitor LiveChat service is experiencing an outage. This is not affecting the Respondus Monitor application itself, only the LiveChat help system built into it. The underlying LiveChat technology is from a third party; status and updates on this issue can be followed here: https://status.livechat.com/.

The Respondus Monitor LiveChat service is functioning normal now.

The Respondus Monitor LiveChat service is experiencing intermittent issues. This is not affecting the Respondus Monitor application itself, only the LiveChat help system built into it. The underlying LiveChat technology is from a third party; status and updates on this issue can be followed here: https://status.livechat.com/.

The Respondus Monitor LiveChat service is functioning normal now.

We are currently investigating an issue with users being unable to start a Respondus Monitor session.

Amazon identified a problem with the Amazon Cognito Identity Pools in the US-EAST-1 Region. The AWS issue appears to be resolved and the service is returning to normal.

On Tuesday, June 13 2023, at approximately 12:00 pm PST (19:00 GMT Tuesday), AWS began reporting degradation across several services, including access to the AWS Management Console. We don’t currently have reports of issues with LockDown Browser or Respondus Monitor, but are carefully watching the situation.

The issues reported by AWS have been resolved. There was no impact to LockDown Browser or Respondus Monitor users during this time.

The Respondus Monitor service is currently experiencing some issues that we are investigating.

The problem appears to have been resolved and service is returning to normal.

The problem affecting Respondus Monitor is resolved. We continue to investigate the source of the problem which affected the start of new Respondus Monitor sessions for approximately 10 minutes.

We are currently investigating an issue with users being unable to start a Respondus Monitor session.

There is a widespread AWS outage that is currently affecting Respondus Monitor - https://health.aws.amazon.com/health/status We are monitoring the situation closely.

The AWS problems have resolved and operations are back to normal.

We are currently investigating an issue with the latest update of Blackboard Ultra.

The issue has been identified and Blackboard is rolling out a fix to the affected sites.

Blackboard has provided a fix for all affected Blackboard Ultra production instances and will be updating customer test/staging servers on Friday 4/22.