Waterford outage history

We are currently experiencing a delay in the processing of student data due to recent events in Amazon Web Services (AWS). As a result, some dashboards and reports may temporarily display outdated information. Our team is monitoring the situation and will provide updates as data processing returns to normal.

The issue has been identified and a fix is being implemented.

We are continuing to work on a fix for this issue data may remain out of date for an additional 12-24 hours.

This incident has been resolved.

Some users have difficulty signing in to Waterford and receiving timeouts or error pages.

We are currently investigating this issue.

The issue has been identified, and a fix is currently in progress

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

During the upstart enrollment process, users may be unable to complete the enrollment form due to the zipcode field lookup and validation timing out. We have contacted our geocoding partner responsible for this service and are awaiting a resolution.

A fix has been identified, and a resolution is in progress.

A fix has been implemented and the enrollment form is now functioning correctly.

This incident has been resolved.

We have identified longer than usual wait times when launching and playing activities and are actively investigating the issue.

We have identified the root cause and are working on a mitigation strategy.

The issue has been identified and a fix is being implemented.

Additional mitigation steps have been taken, and wait times are back to normal levels.

The issue has been resolved however, it impacted our data replication processes reports, dashboards and upstart assignments may be delayed as jobs resume normal operation.

We are currently experiencing elevated error rates with our hosting provider. A root cause has been identified, and mitigation steps are currently in progress.

Mitigation efforts are continuing to progress. Elevated error rates are beginning to decline.

Mitigation efforts have completed services should be fully operational

This incident has been resolved.

Some users may experience long load times and timeouts on the dashboards, staff and students, and reporting tabs. The team is aware of this issue and is actively investigating.

We are continuing to investigate this issue.

The team has identified a solution and is working to restore services.

Restoration of services is currently in progress

Services have been restored

This incident has been resolved.

We see an increase in timeouts and 500 errors when launching student sessions from my.waterford.org and launching the manager tile from teacher.waterford.org. The issue is under investigation.

The root cause has been identified. We are working to restore services.

The root cause has been addressed, and affected functionality has been restored.

This issue has been resolved

Currently, in teacher.waterford.org, users may experience issues loading the dashboard tab, staff and students tab, courses and goals, or reports tab when searching at the student level.

The source of the issue has been identified, remediation steps are currently in progress estimated time to completion is approx 60 mins.

Remediation is still in process, taking longer than initially estimated.

The issue has been resolved, and mitigation steps have been completed.

Our hosting partner is currently experiencing a service degradation affecting Waterford Reading Academy and Waterford Early Learning this has resulted in longer than usual page load times and 504 Gateway Timeout errors. We are working with our partner to address these issues and will provide additional updates as more information becomes available.

Our hosting partner has identified root cause and is working toward a solution, more information about this incident and current status of our hosting partner check https://status.aws.amazon.com/.

We are continuing to work with our hosting partner to restore performance and services critical to the authentication and operation of Waterford Reading Academy student and teacher applications. Some applications have been restored and are now operational updates will continue until all have been fully restored. Ongoing work and the most up-to-date progress can also be found at https://status.aws.amazon.com/

Degraded performance is improving some residual slowness and erroring may still occur. Users should be able to Authenticate and use Waterford Reading Academy student, teacher, Waterford Mentor, and Upstart applications with minimal wait time. We will continue to monitor for optimal performance and outstanding issues.

All applications are functioning normally.

Some students and teachers are experiencing 500 errors while attempting to authenticate we are currently investigating cause.

We are continuing to investigate this issue.

Issue has been resolved.

This incident has been resolved.

We are aware of the issue and currently investigating root cause, this issue affects users directly logging into Waterford Early Learning manager and users attempting to open Waterford Early Learning manager from Waterford Reading Academy Teacher application.

The issue has been identified and resolved

For a brief period some students experienced error message "Unable to locate media" when attempting to launch sequenced curriculum in Waterford Reading Academy.

Currently one of our partners utilized to deliver Waterford Mentor is experiencing elevated error rates affecting users ability to sign-in to the product.

Partner has been notified and is investigating underlying cause, currently awaiting response.

Services are functioning again but may still be in a degraded state, continuing to monitor issue until our partner communicates full resolution

We are continuing to monitor for any further issues.

Partner has confirmed resolution of this issue Waterford Mentor products are now operational.

Some students while attempting to launch reading or math from Waterford Reading Academy are redirected to Waterford Early Learning Manager login page.

Some users experiencing 500 errors while attempting to login to my.waterford.org Some users experiencing 500 errors while attempting to login to teacher.waterford.org

Root cause has been identified and addressed further mitigation steps are in progress.

Waterford engineers are currently investigating this issue and will post updates soon.

We have identified the cause, and are currently monitoring key performance indicators.

Monitoring of performance indicators have proven that maintenance steps taken have been successful and issue is resolved.

A database update completed late combined with some issues with Amazon's load balancers during a production update caused some services to fail. This has been resolved by redeploying the production and database updates. We apologize for any inconvenience this may have caused. The issue occurred between 11:00pm MDT through 7:00am MDT.