Cronologia delle interruzioni di Csiu

This morning, the CSIU FIS software is currently experiencing issues when trying to produce reports in the FIS system. Our team is currently working to diagnose and resolve the issue with the highest priority. We apologize for this inconvenience. Thank you for your patience and understanding.

The issue has been identified and a fix is being implemented.

We have identified the issue and have implemented the remediation.

FIS reporting services for clients have been fully restored.

We are aware of an issue that affected some clients using the eSchoolData Student Information System that resulted in a brief outage this morning. We have investigated and believe that we have identified the source of the issue and have implemented a resolution. We consulted with the hardware vendor and rolled back an upgrade that was applied yesterday evening. We are working with the vendor to identify the cause of the issue, as it should not have resulted in any disruption. We apologize for the inconvenience.

We are aware of an issue affecting some clients using the eSchoolData's Student Information System. We have investigated, and believe that we have identified the source of the issue. We are working to implement a resolution as quickly as possible.

We continue to investigate the issues that some clients are having with the Student Information System.

We are actively troubleshooting an issue affecting the eSchoolData Student Information System. Some clients are reporting login and slowness issues. We are working to resolve the problem as quickly as possible. You may experience connectivity issues this afternoon as we work through the problem. We are sincerely sorry for the inconvenience. We will update you when we have more information.

We are currently monitoring the Student Information System following today's incident. We believe we have isolated the issue, but will leave this incident active until further notice.

We have been actively monitoring the system and have not received any further reports of issues since the afternoon of December 19. We are sincerely sorry for the disruption that you experienced Monday.

The CSIU is aware of an incident that is preventing certain clients from accessing their CSIU-hosted systems. We are working to investigate and resolve the issue as quickly as possible.

The incident has been identified and resolved. Access to all affected CSIU-hosted systems has been restored. We apologize for any and all inconveniences the service disruption caused.

A necessary hardware reboot is required that will take down some services briefly. We anticipate the outage will be short, and apologize for any inconvenience experienced.

This incident has been resolved.

There is currently an issue with our call-in system. Callers receive a message saying that we are closed. WE ARE NOT CLOSED. Some clients have been able to get through, but it is inconsistent. If you are unable to get through, please email [email protected] or put a ticket in using the Online Support Center.

The issues experienced with our call-in system yesterday have been resolved. Calls are routing properly this morning.

Code for a future release was inadvertently released that will prevent you from being able to submit a support request through the FIS system. This issue is being addressed and we anticipate that it will be fixed overnight. In the meantime, if you need to submit a support request, please email [email protected] or call the support number. Thank you for your patience as we work towards a resolution.

The support request page has been fixed. Thank you for your patience while the issue was resolved.

Please be advised we have discovered an issue between us and one of our ISPs beginning at approximately 10:09 AM that is causing some clients to lose connection to our hosted services. We have a ticket open with our ISP and hope to get more details on what is causing this as soon as possible.

We apologize for the length of time it has taken to resolve this issue. Our ISP has finally determined that our websites have been targeted by a Distributed Denial-of-Service (DDoS) attack. A DDoS attack event does NOT compromise data. The intent of a DDoS attack is to disrupt Internet Service. We have been working all day with our ISP as they implement potential fixes to address this issue. We apologize for the inconvenience this has caused, and assure you that we are working around the clock with them to restore your services.

A fix has been implemented by our ISP and we are monitoring the results. The results are promising at this point.

The fix that was implemented resolved the initial issues, but we are again seeing issues with connectivity from one of our ISPs.

The attackers pivoted and the ISP had to re-evaluate their fix. We are once again back to a working state.

We are aware of recurring connectivity issues to our hosted services. We are working with our one ISP to investigate.

We are aware of recurring connectivity issues to our hosted services. We are working with our one ISP to investigate.

We continue to dialog with our ISP as they work to keep this attack at bay. They have once again stopped the attack and services have been restored again. We will leave this incident in a monitoring status until we are confident this issue is no longer recurring.

We have been alerted once again about connectivity issues occurring. We are already in contact with our ISP to investigate. We will update as soon as we have more details.

Connectivity to services has been restored. We continue to work with our ISP to investigate this attack. We will leave this incident in a monitoring status until we are confident this issue is no longer recurring.

Connectivity issues are occurring again. We remain in contact with our ISP, investigating this attack. We will update as soon as we have more details.

Connectivity to services has been restored. We continue to work with our ISP to investigate these attacks. We will leave this incident in a monitoring status until we are confident this issue is no longer recurring.

Connectivity issues are occurring again. We are actively in contact with our ISP, investigating these attacks. We will update as soon as we have more details.

Connectivity to services has been restored. We continue to actively work with our ISP to investigate these attacks. We will leave this incident in a monitoring status until we are confident this issue is no longer recurring.

Connectivity issues are occurring again. We are in contact with our ISP, who are investigating three active current attacks. We will update as soon as we have more details.

We continue to work with our ISP to put steps in place that have lessened the impacts of the attacks, but they continue as multiple coordinated attacks originating from thousands of locations around the world. The attacks have reached speeds of up to 20Gbps when our bandwidth is considerably less. We continue to work with our provider for short term, as well as long term solutions and have filed an incident report with the FBI.

The DDoS attack mitigation efforts put in place by our ISP appear to have worked well today. We did not receive any new reports of connectivity issues. We continue to work with our ISP on implementing additional mitigation strategies, and we continue to monitor for any signs of malicious activity. We again apologize for the inconvenience this is causing and appreciate your understanding.

We have seen no additional recurrences since our last update. We will continue to monitor ongoing for changes and alert you as necessary.

We have had no issues affecting connectivity to our systems today. We continue to monitor for any new problems. We are engaged in ongoing discussions with the FBI and our ISP regarding this incident, and we are following up on additional short- and long-term mitigation efforts. Thank you for your continued support, patience, and understanding.

There have been no new connectivity issues reported for several days, so we are closing the incident. We will continue to monitor for any signs of recurrence, and will open a new incident and keep you informed if any new issues arise.

As we shared in February, we experienced a denial-of-service attack \(DDoS\) that resulted in a temporary outage of some services. We sincerely regret the difficulty and inconvenience it caused. ‌ While frustrating, DDoS attacks – including the one we experienced – do not involve improper access to data. These types of attacks flood networks and servers with an excessive amount of anonymous traffic, causing disruptions and outages. We want to confirm for you that this incident did not result in a breach of our systems nor any unauthorized third-party access to data.  ‌ We know this incident raised questions and, as a valued partner, we want you to know that we took immediate action, alerted authorities, and enlisted cybersecurity experts who also confirmed that network security was not impacted due to this incident. In addition, we worked with our local service provider who has since improved their DDoS mitigation device and we have increased network capacity to help defend against similar attacks in the future. ‌ Thank you for your patience and understanding throughout this incident. We understand the trust you place in our systems, and frustrations when they are inaccessible.   Thanks for being a part of the CSIU software family!

We are aware of an issue affecting some Student Information System users. We are now investigating this issue and will provide a status update as soon as we are able to determine a cause.

A fix has been implemented to migrate the affected resources to redundant hardware and we are monitoring the results. Some underlying issues with a host server have been encountered and we have a case open with the vendor to get to the bottom of it. We will update when we feel confident it is fully resolved.

We are aware that the issue has occurred again and are investigating. We will provide an update as soon as we are able to determine the cause.

We are back to a operational state for all customers. We apologize for the inconvenience as we continue to investigate the underlying cause. More information will be provided as we are able.

We have continued to operate with no further disruptions over the past three days. We continue to dialog with our vendors to get a final resolution and will update once more when we feel confident this is fully resolved.

This incident has been resolved.

The issue was triggered by a problem with the server replication product used by the CSIU, which caused one of the SQL Servers to become unresponsive.  After detailed consultation with our hardware and software vendors, we feel confident that the conditions that caused this issue have been resolved. Thanks for your patience while we resolved this issue.

We are aware of an issue affecting Student Information System applications. We are now investigating this issue and will provide a status update as soon as we are able to determine a cause.

We are continuing to investigate this issue.

A fix has been implemented and we are monitoring the results.

This incident has been resolved. We discovered an issue with our underlying infrastructure that was causing the performance issues. This has been addressed, and as always we will continue to keep a close watch and react as needed if future issues arise. We apologize for any disruption or inconvenience this has caused.

Several clients have reported a problem loading Teacher Connect this morning. We are investigating the problem and will provide an update when we have more information. Thank you for your understanding.

We believe we have identified and fixed the issue. We are in the process of verifying with the affected clients.

This issue has been resolved. We are sorry for any inconvenience this may have caused you. Thank you for your understanding.

We pushed out a new deploy method and it unfortunately resulted in a 401 unauthorized error for agencies using single sign on. We have this resolved and will be investigating the cause of this to ensure it won't happen again in our next release. The timing that could be reported for this event would have begun around 7:30PM on 9/11/2020 and could have been experienced until approximately 9:03PM on 9/12/2020. Please accept our sincere apologies for any disruption that may have been experienced and thank you for your understanding.

CSIU experienced a power outage that resulted in a short outage from 4:19PM to 4:35PM. We apologize for any inconvenience this may have caused.

We are aware of an issue affecting the FoxPro Employee Portal on eService. We are now investigating this issue and will provide a status update as soon as we are able to determine a cause.

A fix has been implemented and we are monitoring the results.

We are continuing to monitor for any further issues.

This incident has been resolved.

We are currently working to resolve an issue that could prevent some users from connecting to the CSIU FIS Software. We are working on a solution, and we will update you when we have the issue resolved. Thank you for your patience. CSIU Computer Services

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

We are experiencing some issues that are affecting network services and application availability. We are investigating the issue and will follow up as we have more information.

All services are back online and we are working with Dell EMC to do a root cause analysis of the issue. We apologize for any disruption you may have experienced this morning. Thank you.

This incident has been resolved.

Some clients have been reporting timeout and/or extreme slowness issues when trying to access the FIS, SIS, or FoxPro applications. The problem appears to be outside of our network and not inside of PAIUNet. We are working with our Internet Service Provider, PenTeleData, to identify the source of the disruption. We will notify you when we have more information.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

Clients are reporting extreme slowness in the SIS. We have been working closely with eSchoolData technicians to determine the cause. A patch was applied yesterday, which unfortunately did not resolve the problem. We will continue to work together until the problem is resolved. We understand your frustration and sincerely apologize for the inconvenience this situation is creating for you and your district. We will update you again as soon as we have more information.

We are continuing to work on a fix for this issue.

As part of our troubleshooting efforts, the SIS will be unavailable for approximately 5-10 minutes beginning at 2:10 today.

We continue working with the team from eSchoolData and engaging Microsoft consultants in the discussions regarding the SIS slowness. Although some clients have reported a slight improvement in performance after adjustments were made, the speed is not back to acceptable levels. The data logged in the trace process did not definitively identify the bottleneck, but it did point to a potential issue that we are investigating. We will update you again as soon as we have more information.

We continue working with the team from eSchoolData, Microsoft consultants and Dell in the discussions regarding the SIS slowness. The recommended adjustments from last night have not resolved the issue, but the system has shown a slight increase in speed. We have identified where the bottleneck is occurring and continue collaborating with consultants on the cause and optimal solution. We will continue updating you as we have more information.

We have been troubleshooting our slowness issue extensively over the past four days in an attempt to narrow down the potential underlying issue. It has proven to be quite elusive to our staff, eSchoolData and a number of consultants. Our Dell EMC consultant believes with high confidence that once the modifications they have been administering throughout the day are complete, they should resolve the speed issues you have experienced. In addition, we are still actively pursuing all other possibilities with our Microsoft consultant as well. They will continue to work with our staff throughout the evening. We understand your frustration and sincerely apologize for the problems and issues this situation is creating for you. We assure you that we are doing everything we can to resolve this issue. We will continue updating you as we have more information.

Update on slowness in the SIS The Dell EMC modifications addressed in the previous email finished at approximately 3:45 p.m. with very encouraging results. An additional optimization is now underway and expected to finish this evening. We encourage you to give us your feedback in the morning. Investigations with any potential Microsoft database or server related issues continues as well. Staff are now working with engineers on additional reviews and possible recommendations. Thank you for your feedback and patience while we work through this unfortunate situation. We will continue updating you as we have more information.

Update on slowness in the SIS The changes made to our infrastructure and the eSchoolData software appear to have made a vast improvement today. Staff have been monitoring the systems and short of positive feedback from clients today, we have not seen or had any reports regarding speed. We encourage your district to continue using the SIS today and let us know if you experience any issues. Regardless of the current state of the overall system and findings, our staff and consultants will continue monitoring and scrutinizing any other possible configurations, settings and indexing that can only add to the overall performance. Implementation of some of these resolutions are happening now, while others will happen this evening and/or over the weekend. We are once again sorry for failing you at this most vital time of the year. We will continue updating you as we have more information.

Update on slowness in the SIS Although the changes made to our environment made improvements, we have noticed continued performance issues at peak times. We continue working with Microsoft, Dell and eSchoolData on troubleshooting. Additional tools have identified potential areas for software optimization. eSchoolData is actively addressing these areas, and plans to provide a software update for us today. We will continue updating you as we have more information.

Update on slowness in the SIS The optimization changes that were applied to the software last night did make a slight difference today, especially to a certain segment of the clients. However, the majority of you still experienced performance issues during the peak time this morning. Using more collected data, constant monitoring, and further advice from Microsoft consultants, eSchoolData has been able to focus on their central database for clues. Around noon time today, we applied a hot-fix from eSchoolData to create more indexes and optimizations to the central database with good results. Right now, CSIU staff and eSchoolData are building an additional server which will exclusively host the central database. When we know the timeframe the central database will be moved to this new server, we will send out an announcement on the https://csiu.statuspage.io/ about the window of downtime. It should not take more than two hours. We will continue updating you as we have more information.

We have completed the additional server build for the central database. We plan to complete this migration starting at 10:00 PM tonight, and anticipate this being completed by midnight. Services will be disrupted during this maintenance window. Thanks for your understanding and patience.

The database migration to the new server is completed. All services should now be resumed. More updates will be coming as we have further information.

This incident has been resolved.

We are aware of a sporadic connection issue affecting clients connecting through our PenTeleData ISP. We have an open ticket with them and they are now investigating this issue. We will provide a status update as soon as we are able.

We have seen no sporadic connection issues in more than an hour. We are moving this to monitoring status.

There have been no further reports since Friday night. We are closing this incident.

Connections to our 204.186.159.52 server are not getting through. There was a service problem on the server. This is being rebooted now and should be back up momentarily.

The server problem has been resolved and the server should now be accessible again.