Störungsverlauf von Veracross

We have detected an outage for Documents in the US.

We are continuing to work on a fix for this issue.

This incident has been resolved.

Between Friday 12 June at 21:00 UTC and Saturday 13 June at 14:00 UTC EMEA and APAC users experienced errors when accessing Portals. We identified a misconfiguration in the backend and resolved the issue.

Syncs for our Ravenna Integration were failing due to the same issue impacting our Data Importer. That issue has been resolved and Ravenna syncs should once again complete successfully.

We are currently investigating reports that Data Importer imports are stuck in pending status.

We have resolved the issue with the Data Importer that caused imports to be stuck in a pending status. To prevent inadvertent duplicate record imports in scenarios where users may have attempted the same import multiple times, imports that were stuck in pending status have been removed. Users should complete a new import process for any prior attempts that had been stuck.

On May 2, some users on US Platform Groups experienced intermittent errors when logging in due to issues in a portion of the authentication service. Users already logged in were not impacted. The issue has been resolved, and preventative improvements are underway.

We are currently investigating reports of degraded performance in Axiom for a subset of users.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

We are currently investigating an issue impacting Composer for a subset of users.

We are continuing to investigate this issue.

A fix has been deployed and Portals performance has stabilized. We are still closely monitoring Composer performance.

After monitoring for several hours, we can confirm that all systems are now operating normally. Between 9:35 AM ET and 11:10 AM ET, some users experienced degraded performance in Composer and Portals during a period of high usage. Our systems are designed to automatically add capacity when usage increases. However, when this process ran today, some of the additional capacity was not configured correctly, which limited our ability to absorb the increased demand. Once identified, we corrected the configuration and performance returned to normal. We apologize for the disruption and appreciate your patience.

Between 3:35 PM ET and 3:55 PM ET, a subset of users experienced errors when accessing Portals in the US. We identified an issue with a software update being performed and resolved the issue.

We are currently investigating an issue impacting the accessibility of Portals for a subset of users.

We have identified this issue and have updated the scope to reflect the impact on multiple Veracross platforms.

We are continuing to work on a fix for this issue.

We continue to work to address this issue. Intermittent access issues for schools on Platform Group US 30 are expected over the next 15 minutes. We apologize for this disruption and thank you for your continued patience.

We are continuing to monitor system health closely. Some users are still experiencing degraded performance. We we will be performing additional maintenance work to address this issue tonight. School contacts at impacted schools have been notified directly via email, and an in-app notification has been posted for users at those schools.

A subset of users on Platform Group US 30 are continuing to experience degraded performance across multiple platforms this morning despite the additional maintenance work we performed last evening. We are continuing to work to address these issues and will provide additional updates.

We will be performing additional emergency maintenance on Platform Group US 30 to continue mitigation efforts related to the ongoing performance issues. Learn more about determining your Platform Group here: https://community.veracross.com/s/article/What-is-My-Schools-Platform-Group This maintenance is scheduled for today from 12:00 PM to 12:20 PM ET. During this time, all Veracross services for Platform Group US 30 will be unavailable. We recognize the impact of this disruption and have determined this maintenance is necessary to support ongoing stabilization efforts. An in-app notification will be displayed to all impacted users in advance of the maintenance window. Our team will continue to monitor performance closely following the maintenance and will provide further updates as more information becomes available. We sincerely apologize for the continued disruption and appreciate your patience as we work to restore full service.

The emergency maintenance for Platform Group US 30 has been successfully completed. Initial results show meaningful improvements, but we continue to closely monitor performance.

After monitoring performance since the emergency maintenance was completed this afternoon, we can confirm that system performance has stabilized. Additionally, we sent an email update to designated school contacts at each school to provide a more comprehensive update regarding this incident.

We are investigating an issue with downloading the PDFs for checks in AP check batches.

This incident has been resolved.

Our Veracross Community AI Service Assistant is currently displaying an error message. We are investigating the issue in order to restore the chatbot feature as quickly as possible. In the meantime, you can still use normal Community search functions to find knowledge articles, and can still contact our support team through the usual support case entry form.

This incident has been resolved.

We have received reports of degraded system performance and database connection errors impacting a subset of users. We are continuing to investigate further.

We have identified intermittent performance issues caused by increased load on certain backend systems. The impact is limited to a subset of users in North America, while others may not experience any issues. Our engineering teams are actively working to optimize performance. For additional context, please refer to the email sent to school point people on Tuesday, February 3 (ET).

We have resolved the intermittent availability issues some users experienced earlier in the week. Please refer to the forthcoming email to school point people for more details about specific work done to resolve this issue. Thank you for your patience.

We are investigating an issue that is impacting Composer, Online (Re-)Enrollment, Program Registration, and Admissions 2.0 for a subset of users.

Composer performance has returned to normal.

This issue has been resolved, but we continue to closely monitor system performance.

We are currently investigating an issue impacting the access and submission of Behavior & Comments records and approval of Student Logistics Requests for a subset of users.

The issue impacting Behavior & Comments records has been fully resolved. After further investigation, we determined that Student Logistics Requests were not impacted by this issue and have updated the title of this incident accordingly.

We have identified an issue impacting the display of person record details screens. We are currently investigating this issue.

This incident has been resolved.

We are currently investigating this issue.

Unfortunately, workspace-related functionality is now disabled again due to performance issues. Workspaces will not show in the workspace picker or on homepages. Additionally, new queries cannot be saved. However, existing queries are available and can be updated. We continue to make restoring functionality our top priority and will provide another update by 5 pm Eastern (22:00 UTC).

We have implemented additional improvements, and all query functionality has now been restored, including the ability to create and save new queries. While the main Workspaces page is visible again, accessing individual workspaces will still result in an error message at this time. We will provide an update at noon ET (17:00 UTC) tomorrow.

We continue to work on restoring access to individual workspaces. Query functionality remains fully available. We will provide another update by 5 pm Eastern (22:00 UTC).

We are continuing to work on restoring access to individual workspaces while ensuring system stability. We will provide another update by EOD tomorrow.

As communicated to school point people in today’s email from our CTO, the scope of this issue is now limited to workspace functionality. Further updates about workspace availability will be published directly within Axiom, providing greater visibility for impacted users. Please refer to our CTO update for more details, and thank you for your patience as we’ve worked to address this issue.

We will be performing emergency maintenance on a core database system beginning at 4:30 PM ET today. We expect this maintenance to last approximately 15 minutes. During this time, some applications and services will be unavailable for all customers. We apologize for the inconvenience and will be providing additional updates on restoring Axiom shortly.

This incident has been resolved.

We are currently investigating this issue.

The issue has been identified and a fix is being implemented.

In order to restore Axiom as soon as possible, we have temporarily made workspaces unavailable. Workspaces will not show in the workspace picker or on homepages. Additionally new queries are unable to be saved, and existing queries are available but cannot be updated or deleted at this time. We are continuing work to restore full functionality for workspaces and queries, and will provide further updates as soon as possible.

We are aware of continued performance issues and are actively working toward a resolution.

We are performing emergency maintenance at 4:30 PM ET (21:30 UTC) to help address the underlying issue impacting Axiom performance. This will result in a brief period of downtime (approximately 15 minutes) for applications impacted by this issue: Axiom, Data Export Packages, and the Data API.

Axiom performance is stable following the emergency maintenance and temporary workspace and query restrictions that have been put in place. These temporary restrictions will remain in place while we work toward a full resolution for this issue. Please refer to the email communication sent to school point people for more details about this incident.

Axiom Workspace and query saving restrictions continue to be in place. Bringing these back to normal is our top priority, though we do not have an ETA for resolution yet. Another update will be provided by 5:00pm Eastern. If you need help finding queries that are saved in your workspaces, please open a ticket and we will be happy to assist you.

We have identified the problematic area of the system, but have not yet identified a solution. This continues to be our top priority, but at this time we do not yet have an ETA for resolution. The next update will be provided at noon ET (17:00 UTC) tomorrow.

We continue to work on restoring Workspace related functionality as our top priority. We have restored the ability to delete and overwrite queries and charts; however, saving new queries or charts is still disabled. Workspaces pinned to homepages have been partially restored; clicking on them will show their queries but not nested workspaces. Finally, navigating to Workspaces is still disabled. This is our current focus. The next update will be no later than 5pm Eastern (22:00 UTC).

We have rolled out an update to the part of our back-end system that we identified as the cause of the issues. These updates have yielded noticeable performance improvements, and as a result we have restored all functionality. The system continues to be stable after restoring all functionality, but we will be monitoring performance closely tonight and tomorrow to confirm the issue is fully resolved.

We have been carefully monitoring the system today and it has remained stable with the improvements made this week, and are therefore marking this issue as resolved. The issue stemmed from a problem in loading Workspaces for users, which show up in a few places (such as the Workspaces tab, search results, and when saving a query). Because this is a frequent operation in the system, the issue had the side effect of impacting general Axiom performance. We disabled this feature in order to keep the rest of the system available, deployed several improvements, and then re-enabled it, resulting in a stable experience.

We are currently investigating reports of authentication issues that are impacting a subset of users.

An upgrade this morning to our authentication system this morning resulted in users in some contexts being unable to log in to Veracross. Users previously logged in were not impacted. We have fixed the issue and users should be able to log in to Veracross again. We are monitoring this issue.

After monitoring this issue closely, we have confirmed that this incident has been resolved. We apologize for the inconvenience this issue has caused.

We are currently observing elevated errors from BlueSnap impacting payout report retrieval and causing receipt failures across our systems. BlueSnap previously reported that a recent infrastructure-related outage had caused sporadic connectivity failures. We are monitoring the situation closely and validating the impact on our platform. Further updates will be provided as soon as more information becomes available.

This incident has been resolved. Please note that any payments that failed will not be automatically retried/reprocessed and will need to be resubmitted by the payer. We apologize for this disruption and thank you for your patience.