19 Vanco incidents · May 2020 — official updates, affected components, duration and resolution details.
CXP has started receiving 502 errors from WorldPay while attempting to process transactions starting at roughly 8:00 am this morning.We are investigating. WorldPay has just confirmed it is an issue on their side starting at 7:57 am
Worldpay appears to be processing transactions again, but we are continuing to monitor the situation.
This incident has been resolved
We are currently investigating this issue.
The issue has been identified and a fix is being implemented.
Security and DevOps is applying an update that to switch from a hotfix had earlier to a possible permanent fix. We continue to monitor the change, and if errors continue to be detected, will put the"hotfix" back.
We see outage on the Payments for EDU. We are currently working on identifying the issue
We are continuing to investigate this issue.
The issue has been identified as a known issue from the processor side, WorldPay. We will provide update as soon as we receive new information from WorldPay. No ETA has been provided at the moment.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We are currently investigating this issue.
We are continuing to investigate this issue.
The issue has been identified and a fix is being implemented.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We are currently investigating this issue.
Our SC/TE deva nd system teams are continuing to investigate CPU utilization issue. Fixing this issue is our top priority. We continue to work as quickly as possible to resolve the issue. Thank you for your patience.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We are currently investigating the issue
We are continuing our investigation on the issue
As we are working on investigating the issue, we are currently rolling back the change from RHEL7 to CentOS6 as well as fixing today's ACH file. We will continue our investigation with the PMX - RHEL7 application, and we will communicate new date for the redeploying the PMX RHEL7 back to production.
Team is still working on fixing the 25 March ACH file. 26 March processing is currently looking normal.
This incident has been resolved.
Our ConnexPoint team is continuing to investigate login issue. Fixing this issue is our top priority. We continue to work as quickly as possible to restore login issue. Watch for our next update in approximately 30 mins. Thank you for your patience.
The issue has been identified, and ConnexPoint and DevOps teams are working on the fix
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
PMX Payment availability is currently delayed as payments have not been released yet. We are currently investigating a solution and an ETA for releasing payments.
We are continuing to investigate this issue.
The issue has been identified and corrected, payments in the process of being released.
Payments are being released, this is resolved.
We’ve discovered some Service Center and GivePlus users are experiencing connectivity issues due to 3rd party system we are using, Imperva for our Web Application Firewall. We are currently looking into this and monitor Imperva status page. We will post an update as soon as we have more information. We apologize for any inconvenience.
Good news! 3rd party system, Imperva implemented a fix and Service Center and GivePlus are operational. We are continuing to monitor this situation and will post another update after we’ve confirmed that everything is fully resolved. If you continue to experience problems, please open a ticket with our support team.
Update - Customer experience varies and GivePlus users are still experiencing partial outage
The issue has been identified and a fix is being implemented.
3rd party system, Imperva is continuing working on resolving the issue. Service Center and GivePlus users are still experiencing intermittent issue accessing the systems.
Imperva continues to work on the fix and last update on their status page is : "A subset of customer sites continue to experience DNS failures based on varying expiration of DNS record time-to-live (TTL) values in resolver caches. Imperva is contacting ISPs to request that they flush their DNS cache, which should resolve these DNS failures. Customers may achieve a faster resolution by also contacting their ISP directly. If the issues persist after your ISP has been contacted, please reach out to Imperva Support at [email protected] for additional assistance. We are committed to a full resolution for this issue and further updates will be shared via this status page."
Imperva's latest update stated that they have implemented a fix, and their monitoring indicates issue has been resolved. They will closely monitor stability.
The issues we experienced last night that caused partial outage to several of our Faith systems are fully resolved. Out 3rd party system has confirmed the resolution and resolved the issue on their side. We apologize for any inconvenience and thank you for your patience as we worked through this issue.
We are currently investigating a partial outage affecting Data API within Connexpoint. Some portion of payments and other calls are failing.
Rolled back today's data api deploy and we are monitoring results. So far it seems to be positive.
This incident has been resolved.
Our Transaction Engine team is continuing to investigate outage issue. Fixing this issue is our top priority. We continue to work as quickly as possible to fix the issue. Watch for our next update in approximately 30 mins. Thank you for your patience.
The issues we experienced today that caused partial outage are fully resolved. We are working on a postmortem update and will share that in the next few days. We apologize for any inconvenience and thank you for your patience as we worked through issue.
We’ve discovered some Transaction Engine (TE) users are experiencing issues with payments. We are currently looking into this and will post an update as soon as we have more information. We apologize for any inconvenience.
The issues we experienced today that caused payment issues are fully resolved. We are working on a postmortem update and will share that in the next few days. We apologize for any inconvenience and thank you for your patience as we worked through the issues.
Transaction engine is experiencing a partial outage with ACH and CC payments currently. We are investigating the cause and working towards a resolution.
The Transaction engine boxes have been reset and ACH processing is re-running. CC payments appear to be working again.
ACH processing re-run successfully, CC transactions look good.
Our CXP team has identified what is causing the outage for all transaction processing through WorldPay and FirstData. We are working quickly to implement a fix. Deployment rollback is identified as a temporary workaround. Watch for the next update in approximately 6:00 pm CT.
We are continuing to work on a fix for this issue.
Our CXP team has given us the all-clear and we consider the outage for all transaction processing through WorldPay and FirstData issue resolved. We apologize for any inconvenience and thank you for your patience as we worked through {this / these} issue.
Our CXP team has given us the all-clear and we consider the outage for all transaction processing through WorldPay and FirstData issue resolved. We apologize for any inconvenience and thank you for your patience as we worked through this issue
We’ve discovered some ASAP A3 users are experiencing login issues. We are currently looking into this and will post an update as soon as we have more information. We apologize for any inconvenience.
The issues we experienced today that caused {short explanation} are fully resolved. We are working on a postmortem update and will share that in the next few days. We apologize for any inconvenience and thank you for your patience as we worked through {this / these} issue.
We are investigating reports of Auto Replenish not processing. We will post an update as soon as we have more details. We apologize for any inconvenience.
This incident has been resolved.
Parents are unable to submit Free or Reduced Meal Applications because when the parent logs into their account on MSA and click on the Free or Reduced App button, the system throws an error.
We are currently investigating this issue.
We are continuing to investigate this issue.
The issues we experienced that caused AdminHQ Reports to not run are resolved. We are working on a postmortem update and will share that in the next few days. We apologize for any inconvenience and thank you for your patience as we worked through this issue.
We are investigating reports of MSA Admin Reports not running. We will post an update as soon as we have more details. We apologize for any inconvenience.
Our FSS team is continuing to investigate MSA Admin Reports not running. Fixing this issue is our top priority. We continue to work as quickly as possible to fix this issue. Watch for our next update in approximately 10:30am CT . Thank you for your patience.
Our FSS team is continuing to investigate reports of MSA Admin Reports not running. Fixing this issue is our top priority. We continue to work as quickly as possible to remedy this issue. Watch for our next update in approximately 11:15am CT. Thank you for your patience.
Our FSS team has identified what is causing reports of MSA Admin Reports not running. We are working quickly to implement a fix. Watch for the next update in approximately 12:30pm CT.
-
Our FSS team has identified what is causing reports of MSA Admin Reports not running. We are working quickly to implement a fix. Watch for the next update in approximately 1:40pm CT.
Good news! We implemented a fix and reports of MSA Admin Reports not working is up and running. We are continuing to monitor this situation and will post another update after we’ve confirmed that everything is fully resolved. If you continue to experience problems, please open a ticket with our support team.
We are continuing to monitor for any further issues.
This incident has been resolved.
Our {team name} team is continuing to investigate {issue}. Fixing this issue is our top priority. We continue to work as quickly as possible to {restore fix remedy} {issue}. Watch for our next update in approximately {timeframe}. Thank you for your patience.
We are continuing to investigate this issue.
This incident has been resolved.