48 Catertrax incidents · september 2013 — official updates, affected components, duration and resolution details.
CatertTrax was made aware of an issue with several payment gateways. This is impacting their ability to create new credit card wallets successfully. Payment processing is functional at this time, and the payment gateway companies are aware of the problem and are working to deliver a fix.
This incident has been resolved.
We are currently experiencing a service disruption that is affecting all users. CaterTrax services are restoring but are running in a degraded state. Users may experience degraded performance until systems are fully restored. Our Technology Team has identified the root cause and is continuing to work on implementing a solution. We will send an additional update within 60 minutes.
We are continuing to investigate this issue.
CaterTrax services have been restored and are operating as expected. We will continue to monitor and provide a final update within two hours.
A fix has been implemented and we are monitoring the results.
We are pleased to inform you that all services have been fully restored. Our Technology Team has found no further issues. Thank you for your patience during this time.
We are currently experiencing slowness with our application. Our Technology Team is working to identify the root cause and implement a solution. Users may be affected. We will send an additional update by 2:00 PM Eastern Standard Time.
We are continuing to investigate this issue.
We are continuing to experience slowness with our application. Our Technology Team has identified the issue within the Microsoft Azure platform and is working with them to implement a solution. Users may still be affected. We will send an additional update by 4:00 PM Eastern Standard Time.
The system slowness is subsiding for today as overall traffic decreases. Our Technology Team is continuing to work with Microsoft to address the root cause of the issue. Users should no longer be affected by slowness in the application. We will send an additional update by 8:00 PM Eastern Standard Time.
The root issue causing system slowness has been resolved. Our Technology Team will be monitoring the application throughout the night. Users should no longer be experiencing slowness in the application. We will send an additional update by 8:00 AM Eastern Standard Time.
We are pleased to inform you that the slowness issue has been resolved. Our Technology Team found no further issues overnight. Thank you for your patience during this time.
Please be advised that we are aware of and actively troubleshooting an issue that is currently impacting the ability of some users to log into their sites at this time. This issue is being treated as our highest priority and our team is working hard to get the problem resolved as quickly as possible.
Our team has identified an issue that is currently causing intermittent site latency and 504 Gateway Timeout errors for some users. This issue is our highest priority and we are working to resolve it as quickly as possible.
Our team has identified a few issues that are currently causing some intermittent site latency as well as impacting the ability for some users to log into their sites at this time. These issues are our highest priority and we are working to resolve them as quickly as possible.
Our team is continuing to investigate new reports of degraded performance and latency issues. While the system is currently operating within expected response times, our focus continues to be on performance improvements and resolving open issues. Next update 8:00 AM EST tomorrow.
The system continues to be operating within expected response times. While we work on the remaining open issues, we are closely monitoring performance and latency. Next update 8:00 AM EST tomorrow.
Our team has identified and mitigated issues causing intermittent performance degradation to the TRAX Platform. While the system is currently operating within expected response times, we will continue to monitor system performance to ensure a consistent experience for you and your customers. Our focus will shift to additional tuning activities. Next update Tuesday 3/14 at 8:00 AM EST.
The TRAX Platform remains within expected response times. Additional tuning activities will continue this week as we monitor system performance. Next update Friday 3/17 at 8:00 AM.
The TRAX Platform has returned to expected response times. Additionally, the intermittent site latency and user login issues have been resolved. This will be the last update.
We have identified that some CaterTrax websites (.catertrax.com) may be experiencing slowness that is impacting service. Our teams are currently investigating the issue and will provide updates as we have them.
We are continuing to investigate this issue.
System performance has returned to normal, expected levels at this time. We are continuing to investigate this issue. If you are continuing to experience issues please contact CaterTrax Support.
The performance issue identified yesterday, Monday 2/3, has been resolved and the TRAX Platform is operating as expected.
The TRAX Platform is currently unavailable. The CaterTrax team is currently working to identify the cause and resolve the issue. Additional updates will be provided throughout the investigation, every 30 minutes.
The TRAX Platform is currently unavailable. The CaterTrax team is continuing to work to identify the cause and resolve this issue. Additional updates will be provided throughout the investigation, every thirty minutes.
The TRAX Platform is now online. The CaterTrax team is continuing to monitor systems at this time. Additional updates will be provided regarding TRAX Platform status.
We are continuing to monitor for any further issues.
The TRAX Platform is online and has been operational since 9:30 AM EST. The CaterTrax team continues to monitor systems at this time. Additional updates will be provided regarding TRAX Platform status.
Since 9:30 AM EST the TRAX Platform has been online and stable. The CaterTrax team is continuing additional monitoring of systems and will provide an additional update later today.
Since 9:30 AM EST the TRAX Platform has continued to be online, operational, and stable. The CaterTrax team is following through with additional monitoring of systems for a full 24 hours. The next update will be communicated tomorrow, 9/19/2019, at 8 AM EST.
The TRAX Platform was not functional yesterday (Wednesday) morning from approximately 7:40 AM EDT to 9:30 AM EDT. The TRAX Platform was brought back online at 9:30 AM EDT and has remained operational and stable. This incident is now resolved and the CaterTrax team will continue to monitor our systems to prevent any additional disruptions of service.
From approximately 6:30 AM to 9:00 AM EST, there was a disruption of service to payment processing functionality on the TRAX Platform. The CaterTrax team identified the issue, implemented a fix, and as of 9:00 AM EST payment processing functionality was restored to all impacted sites. The CaterTrax team is actively monitoring the issue and will update this site if more information becomes available. We apologize for any inconvenience this may have caused. For real-time updates, check back to this site. If you have additional questions, please do not hesitate to contact our client support team at 1-800-975-8729.
This incident has been resolved.
Through application monitoring and based on increases in customer support requests, the CaterTrax team has identified that some end-users and administrators are experiencing errors when performing certain TRAX Platform functions. Our teams are working as quickly as possible to determine the underlying cause of this issue. Please continue to reach out to our dedicated Client Support Team with questions or concerns by logging a support ticket or calling them directly at 1-800-975-8729. We will continue to update this site as updates become available.
The CaterTrax team identified and fixed the previously reported issue with the TRAX Platform. This issue, which affected a subset administrators and end-users, was resolved at approximately 12:05 PM EST. Our team will continue to monitor this incident for the next 24 hours.
The CaterTrax team has resolved the issue being experienced by some administrators and end-users yesterday. The team will continue to proactively monitor the platform and the infrastructure that supports the application.
Through monitoring, the CaterTrax Team has documented intermittent platform latency when some administrators are completing various online functions. Our teams are working as quickly as possible to optimize the speed of the TRAX Platform. Please continue to reach out to our dedicated Client Support Team with questions or concerns by logging a support ticket or calling them directly at 1-800-975-8729. We will continue to update this site as updates become available.
We are continuing to investigate this issue.
Our teams will be performing maintenance tasks on our servers Saturday, December 15th 5:00 AM to 6:00 AM EST. During this time, you will be unable to login in to the administrative side of the TRAX Platform. Additionally, you and your clients will see a temporary maintenance page when visiting your site. We will continue to monitor the TRAX Platform for intermittent latency beyond this coming weekend and into next week.
During the week of Monday, December 10, the CaterTrax team identified an issue impacting the page speed performance of CaterTrax’s catering administrator and customer facing features. After review and investigation by CaterTrax’s Infrastructure, Development, and Customer Support teams a fix was identified and applied to CaterTrax’s production systems on Sunday, December 16. Since the deployment of these focused optimizations, CaterTrax’s application performance has returned to normal levels. The CaterTrax team continues to monitor system activity to ensure expected performance.
This morning, some CaterTrax administrators were experiencing errors when trying to login to the administrative side of the TRAX Platform. The front of house and customer online ordering was not impacted by this incident. These issues were intermittent, impacting a fraction of sites. Our team has mitigated the errors and is continuing to monitor the situation. The login issues have been resolved but if you find you’re are still receiving an error message when trying to log in as an administrator, please contact CaterTrax support at 1 (800) 975-8729.
At approximately 8 AM this morning our team noticed an issue that was impacting administrators ability to login to the administrative side of the TRAX Platform. At approximately 8:30 AM the team had mitigated the issues and monitored and further investigated the root cause. The team has identified the cause and reported the incident as resolved. If you believe you are still experiencing issues please contact CaterTrax support at 1 (800) 975-8729.
At approximately 11:00 AM EST, administrators started logging support tickets regarding experiencing occasional issues with administrative functionality of the TRAX Platform. Our teams are investigating the issue and looking to identify the root cause as quickly as possible. We will provide an update via this site as they are available.
The teams continue to investigate the issue, but it has been determined the issue is intermittent and is being experienced by a small percentage of administrators. We will provide an update via this site as they are available. Our Client Support team is available to assist clients with questions.
The team has made changes to the TRAX Platform to rectify the intermittent issues some administrators were facing earlier today. The team is still investigating the root cause for the intermittent disruption in service and will continue to monitor the incident for 24 hours. Our administrators should no longer be experiencing any issues with administrative functionality for the TRAX Platform. Our Client Support team is available to assist clients until 5:00 PM EST today.
The Technology team has completed its investigation related to the incident with login functionality originally reported and resolved on Tuesday, May 29th. The original incident impacted a subset of our administrative users and was specific to the administrator login functionality. Our team has identified the root cause and continues to monitor the TRAX platform for any additional occurrences of the error.
Due to internal monitoring this morning we noticed that select gateways are experiencing intermittent disruption in service processing credit card gateways. Clients using Payeezy, Converge, and Vantiv gateways are experiencing issues processing their payments from credit cards. All other aspects of the TRAX Platform is operational. We will update this site as the team investigates the issue. Our Client Support team is available and is fielding inquiries from clients.
Clients can now process credit cards through Vantiv, Payeeze, and Converge gateways. Our teams will continue to monitor the situation.
This incident has been resolved.
At approximately 10:45 AM EST due to internal monitoring, we noticed the platform was experiencing a disruption in service. Our teams are working as quickly as possible to investigate the issue. We will update this site with information as it becomes available. Our Support team is available and is fielding phone calls and emails. We have switched over our phone message to indicate we are aware that there is a technical incident.
At approximately 11:45 AM EST site performance was restored and our clients and their customers are no longer experiencing an intermittent disruption in service. Our teams are working to identify the root cause for the disruption. We will continue to monitor the situation for 48 hours to ensure platform performance is consistent.
This incident was monitored for the last two days and we are happy to report that it is now resolved.
At approximately 12:55 PM EST due to internal monitoring, we noticed the platform was experiencing a disruption in service. Our teams are working as quickly as possible to investigate the issue. We will update this site with information as it becomes available. Our Support team is available and is fielding phone calls and emails. We have switched over our phone message to indicate we are aware that there is a technical incident. We understand for our east-coast clients lunch time is a very busy time in your schedule and we apologize for the inconvenience.
Our team identified opportunities that resulted from maintenance that occurred this weekend to further enhance our servers. This additional maintenance caused unforeseen issues which resulted in a disruption in service. All sites are now operational as of 1:28 pm EST. Our Support team is available Monday - Friday 8:00 AM - 7:00 PM EST to answer any of your questions or requests for support.
This incident has been resolved.
At approximately 8:15 AM EST our clients are unable to login to the TRAX platform using their administrative credentials. Our Support team is fielding calls and emails about the issue. While our Infrastructure team works to investigate the issue, some clients have had success with clearing their cache in the web browser they are using. We will update this site as more information becomes available. We apologize for the inconvenience and understand the morning is a busy time for many of our operations.
Thank you for your patience this morning as our team investigated the administrative login issue. We are happy to report the issue has been resolved and we will continue to monitor the service levels of the administration login and all other core functionality of the TRAX Platform. The incident has been tied back to our maintenance window this past weekend. The technology team worked this past weekend, Saturday - Sunday AM, during our scheduled maintenance window to avoid hours when our clients would need access to the TRAX Platform. These enhancements to our infrastructure were completed as planned and testing concluded by 3:00 AM EST on February 11th. Early this morning, our team first began to detect failed administrator login attempts and general slowness for the platform. After careful investigation our team identified that configurations within our security appliances were incorrectly blocking some requests between the TRAX Platform and our authentication (login) services. The authentication services were failing intermittently for specific sites and users based on how their request was routed and serviced within our cloud environment for the servers. Our team first applied changes to increase overall application performance and then resolved the login errors across the board by approximately 11:45 AM EST. We apologize for the inconvenience, we understand this is not the ideal way to start off your week or any business day. As a result of this weekend’s project, you’ll notice a significant improvement to the page load times for your site. This is a direct result of the important work the teams performed this weekend. Our Support team is available Monday - Friday 8:00 AM - 7:00 PM EST to answer any of your questions or requests for support.
This incident has been resolved.
At approximately 8:00 AM we discovered the phone system used by CaterTrax is currently experiencing a disruption in service. At this time, we cannot place or receive phone calls but we can receive voicemails. The best way to submit a support request at this time is to submit a support ticket through your administrative credentials to the TRAX Platform to receive assistance. We apologize in advance for any delays in response you may experience.
Our phone service has been restored overnight. Clients that need to talk with our teams are welcome to give us a ring! Thank you for your patience and submitting all your requests for support through tickets yesterday while our phones were unavailable.
At approximately 10:38 AM EST, we are reporting some sites experiencing intermittent disruptions in service . We are working actively on a solution and will update you as we have information. We apologize for the inconvenience and understand how much you rely on our technology to run your business.
Thank you for patience as our teams continue to investigate intermittent slowed page load speed for some of our sites. Any administrator that called in and placed a support ticket (approximately 20 locations), will be contacted by the Client Support team personally to notify when all sites are restored to normal. Meanwhile, continue to check this site for further updates this week regarding this incident.
The incident from Monday, October 30th is still being analyzed by the team. They also continue to monitor page load speed on sites and all sites have been performing within reasonable ranges for the last two days. Thank you for your patience, we are aiming to make updates to this site as details become available.
The root cause for the incident on Monday, October 30th is being attributed to server resource allocation configurations. The technology teams have been monitoring performance with the TRAX Platform and are happy to report that all sites continue to function as normal. We value having you as a client and understand the impact a distribution in service can cause your business. Thank you for your patience as the teams working tirelessly to ensure availability our of software.
Starting at 10:58 AM this morning, our technology teams identified extreme site slowness that temporarily impacted all sites on the TRAX Platform. Our technology teams quickly found a resolution and have fully restored all sites. Our team is still investigating the root cause of this occurrence and will update this site with more information as it becomes available. We apologize for any inconvenience this may have caused. For real-time updates, check back to this site. If you have additional questions, please do not hesitate to contact our client support team at 1-800-975-8729.
This incident has been resolved.
At approximately 9:50 AM our corporate website became unavailable. We are working with our hosting provider to troubleshoot the issue. Catertrax.com is hosted independently in an exclusive WordPress environment separate from the TRAX Platform with a different hosting provider. We appreciate clients letting us know they can't get to their favorites spots on our corporate site, but rest-assure there are no issues with our software. We will update this communication as information becomes available.
What our corporate site experienced is part of a wider spread issue with our hosting provider. The site was unavailable for about 15 minutes, but is running as expected. We will continue to monitor, but don't anticipate any further issues.
It has been more than 24 hours and our corporate site has not experienced any additional hosting issues. AWS reported that the issue was isolated to the Northeast yesterday morning.
Our teams are continuously monitoring and taking steps to prevent cyber-attacks, like the May 12, 2017 attack that has impacted thousands of businesses worldwide. This widely publicized cyber-attack was documented by several names, including: “Wanna Decryptor”, “WannaCry Ransomware”, “WannaCry”, and “WCry”. Please be aware that we were not affected by the current version of this ransomware. This most recent ransomware exploited a specific vulnerability in the Microsoft Server Message Block (SMB) server that allows remote execution of code on machines. This would enable an individual to deliver files and code to a computer without local intervention. This vulnerability was discovered in March of 2017, and a critical security update was released by Microsoft (KB4013389). All Windows computers with automatic updates enabled would have applied the update on their own during their next patch cycle. CaterTrax has automatic updates enabled for Microsoft. Therefore, the patch for this ransomware was applied as part of our standard update schedule on April 2, 2017. We have not identified any disruption of our production or headquarters environments. We have confirmed that all computers and servers in our network at CaterTrax headquarters have received the critical security patch, eliminating the current vulnerability that ransomware leverages. Additionally, we have enterprise-level firewalls at CaterTrax headquarters and our hosting provider with several layers of security scanning and filtering to remove high-risk traffic. CaterTrax is taking as many preventive measures as possible to protect and maintain our technology platform and employee networks to provide the best possible service for your business needs. We will continue to keep you informed with key updates like this as they arise.