Störungsverlauf von Velocity Host

cPanel-vh4 server is having a MySQL database issue at present. If your site is on this server it will be experiencing your website as being Up/ Down. Our team is currently working on this issue and will have an update in the next 1 hour. thank you. (vh) Admin Team

Some sites are starting to come back on line. Our team continue to review the issue and work with cPanel to deliver resolution ASAP. thank you (vh) Admin

Our engineers are still working to resolve the issue. Once more information comes to hand we will update. Next update at 12pm. thank you (vh) Admin

Our engineers have identified the root cause of the MySQL engine corruption and are working to restore missing indexes. Once the restore process has completed we will be able to test all connected DB instances (from each account and website). When testing has completed we can confirm the issue has been resolved. We aim to have this process completed within the next hour and to begin testing the fix by 13:00 pm. thank you for your patience (vh) Admin

This issue is now marked as resolved. The DataBase index corruption has been restored from a previous backups. Sites are now stable and working as expected. The longest part of this process has been properly identifying the root cause of the issue. Once the issue was identified applying the fix was a 45 min process and required a server reboot to apply. Please check your sites and if there any questions please open a support ticket and one of our team will reply asap. thanks again for your patience. have an awesome day. (vh) Admin

Thank you to everyone for your patience on this issue review. **Time:** 12am **Date:** 23.02.2021 What we have determined by reviewing logs and timelines that the root cause of this issue stemmed from a hard crash that forced the cPanel server offline at 12am for a short time. Service was restored shortly after with no indication of an issue. The hard crash along with automated update scripts running in the background at the time created the database corruption when the cPanel server crashed. The issue didn't immediately present to our team due to it's intermittent nature, even with monitoring sensors checking uptime of services the issue went unnoticed until reported by a client at approximately at 6am and then verified by our team shortly after. Our team worked with cPanel to try and identify the root cause of the issue and how to resolve it in a timely manor. Due to this being a shared server we needed to keep data integrity at the forefront of our efforts as rolling back to a previous backup would mean hours of new data could be lost, our goal was to make sure we covered every angle to safeguard the most recent transactions to minimise any data loss. With this goal in mind it meant working longer and digging deeper to determine the correct fix even if it meant a longer period of unstable site access in the intrum. Part of this issue was that not every site was effected as we had first thought, sites that where unaffected had a valid entry in the index, only sites that had their entries corrupted experienced the intermittent up/ down database connectivity. > Shout out to our team who worked tirelessly to fix this issue as quickly as time permitted, @Felix big thanks for your great comms and troubleshooting. If you have any further questions on the postmortem please feel free to open a support ticket with “cPanel-vh4 intermittent MySQL Database drops” in the subject field and one of our team will get back to you ASAP. thanks for your patience in this matter. Yours Sincerely \(vh\) Admin Team

There is currently a reported single feed power outage to one of our racks at Micron21 DC. The incident is currently under investigation. We will report back once our team has reviewed and advised. Thank you. (vh) A-Team

We are continuing to investigate this issue.

The cause of the issue has been identified as a faulty power supply in one of our storage servers. All other servers in the rack are now running on dual power again. The storage server in question is still running on it's secondary redundant power supply. A request has been placed with SuperMicro to send a replacement power supply to site ASAP. In the mean time our team will keep an eye on the storage server operations and begin to disperse VM's residing to new storage media in the event the server needs to be shut down for further investigation. we'll keep updating this feed until the issue has been resolved. Thank you. (vh) A-Team

OK ! good to have spares on hand :) Or DC team managed to find a spare Power Supply on hand which has replaced the faulty one in advance before SuperMicro could get to site. Great Work! Thank you DC team! This issue is now marked as resolved. Have a great weekend everyone. Kind Regards (vh) A-Team

This incident has been resolved.

At 3:50 am this morning engineers noticed a host in our CLOUD Cluster experiencing vSAN stability issues. Host ESX 08 stalled and was unresponsive, VMware HA did not kick in and a manual power cycle was required forcing the host to release VM's so they could be (vMotioned) moved to other available hosts. Engineers are working restoring VM's to other hosts. The host is currently pending VMware review. A postmortem will be supplied once logs are collected and reviewed by VMware to identify the root cause of the issue. Thank you (vh) Support

Status Update All VM's have been restored to new hosts. We are currently implementing a plan to shift all workloads off vSan to other available storage. Thank you. (vh) Support

We are continuing to investigate this issue.

This incident has been resolved.

At 2 and 7:30 am this morning engineers noticed a host in our CLOUD Cluster experiencing vSAN stability issues. Host ESX 11 stalled and was unresponsive, VMware HA did not kick in and a manual power cycle was required forcing the host to release VM's so they could be (vMotioned) moved to other available hosts. The host has been placed in maintenance mode awaiting VMware review. A postmortem will be supplied once logs are collected and reviewed by VMware to identify the root cause of the issue. Thank you (vh) Support

A host in the CLOUD cluster had been experiencing IO latency issues causing the host to lock up. Engineers where required to force reboot the host triggering VMware HA to move all VM's to new hosts. A small interruption to service for VM's on the affected host would have been experienced. Services are being monitored and should be back to normal.

We are currently investigating this issue.

Currently investigating network level attack.

We have currently rate limited public internet for all public facing VM's while we track the network saturation point of origin. Services are restored to full operation.

The issue has been identified as a public facing VM with FireWall turned off. Due to this attackers where able to trigger a UDP attack flooding public network traffic creating network latency. The VM in question has been isolated, security increased and all outgoing traffic throttled. thank you for your patience and enjoy the remainder of the long weekend. (vh) Admin Team

Currently experiencing network routing issues, under investigation.

We are continuing to investigate this issue.

A layer 3 routing issue has been identified for up stream traffic, our team is still investigating and implementing incremental planned changes to isolate the issue further. Please keep an eye on our status pages for further updates. Thank you. (vh) Admin Team

Our team is planning a series of troubleshooting steps scheduled in for this evening starting after 6pm, we will continue to monitor the affects of each change before applying the next round of troubleshooting. Thank you for your patience. (vh) Admin Team

We are continuing to work on a fix for this issue, we will update again in 1 hour. Thank you (vh) Admin Team

We are continuing to work on a fix for this issue.

A fix has been implemented and we are monitoring the results.

This incident is being marked as resolved at this time. Thank you for your patience while we worked through the processes involved to safely resolve this issue. (vh) Admin Team

Addressing all \(vh\) clients Thank you for your patience in awaiting this postmortem of last weeks incident. ### Overview: On the 7th April 2020 approximately 1:30 pm \(13:00\) we started seeing intermittent random spurts of traffic latency across our public network. The issue was presenting as slow traffic pings for approximately 6-10 min at a time disrupting connections to Data Center traffic inbound / outbound. At the time of the issue our team had presented a list of possible causes and started to action steps to troubleshoot each possibility. As the day progressed the latency interruption became less disruptive as occurrences where spread further apart eventually stopping at approximately 9:30 pm \(21:30\). Unfortunately we where unable to pinpoint the root cause of the issue at the time due to the inconsistent nature of the events but had setup additional monitoring and continuity plan in the event the issue returned. ### Follow Up Event: 5 days had passed without re-occurrence until Sunday April 12th approximately 5pm \(17:00\), with previous monitoring already in place and more preparation we where able to determine the root cause was a DDOS DNS amplified attack targeting a clients VM which had firewalls turned off at the time. This DDOS attack caused a slowdown in network traffic and was outbound from \(vh\) public facing services. ### Service Improvements To minimize further disruptions our team has performed and put in place the following: * Additional bandwidth throttling measures for direct to public traffic and public facing services * Additional triggers and alarms have been created to monitor client public traffic throughput hitting maximum allowed levels i.e. 70%, 80%, 90% of allowed throughput will alert our team and pinpoint the service abusing shared bandwidth. * Direct to Public and vCloud traffic have been split into two separate networks to minimise disruption across public, private and hybrid network spaces. If you have any questions on the postmortem please feel free to open a support ticket in the \(vh\) cp platform [https://control.velocityhost.com.au](https://control.velocityhost.com.au) and use **"April 7th postmortem"** in the subject field. Enjoy the rest of the Easter long weekend and thanks again for your patience. Kind Regards \(vh\) Admin Team

At approximately 7.39pm cPanel-vh1 was under what we believe to be a DOS or DDOS attack on the server. We are still investigating what type of attack transpired. (vh) Monitoring detected and alerted our team of high CPU 100% Team members witnessed flapping of sites up/ down at this time. Action was taken to disconnect network traffic from the server and proceed to log in via console to review further. After a quick troubleshooting session a server reboot was initiated to clear CPU load and perform a data integrity check. On system reboot a drive FSCK was performed to scrub and check data integrity on disk which prolonged the outage window. Our primary concern at all times is data integrity as 1st priority. The data scrub took longer than expected and once complete systems came back online. According to our monitoring tools @ 7:52 Services had been restored. total outage window 13 min. Once we have more information on the type of attack we will provide further details. Thank you.

An intermittent network issue was detected affecting external > internal service access to all (vh) Internet facing services at Micron21 DC. The issue specifically pertained to network traffic over our top of rack switches. Dell and Cisco engineers had been engaged to assist with troubleshooting the root cause of the issue along with our DC team. The issue is currently marked as resolved, at this stage we are unclear on the root cause and our team are reviewing the incident in more detail gathering all relevant data so we can report back. We expect more information to be available by close of business Monday 2nd of March 2020. Thank you for your patience and understanding. Kind Regards (vh) Support Team

### The Alert & Response On Saturday 29th Feb 2020 our engineers where alerted by internal monitoring systems of a potential network issue. Engineers arrived on site to Micron21 Data Center to discover all network paths flapping up/ down for external internet services creating intermittent network connectivity, engineers began reviewing the issue in more detail. The issue affected our top of rack Dell Force10 switches flooding them with network traffic, our team followed standard troubleshooting steps to identify the root cause and mitigate the problem, in parallel a support ticket was opened with Dell Networking team to initiate a remote session into the switches to begin analysis. On initial investigation Dell engineers where unable to identify the root cause of the problem. A third parallel task was initiated contacting Cisco to help with a root cause analysis and troubleshooting. Cisco engineers looking into issue performed packet captures and where able to identify packets with a TTL \(Time To Live\) of 1ms \(this is a very short interval\) for all packets entering the switches, with this information at hand Dell engineers where able to identify that all traffic packets into the switches where routing directly to the switch CPU'S creating the congestion and flooding the switches creating reduced network flow. ### Issue Identification The issue was identified as a loop potentially created by a bug, fault or external source, Dell requested for senior engineers to be involved which where another 4 hours away so they could trouble shoot and properly identify the cause, taking this action would have extend the outage window by many hours. At this stage we where faced with the potential of another minimum 4 hours of down time or to manually start performing an up/ down troubleshooting process on every port across the switch stack to see if we could identify and disable the packet loop creating the outage. ### Our Course Of Action We decided that customer uptime was a priority over waiting for Dell senior engineers to come online and at a minimum another 4 hours before they could engage in troubleshooting, the decision was made to manually up/ down every port on the switches to identify and isolate the cause of the data loop. After approximately 30 min of manually removing and replacing cables from the switch stack and after a given amount of time the loop was broken and traffic started flowing again. At this stage we are unable to confirm the root cause of the issue that triggered the loop creating congested traffic flooding all top of rack switch ports. ##### **Potential Causes:** * Bug in the switching structure * Fault in the switching structure * External source triggering the loop by flooding the Switch CPU’s This is the first time a problem of this type has been seen in our environment over the last 14 years and there are many unanswered questions that need to be explored. ### Next Steps * Implement procedures for quickly troubleshooting this type of event if it ever arises in the future. * Schedule downtime to upgrade top of rack switched to the latest FOS \(Force10 Operating System\) - Yet to be confirmed. ### Synopses * A networking issue was detected * Engineers where dispatched to site for investigation * Dell and Cisco engineers where engaged to assist in troubleshooting the root cause of the issue and provide additional insight into the problem. * Due to prolonged troubleshooting process with Dell and Cisco, senior engineers from Dell not being available for an unreasonable amount of time a management decision was made to manually find and fix the issue. * After manual troubleshooting the identified “network” packet loop was broken and internet services restored. * An action plan has been formulated for faster troubleshooting of any potential future issues of this nature and potential switch firmware updates - TBA. The \(vh\) team want to thank you for your understanding over this period, if there are any questions please feel free to open a support ticket via our accounts portal [https://control.velocityhost.com.au](https://control.velocityhost.com.au) or contact your account manager directly. Yours Sincerely Gerardo Altman Director Of Problem Solving. [https://velocityhost.com.au](https://velocityhost.com.au)