52 Alphavps incidents · 2020年3月 — official updates, affected components, duration and resolution details.
我们已开始在以下节点进行先发制人的硬件维护: 软件01 索斯托拉基06号 索斯托拉基07号 在受影响的节点上托管的VPS实例可能会出现暂时中断. 预计每VPS的停机时间约为15至30分钟. 我们的小组正在密切监测维护工作,并将随着工作的进展提供进一步的最新情况.
我们开始研究SOFSTORAGE07。 VMs被优雅地关闭了,不久后会被重新推出.
我们完成了SOFSTORAGE07的工作,虚拟机器正在启动。 我们正在推进SOFSTORAGE01, 虚拟机器正在关闭, 维护开始.
我们完成了SOFSTORAGE01的工作,虚拟机正在启动。 我们正在推进SOFSTORAGE06, 虚拟机器正在关闭, 维护开始.
SOFSTORAGE06的维护工作已经完成,所有VMs正在启动.
这一事件已经得到解决.
自动翻译自官方事件更新。
我们目前正在遭受一场针对我们网络基础设施一部分的大规模DDoS攻击。 这次攻击造成间歇性连通性问题并增加了一些客户的耐用性. 我们的网络工程组正在积极部署更多的永久性缓解措施,以充分应对这一袭击并进一步加强我国基础设施的复原力。 临时缓解措施已经到位,随着进攻的发展,我们继续监督和调整防御措施. 我们为这可能造成的任何干扰表示诚挚的歉意并赞赏你耐心地努力争取一项全面和永久的解决办法。 随着进展,将提供进一步的最新资料.
我们正在继续努力解决这个问题。
我们在纽伦堡应用了缓解法 索非亚问题尚未解决,应当很快实施。 面对勒索/狂暴的DDoS袭击 我们无法公开分享更多细节.
在恶意攻击者不断改变攻击模式的同时,我们也在施以缓解措施,大多数客户都恢复在线. 当地执法部门也参与其中 进一步情况待查.
我们已经在各地实施缓解措施,交通恢复正常. ICMP可能存在差异,但我们相信这一问题正在得到全面解决.
我们继续监督形势,实施缓解调整. 所有地点的所有服务都是在线的,目前被认为是稳定的.
我们正在继续监测任何其他问题.
我们正在监测情况,并调整不同地点的缓解情况,以确保缓解装置的无缝工作,尽量减少中断.
这一事件已经得到解决.
自动翻译自官方事件更新。
在目前供应商出乎意料地撤出纽约后,我们已开始紧急维护我们的纽约地点的网络,以部署新的上游连接。 更多详情通过电子邮件发送给所有受影响的虚拟机客户端. 在维护期间,一些客户在进行网络迁移活动时可能会遇到间歇性连接中断或临时故障. 维护窗口预计可长达8小时,不过我们认为实际中断时间会大大缩短. 我们对通知时间过短和造成的不便表示诚挚的歉意。 这一升级是朝着改善我们纽约服务的长期稳定和质量迈出的重要一步。 随着进展的持续,将公布进一步的最新情况.
我们完成了网络升级,截至1小时,所有网络都稳定。 我们会继续监视几个小时 直到结案.
这一事件已经得到解决.
自动翻译自官方事件更新。
今天早些时候在 Dalstorage01 上发生了一次关键的RAID阵列故障. RAID控制器进入了不稳定状态并继续进行写入操作,导致多张KVM磁盘图像被腐败. 该节点被立即下线,以防止进一步撞击. RAID控制器已经更换,恢复程序正在进行中。 由于阵列的大小,恢复需要时间,无法保证数据的一致性. 目前,达拉斯的可用能力有限. 我们正在积极部署额外的硬件,但确切的ETA尚未确认。 在其他地点向受影响的客户提供替代服务。 应请求,可以进行最佳数据检索。 随着进展的继续,将提供进一步的最新资料。 如果您的 VPS 受到此影响, 您已经被发送了电子邮件, 并附带进一步的步骤 。 如果没有电子邮件,请立即开罚单.
这一事件已经得到解决.
自动翻译自官方事件更新。
We've been alerted of an issue with our sofstorage22 node. We've dispatched DC engineers to check the status of the server and attempt recovery.
We continue working on the sofstorage22 host node. Unfortunately, at the moment we're unable to restore the RAID array. We're continuing with data recovery and further updates will be posted within 24 hours.
This incident has been resolved.
We've been alerted of a problem with our nuestorage03 KVM storage Node. Upon checking, we've discovered the node is completely unreachable by IPMI. We've dispatched on-site techs to check the condition of the node. Further updates to follow.
This incident has been resolved.
We've been alerted by our system that our nueryzenkvm01 host node is currently down. As we're unable to open an IPMI connection, we have engaged Datacenter technicians on-site. Updates to follow.
We're currently experiencing a hardware failure and we'll be replacing some parts of the chassis with on-site spares. ETA of resolution: 2-3 hours.
This incident has been resolved.
On July 4th, 2025, we detected a catastrophic RAID10 SSD array failure on node nueovz04. Fortunately, this node was in the process of being decommissioned and actively migrated to new infrastructure, so the impact is limited to a small number of customers. Since discovery, we have attempted multiple recovery approaches, including replacing RAID controllers and utilizing advanced data recovery techniques. We are currently preparing an identical server chassis to import the array and attempt further recovery efforts. At this time, we cannot guarantee recovery success or provide a precise ETA, but we are targeting the first half of the upcoming week for the next update or resolution. We sincerely apologize for the inconvenience and thank you for your patience while we work through this. — The AlphaVPS Team
Status: Recovery Unsuccessful – Services Being Recreated Following extensive recovery efforts on node nueovz04, we regret to inform you that we have not been able to restore the RAID10 SSD array. Our team has attempted: - Replacing the RAID controller cache - Replacing the entire RAID controller with multiple known working controllers - Transplanting the drives into an identically configured chassis, with fully matching firmware and hardware Despite all recovery attempts, the RAID array remains inaccessible. At this stage, we must consider the data on this node to be lost. We are now moving forward with recreating affected services on a new node. Our team will be reaching out individually to each impacted customer to coordinate this process. If you were hosted on nueovz04 and have not received an email from us, we urge you to contact our support team as soon as possible. We deeply regret this outcome and sincerely apologize for the disruption. Thank you for your patience and understanding during this time. — The AlphaVPS Team
We've detected an issue with our sofovz06 node. We're working on resolving it. Please stand by for updates.
We've resolved the issue as of 14 hours ago. We've monitored the situation throughout the time and believe that system is now stable.
We've been informed of an urgent DataCenter maintenance in our NYC facility, affecting some of our services there. ETA of service restoration is 1-2 hours.
The incident is fully resolved.
We've been alerted of a catastrophic RAID failure on our nuestorage07 node. Earlier this morning, the controller started flapping, which jumped the I/O load to over 90. This made every VM on the host hypervisor unresponsive. We've proceeded with replacing the RAID adapter and we're moving towards an attempted recovery. At this point, we're unsure if recovery would be possible.
We're currently working on bringing the node back online in a stable state. We estimate that it would take us 3-4 additional hours before we start booting the virtual machines up and assessing the damage. Please stand by for further updates.
We are starting with restoration of individual KVM machines. Further updates to follow.
We're continuing to work on restoring VMs, if your VM is affected - please wait until it's booted up.
We'll be booting up the remaining virtual servers in the next 1 hour and continue with monitoring the hypervisor.
Unfortunately, as we booted up the majority of the affected VMs, the node started becoming unstable again with flush operations starting to get queued. We're looking into alternative solutions at the moment. Further updates to come.
We've proceeded with replacing additional hardware. After we've performed manual fsck on each virtual machine's virtual drive - we've proceeded with booting up all affected VPS. As of now, all services are restored. We'll be monitoring the situation throughout the next days.
We've monitored the situation for the past 11 days and as of now, everything is stable and we're closing the issue.
We've identified an issue with our Nuremberg network. We're working on resolving it.
A fix has been implemented. The issue was related to a reboot of a virtual chassis member, causing multiple BGP sessions to start flapping. We're monitoring the environment and we'll apply further fixes as necesary.
We've been monitoring the situation and a fix has been implemented for this specific edge case. We're closing this as resolved.
Our monitoring has alerted us that our laxryzenkvm07 node is currently offline. During our investigation, we've found out that we're unable to reach IPMI/BMC as well. We're dispatching on-site technicians to check the status of the server. Further updates to follow!
Services are back online now.
Our engineers have identified an upstream-related problem, causing packet loss from certain routes towards our London infrastructure. The issue was identified to be upstream-caused. We're awaiting our London connectivity partners to further update us with an estimation of time for service restoration.
The issue seems to be currently resolved but we're awaiting updates from our upstream. We will keep monitoring throughout the day.
Our London connectivity partners state that the issue has been identified and resolved. We will continue to monitor the network throughout the day.
As per now - we consider this to be resolved.
We've identified a hardware issue with our nueovz04 node and we've immediately taken it offline, pending remote hands in DC to swap some hardware components.
The node is back online and we'll be monitoring it for stability and further issues.
This incident has been resolved.
Hello, We've identified an issue in our Sofia location, affecting our network. We've isolated it to our QFX Virtual Chassis Core. We've restored back connectivity on a single member of our Virtual Chassis and no redundancy is present at the moment. We're working on bringing the rest of our VC members online to restore redundancy.
We've identified the main problem, which was related to one of the members of our Juniper QFX Virtual Chassis. At the moment, we believe that one of the virtual chassis members suffered a catastrophic failure of internal storage. Unfortunately, this specific Juniper QFX device was the active master in our VC configuration. As per best practices, we run multiple devices, which can take over the mastership when the current master fails, however in this case - this has not happened, as JunOS was partially running on the failing device. The master killed the internal routing sessions and no packets were being transmitted via the secondary devices. Once we determined the root cause, we've done a manual switchover to another VC member and connectivity was restored. We're working on replacing the failing device and bringing it back in our VC to restore redundancy. Service redundancy should still be considered to be at risk. IPv6 connectivity is not restored on 100% yet, as our priority is bringing v4 redundancy back in place. Further updates to follow.
As of now, IPv6 connectivity is also restored. We're proceeding with replacing the virtual chassis member.
We've installed a new QFX member, after initial configuration and restored full redundancy. As of now, all systems are back to being fully redundant again. We'll continue monitoring the infrastructure and close this issue later today.
We're closing this incident as resolved.
As per our previous email - we're beginning the pre-scheduled migration of nuestorage* nodes.
We're taking down nuestorage01 node down.
Nuestorage01 is ready and booting up, nuestorage02 is currently being shutdown.
We've shutdown nuestorage03. As of the moment - nuestorage02 shows further hardware problems and we'll shut it down again to further debug.
We've completed the maintenance on nuestorage03 and we've taken back nuestorage02 for further checks. As of now - both nuestorage01 and nuestorage03 can be considered stable.
nuestorage02 node is back up and running. As of now - all nodes are back online and all instability issues should be permanently resolved! We're closing this as resolved.
Our monitoring has detected network connectivity issues and packetloss in our LAX location. This seems to be related to our upstream and is out of our scope. We're currently awaiting their network engineers to sort out the problem.
A fix has been implemented and we are monitoring the results.
The incident has been resolved and the main cause was a significant peering disturbance in the LA ANY2 peering platform.
We've started the pre-scheduled network maintenance in Sofia. Minor and short service disruptions might occur.
We've completed the maintenance without any disruption to client services and are closing the maintenance window.
We're investigating a complete network outage in London, related to our upstream. Further news to follow.
As of now, we've restored connectivity to the majority of our clients in London. A small portion of clients residing on a specific network segment are still affected. We're dealing with a large-scale network attack and working closely with our upstreams on completely mitigating the attack.
Our upstream applied some additional mitigation rules on the Corero device and as of now - all clients are back online.
