История сбоев Ionos Cloud

We are currently investigating problems accessing VMs and DBaaS service located in Frankfurt. We will inform you as soon as the functionality has been restored.

This was a duplicate.

We are currently experiencing an issue where resources in the FRA region are unreachable and showing an Unknown state. This issue is currently being investigated and we will keep you informed of any updates.

We are continuing to investigate this issue.

Our teams identified some affected hardware servers that caused the unavailability. We are now in process of migrating the affected VMs to different HW servers, so please check your VMs. We are still working on the root cause investigation.

Our tech teams were able to restore all VMs and products, so all are back available and running. However, we keep monitoring the situation for a while.

The incident has been resolved, the VMs are back working again.

We are currently responding to an incident in our VIT datacenter. We will regularly update this status page with information.

Power has been restored and site support systems are back in operational state. We have started recovering infrastructure components to fully restore service.

Most affected hosts have been brought back into service. We are working on restoring the remaining components.

All but one hosts are recovered. The remaining host suffered a hardware defect and needs to be replaced to recover the workload provisioned there. This is currently underway. We estimate that this work will be completed within the next minutes.

The last host and VMs have been recovered. We are setting this incident into monitoring status. Preliminary information on the Incident: Following a power grid outage, the site suffered a partial interruption of power, which impacted several datacenter support systems. The reason for this is under active investigation. This affected the availability of a subset of infrastructure components and hosts, subsequently impacting the VMs provisioned on them or dependent on those components. A full Root Cause Analysis (RCA) is currently being compiled and will be shared here once completed. Thank you for your patience.

We are marking this incident as resolved. The Root Cause Analysis will be published here once compiled.

We are currently investigating instances of performance degradation in job processing. Our Provisioning Team is currently working on fixing the root cause of these events.

Still, the global provisioning system is not working properly. Therefore, all other dependencies, like K8s and managed products, are affected, too. We know this is affecting customer setups, and that urgent changes are not processed properly, and we apologize for the inconvenience. We identified possible distractors, but fixing this is a complex task, with need to involve and coordinate several technical teams. We are working with highest prio to get the provisioning back working, and we will let you know asap on any progress.

Our Provisioning Team has been closely tracking queue performance over the last few hours and has noted steady improvements. We are moving this incident to Monitoring status. We will continue to monitor queue processing closely and will update this page on any changes.

This incident has been resolved. We are working on the RCA and will publish it here asap

We are currently experiencing increased processing times for provisioning orders initiated via the Data Center Designer (DCD) or API. Additionally, intermittent connectivity issues may occur when accessing the Data Center Designer or API. Please note that this issue is not limited to Managed Kubernetes services and may impact all cloud workloads involving new provisioning or modifications. Important: The availability, stability, and accessibility of your existing running virtual data center resources remain entirely unaffected. Our engineering teams are actively working to resolve the issue. We will provide a further update as soon as full functionality has been restored.

We are continuing to work on a fix for this issue.

A fix has been implemented and we are monitoring the results.

This incident has been resolved. We are working on the RCA and will publish it here asap

We are currently investigating instances of performance degradation in job processing. Our Provisioning Team is currently investigating the root cause of these events.

For Investigation we set mk8s api to maintenance mode (read-only)

We are continuing to investigate this issue.

We are currently seeing progress in provisioning, and the DCD maintenance page has been removed from the DCD.

The provisioning queue looks healthy again, and all services have been reactivated.

Currently we are facing increased provisioning problems resulting in error code [VDC-5-0]. We understand the impact, and our teams are investigating the root cause to get this fixed asap. Please postpone your provisioning until we resolve the issue, we will inform you of any updates.

We found the root cause and implemented mitigation measures. We expect the errors to decrease significantly now. We will let you know asap when we consider the situation being back to normal.

Our Provisioning went back operational yesterday around 16:30 UTC. Further monitoring showed no issues anymore, so this incident was resolved.

IONOS Cloud Support is temporarily unavailable via phone. We ask Customers and Partners to contact Cloud Support via the DCD Form or Email, instead.

We are continuing to investigate this issue.

Phone Support is now available again

Customers may experience connection problems to the control plane and degraded functionality of kubernetes

The situation appears to have stabilized for the time being. An internal investigation into the exact root cause is ongoing.

Incident involving degradation of the control plane has been resolved and we are monitoring the situation.

This incident has been resolved.

Due to an internal network issue that is currently being investigated, all models are currently unavailable.

We are continuing to investigate this issue.

A fix has been implemented and we are monitoring the service.

This incident has been resolved.

Due to exceptionally high demand in two clusters in our TXL and FRA datacenters, we have temporarily paused Managed Kubernetes Automated Maintenance. To ensure that clusters remain operational without interruption during updates, Automated Maintenance requires temporary cluster "cloning", which increases resource consumption. As most maintenance windows are scheduled to run over the weekend, resource demand peaks sharply during these periods. To protect cluster stability and prevent service degradation, we have paused maintenance for the affected clusters. Any maintenance scheduled from today onward will not run until we resume. We know automated Kubernetes maintenance is an important feature of our platform, and we did not take this decision lightly. We are expanding capacity in the affected clusters as a high priority and will update this page as soon as maintenance resumes.

We were able to reactivate the MK8s maintenances in all clusters in our TXL datacenter

We are currently investigating issues with the MK8s control plane

Control Planes are reachable again. Our MK8s Team is currently investigating the root cause for the service interruption.

We have observed another spike and are setting the incident back to active. Our MK8s Team is informed and are actively working the incident.

We have marked this incident as resolved. Several underlying VMs entered an unhealthy state, which increased the CPU load on the remaining healthy VMs supporting the service. This cascaded into the intermittent availability issues. All systems are now operating normally.

We are currently investigating increased error rate for S3 in our FRA DC

We are reducing the impact of the incident, as our Object Storage Team has traced back the increased error rates to mainly SlowDown codes, which affect only heavily loaded buckets.

Error rate has dropped. We are monitoring the situation further.

We are marking this incident as resolved, as no further abnormalities occurred.

We are currently investigating instances of performance degradation in job processing. Our Provisioning Team is currently investigating the root cause of these events. Customers might see delays and errors in provisioning job processing.

Provisioning is currently operating as expected, and we are closely monitoring the system. Our team is continuing to investigate the root cause of the earlier performance degradation.

We are resolving this incident as provisioning performance has fully normalized.

We are currently experiencing exceptionally high demand for GPU servers, which has temporarily limited our available capacity. As a result, you may encounter provisioning errors when attempting to create a new GPU server or start an existing one. What You Can Do - Try again later: We cannot provide an exact timeline for when capacity will be added. However, we are monitoring the situation closely and will update this status page once substantial free capacity becomes available. - Provision a smaller GPU server: Because availability fluctuates rapidly, we cannot guarantee which specific sizes are currently open. Attempting to provision a smaller instance may increase your chances of success. - If you currently have a provisioned and running GPU server, do not shut it down. Due to the low supply, freed resources may be immediately allocated to other customers, and you will likely be unable to restart your server. Next Steps & Resolution Resolving this resource constraint has high priority. Our teams are working to increase capacity to meet demand as quickly as possible. We sincerely appreciate your patience and understanding.

We are currently experiencing a high error rate with S3 Object Storage in TXL - eu-central-3. This is currently being investigated and we will continue to keep this page updated.

The incident has now been resolved, and accessing S3 Buckets should now be working as expected.

We are currently investigating increased loading times and timeouts for the DCD.

Our DCD Team is currently investigating the incident. We have confirmed that Cloud API remains functional and available. Customers and Partners needing to deploy changes to their setup can still use ionosctl and cloudapi to do so.

We believe that the trigger for the incident has been mitigated. We are currently working on restoring all services related to the DCD.

The DCD is available again. We are monitoring the system closely and will set this incident to resolved once the environment is confirmed stable.

The system is stable. We are resolving this incident. The trigger of this incident will be shared once the analysis has been completed.

IONOS Cloud Support is temporarily unavailable via phone. We ask Customers and Partners to contact Cloud Support via DCD Form and Email, instead.

Phone Support is now available again

MongoDB Playground and Business Edition clusters are currently not available for provisioning in de/fra/2 (Frankfurt East). This is due to a capacity limitation in that location that prevents these cluster types from being created reliably, and it is expected to persist for the time being. We will update this status page as soon as this limitation is lifted. As an alternative, you can: - Provision your Playground or Business Edition cluster in another available location, or - Use MongoDB Enterprise Edition, which remains available in Frankfurt East.

We are currently investigating increased error rate for S3 in eu-central-3

Our Object Storage Team has identified an issue connected with gateways in the affected region and is currently mitigating the problem. Customers may see connectivity issues and timeouts. We will keep you updated on the mitigation efforts.

The mitigation efforts have lead to first improvements. We have downgraded the incident severity. However, we still see elevated error rates and are working towards resolving the remaining issues.

Error rates have significantly decreased. We are monitoring the situation further and will resolve the incident if the situation stays as is.

We are closing the incident as no further error spikes were recorded. We will share the trigger of this incident once the analysis has been conducted.