Platform9 outage history

This incident has been identified and may impact CRUD operations

This incident has been identified and may impact CRUD operations

This incident has been resolved.

An issue has been identified affecting all the PMK Control Planes, and some PMK customers are experiencing service degradation. Our team has identified the issue and is working to implement the fix.

This incident has been resolved.

We are experiencing a major incident for PCD Kubernetes products. Our Engineering team is working on restoring. PCD Virtualization products remain unaffected.

We are continuing to work on a fix for this issue.

This incident has been resolved.

Management Plane is unreachable, our team is actively working on identifying and fixing the issue.

The issue has been identified and a fix is being implemented.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

We're currently investigating an issue related to intermittent management plane instability which may manifest as being unable to login via the UI, API failures, and/or nodes being disconnected.

A fix has been implemented and we are continuing to monitor the environments

This incident has been resolved.

Container image pulls may fail for images hosted by Red Hat Quay image registry due to an ongoing incident. For more information, see: https://status.redhat.com/incidents/qh68rjfg6xs6 Nodes where the images are already cached should continue to function for newly-created containers.

Red Hat has marked the incident as resolved.

We are currently investigating an issue impacting multiple customers' ability to access their management plane instances. Attempts will sometimes result in 502 errors. We will continue to update this post as more details become available.

We are continuing to investigate this issue.

We've identified a related bottleneck where we're observing network congestion on the backend and we've formulated a plan to attempt to mitigate it. The management plane may still be unstable at this time until we've taken these full mitigation steps.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

We're currently investigating reports of issues related to DNS resolution, especially with regard to the Platform9 management plane endpoint. Hosts may intermittently or temporarily show as disconnected. Cluster workloads should remain unaffected.

We've identified the issue with the automated DNS component which was causing issues properly synchronizing CNAME records for our Platform9 management plane instances. Our Operations team is taking steps to restore.

Our Operations team has taken steps to resolve the issue. No further interruptions are anticipated at this time. Our team will continue to monitor the affected environments.

The incident may be considered resolved. No further interruptions have been observed nor reported.

We are currently investigating the issue

A fix has been implemented and environments should start recovering soon

This incident has been resolved.

We are investigating issues with our infrastructure provider.

This incident has been resolved.

There were intermittent issues reaching management planes between 2021-12-15 15:22 - 2021-12-15 16:02 UTC. This resulted from an Internet connectivity outage at our hosting provider (Amazon Web Services). Connectivity has now been restored.

Certain customers cannot reach their management plane deployments. We are currently investigating the issue.

The issue has been identifed to be a network issue with our Data Center service provider. They are working to resolve it.

The network issue has been resolved.

Certain customers cannot reach their management plane deployments. It has been identifed to be a network issue with our Data Center service provider. They are working to resolve it

Our vendor has identified and implemented a fix. Affected customers are now able to access their management plane.

The situation is still being monitored. So far, no issues have been reported.

No issues reported after monitoring the fix for 5 hours. With customers able to access their management plane, the issue stands resolved.

We have detected that the majority of our customers cannot reach their management plane deployments. We are investigating the issue.

Connectivity has been restored

A fix has been implemented and we are monitoring the results.

All Customer Management Planes are Available.

Management plane is unreachable for some customers. We have identified the issue and are implementing fix.

We are continuing to work on a fix for this issue.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

We are currently investigating the issue

Customer Management Planes should be available again. We will continue to monitor the situation

This incident has been resolved.

We are seeing increased error rates for certain customers and are currently investigating to identify the root cause.

We are continuing to investigate this issue.

An issue was resolved by network provider. We are monitoring for issues

A fiber optic issue was identified at our upstream datacenter provider and has been fixed. We are continuing to monitor the situation.

This incident has been resolved.

We have identified a hardware failure that is impacting a small number of customers management planes. We are replacing the hardware and restoring access shortly.

Connectivity to management planes has been restored

We are currently investigating this issue.

The issue has been identified and a fix is being implemented.

The storage outage is resolved and customer management planes will be recovering shortly

The issue should be resolved and we are monitoring the health of the environment

This incident has been resolved.

We are currently investigating this issue.

We have identified the root cause and addressed the issue. We are monitoring the situation. We apologize for the inconvenience

We are continuing to monitor for any further issues.

This incident has been resolved.