Instaclustr outage history

Google Cloud has confirmed an issue affecting multiple services in the GCP europe-west2 region, which has been ongoing since approximately 06:05 UTC. Google has reported elevated errors and intermittent network failures in the region. All nodes in the europe-west2-b availability zone remain unavailable. Affected clusters are partially impacted and continue to run on their nodes in the other availability zones, with reduced capacity. Provisioning new clusters in europe-west2 will also fail at this time. Based on our testing to date, other GCP regions in Europe do not appear to be affected. Our engineers are treating this as a high-priority incident and are monitoring Google's progress closely. We will work to restore the impacted nodes as soon as the region recovers.

We are continuing to investigate this issue.

We are continuing to investigate this issue.

Google Cloud has confirmed an issue affecting multiple services in the GCP europe-west2 region, which has been ongoing since approximately 06:05 UTC. Google has reported elevated errors and intermittent network failures in the region. All nodes in the europe-west2-b availability zone remain unavailable. Affected clusters are partially impacted and continue to run on their nodes in the other availability zones, with reduced capacity. Provisioning new clusters in europe-west2 will also fail at this time. Based on our testing to date, other GCP regions in Europe do not appear to be affected. Our engineers are treating this as a high-priority incident and are monitoring Google's progress closely. We will work to restore the impacted nodes as soon as the region recovers.

Google Cloud has now published a public incident for the europe-west2 region: https://status.cloud.google.com/incidents/Xs2XTHzAVC9YnafnPyzj Google has not yet provided a workaround or an estimated time to resolution. We continue to monitor the situation closely and will provide further updates as information becomes available.

Google Cloud has advised that the outage was caused by a power event in the europe-west2-b zone, which has now been mitigated. Google is recovering existing workloads and has tentatively estimated full recovery by approximately 13:00 UTC. We are seeing some affected nodes in europe-west2-b begin to recover. Other nodes are still unavailable or not yet stable, and our team is verifying the health of each affected cluster as nodes come back online. Provisioning new clusters and nodes in europe-west2 may continue to fail until Google completes recovery. We will provide a further update once recovery is complete.

Google Cloud has resolved the power event that caused the outage in the europe-west2-b zone. All affected clusters have been restored and are operating normally, and provisioning of new clusters and nodes in GCP europe-west2 is working again. Cluster data centre resizes in GCP europe-west2 have also been re-enabled. We apologise for any disruption this may have caused. Google has advised it will publish an analysis of the incident once its internal investigation is complete. If you are still experiencing any issues with your clusters, please contact [email protected].

We are currently investigating elevated latencies and error rates with our Monitoring API. Our team is actively working to resolve the issue and we apologise for any inconvenience caused. Existing customer clusters are unaffected. The Management Console and Cluster Management API are still operational. We will provide updates as soon as they become available. If you have an urgent matter, please reach out to us via the support portal (support.instaclustr.com) or email ([email protected]).

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

We are currently experiencing issues with provisioning clusters through the Cluster Management API and Management Console. Our team is actively working on resolving this problem and we apologise for any inconvenience caused. Existing customer clusters are unaffected, and the Monitoring API and Prometheus API are operational. Additional details will be provided as they become available. If you have an urgent matter, please reach out to us via the support portal (support.instaclustr.com) or email ([email protected]).

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

We are currently experiencing service degradation and potential disruptions in the Google Cloud Platform region us-west1. This incident is a result of an ongoing issue with the cloud provider's infrastructure. You will be unable to provision new clusters or nodes. Existing nodes in us-west1 may be unavailable. Cluster datacentres in other regions are unaffected, but are experiencing networking issues when interacting with nodes in us-west1. Individual impacted customers are being contacted via support tickets. We are actively monitoring the situation and will provide additional details as they become available. GCP status is tracked at https://status.cloud.google.com/. If you have an urgent matter, please reach out to us via the support portal (support.instaclustr.com) or email ([email protected]).

We are continuing to work on a fix for this issue.

Vertical scaling has been temporarily disabled in the GCP us-west1 region due to this cloud provider outage. We will re-enable this functionality when the cloud provider outage is resolved.

Google have completed mitigations and are reporting recovery across multiple products. We will continue monitoring the incident until the remaining products have recovered. Vertical scaling remains disabled in us-west1.

This incident has been resolved.

We are currently seeing compatibility issues between the Instaclustr Terraform provider and the Cluster Management API, affecting operations involving OpenSearch clusters. Standard customer clusters operations are unaffected. The Monitoring API and Prometheus API remain operational. Our team is actively working to resolve the issue and we apologise for any inconvenience. Further updates will be provided as more information becomes available. If you have an urgent matter, please contact us through the support portal at https://support.instaclustr.com or by email at [email protected].

This incident has been resolved.

We are currently experiencing issues with provisioning private network PostgreSQL clusters with 3 nodes through the Cluster Management API and Management Console. Our team is actively working on resolving this problem and we apologise for any inconvenience caused. Existing customer clusters are unaffected, and the Monitoring API and Prometheus API are operational. Additional details will be provided as they become available. If you have an urgent matter, please reach out to us via the support portal (support.instaclustr.com) or email ([email protected]).

We are continuing to work on a fix for this issue.

This incident has been resolved.

We are currently experiencing service degradation and potential disruptions in the GCP us-east4-b zone. GCP has not updated its status page yet, however initial investigation indicates an issue likely with the cloud provider's infrastructure in the us-east4-b zone. While we are seeing partial recovery, in the meantime provisioning new clusters or nodes in the concerned GCP zone will be impacted. Also existing nodes in the GCP us-east4-b zone can face intermittent issues and be offline. You should still be able to access your clusters if you are running a standard configuration. Individual impacted customers are being contacted via support tickets. We are actively monitoring the situation and will provide additional details as they become available. GCP status is tracked at https://status.cloud.google.com/. If you have an urgent matter, please reach out to us via the support portal (support.instaclustr.com) or email ([email protected]).

This incident has been resolved.

We are currently experiencing issues with provisioning Kafka Connect & Cadence clusters into existing VPCs through the Cluster Management API and Management Console. Our team is actively working on resolving this problem and we apologise for any inconvenience caused. Existing customer clusters are unaffected, and the Monitoring API and Prometheus API are operational. Additional details will be provided as they become available. If you have an urgent matter, please reach out to us via the support portal (support.instaclustr.com) or email ([email protected]).

The issue has been identified and a fix is being implemented.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

We are currently investigating an issue causing cluster level Prometheus monitoring query requests to be incorrectly blocked with a 403 error. Other Prometheus monitoring queries are unaffected. Our team is actively working to resolve the issue, and we apologise for any inconvenience caused. The Management Console, Cluster Management API and Monitoring API are still operational. We will provide updates as soon as they become available. If you have an urgent matter, please reach out to us via the support portal (support.instaclustr.com) or email ([email protected]).

The issue has been identified, and our team is actively working on implementing a fix. Cluster-level Prometheus monitoring query requests are being incorrectly blocked with a 403 error. Other Prometheus monitoring queries are unaffected. Existing customer clusters are unaffected. The Management Console and Cluster Management API are still operational. We will provide updates as soon as they become available. If you have an urgent matter, please reach out to us via the support portal (support.instaclustr.com) or email ([email protected]).

A fix is being rolled out and we are monitoring the results.

This incident has been resolved.

We are currently investigating elevated latencies and error rates with our Monitoring API. Our team is actively working to resolve the issue and we apologise for any inconvenience caused. Existing customer clusters are unaffected. The Management Console and Cluster Management API are still operational. We will provide updates as soon as they become available. If you have an urgent matter, please reach out to us via the support portal (support.instaclustr.com) or email ([email protected]).

This incident has been resolved.

We are currently experiencing service degradation and potential disruptions in the AWS region US-EAST-1. This incident is a result of an ongoing issue with the cloud provider's infrastructure. Customers using the AWS availability zone "use1-az4" are impacted, with nodes in this availability zone observed to become unavailable. Clusters are expected to be available overall, and not significantly impacted. We are actively monitoring the situation and will provide additional details as they become available. AWS status is tracked at https://health.aws.amazon.com/health/status. If you have an urgent matter, please reach out to us via the support portal (support.instaclustr.com) or email ([email protected]).

We are continuing to monitor the issue.

This incident has been resolved.

We are currently investigating elevated latencies and error rates with our Monitoring API. Our team is actively working to resolve the issue and we apologise for any inconvenience caused. Existing customer clusters are unaffected. The Management Console and Cluster Management API are still operational. We will provide updates as soon as they become available. If you have an urgent matter, please reach out to us via the support portal (support.instaclustr.com) or email ([email protected]).

This incident has been resolved.

We are currently experiencing issues with provisioning Cadence and Kafka Connect clusters on AWS when provisioning into a Private Network through the Cluster Management API and Management Console. Our team is actively working on resolving this problem and we apologise for any inconvenience caused. Existing customer clusters are unaffected, and the Monitoring API and Prometheus API are operational.  Additional details will be provided as they become available. If you have an urgent matter, please reach out to us via the support portal (support.instaclustr.com) or email ([email protected]).

A fix for this issue has been deployed.

We are currently experiencing issues with provisioning clusters through the Cluster Management API and Management Console. Our team is actively working on resolving this problem and we apologise for any inconvenience caused. In addition, operations on existing clusters such as replacing nodes and restores aren't functional, and we are investigating elevated latencies with the Monitoring API. Additional details will be provided as they become available. If you have an urgent matter, please reach out to us via the support portal (support.instaclustr.com) or email ([email protected]).

We are further seeing elevated latencies and error rates with our Monitoring API and Cluster Management API and additionally seeing issues logging into our Management Console. Our team is actively working to resolve the issue and we apologise for any inconvenience caused. Existing customer clusters are unaffected. The Management Console and Cluster Management API are partially degraded at this time. We will provide updates as soon as they become available. If you have an urgent matter, please reach out to us via the support portal (support.instaclustr.com) or email ([email protected]).

The issue has been identified and a fix is being implemented.

A fix has been implemented and operations such as provisioning clusters, replacing nodes and restores are now functional as-well. The Management Console and Cluster Management API are now fully operational. We are still seeing some elevated latencies with the Monitoring API and have implemented a fix for this. We will provide more updates as soon as they become available. If you have an urgent matter, please reach out to us via the support portal (support.instaclustr.com) or email ([email protected]).

This incident has been resolved.

We are currently experiencing issues with provisioning Cadence and Kafka Connect clusters on AWS when provisioning into an existing VPC through the Cluster Management API and Management Console. Provisioning a Cadence cluster into the same VPC as its linked Cassandra cluster or a Kafka Connect cluster into the same VPC as its linked Kafka cluster will fail. Our team is actively working on resolving this problem and we apologise for any inconvenience caused. Existing customer clusters are unaffected, and the Monitoring API and Prometheus API are operational. Additional details will be provided as they become available. If you have an urgent matter, please reach out to us via the support portal (support.instaclustr.com) or email ([email protected]).

This incident has been resolved.

We are currently experiencing issues with provisioning OpenSearch clusters on all providers through the Cluster Management API and Management Console. Our team is actively working on resolving this problem and we apologise for any inconvenience caused. Existing customer clusters are unaffected, and the Monitoring API and Prometheus API are operational. Additional details will be provided as they become available. If you have an urgent matter, please reach out to us via the support portal (support.instaclustr.com) or email ([email protected]).

The issue has been identified and a fix is being implemented.

A fix has been deployed and the ability to provision OpenSearch clusters has been restored.

We are currently investigating increased latencies and error rates affecting our Cluster Management API. Our team is actively working to resolve the issue, and we apologize for any inconvenience this may cause. During this time, you may experience slower load times on the Management Console and delayed API responses. Existing cluster operations are unaffected. We will provide updates as soon as they become available. If you have an urgent matter, please reach out to us via the support portal (support.instaclustr.com) or email ([email protected]).

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

We are currently experiencing delays with applying configuration and Kafka Integration changes to Clickhouse clusters via Terraform, the Cluster Management API and Management Console. During this time, making configuration changes to Clickhouse clusters may take longer than usual. Our team is actively working on resolving this problem and we apologise for any inconvenience caused. Existing customer clusters are unaffected, and the Monitoring API and Prometheus API are operational. Additional details will be provided as they become available. If you have an urgent matter, please reach out to us via the support portal (support.instaclustr.com) or email ([email protected]).

The issue has been identified and a fix is being implemented.

We have additionally identified that making changes to Kafka User via Terraform, the Cluster Management API, and the Management Console is also experiencing delays. We have implemented a fix that has significantly reduced the delays experienced when applying configuration and Kafka integration changes to Clickhouse clusters and also reduced the delays when making changes to Kafka Users on Kafka Clusters via Terraform, the Cluster Management API, and the Management Console. We will continue to monitor the results of the fix and provide additional updates as they become available. If you have an urgent matter, please reach out to us via the support portal (support.instaclustr.com) or email ([email protected]).

This incident has been resolved.

We are currently experiencing delays with provisioning clusters through the Cluster Management API and Management Console. During this time, cluster provisioning (and other related operations such as adding and removing nodes) may take longer than usual. Our team is actively working on resolving this problem and we apologise for any inconvenience caused. Existing customer clusters are unaffected, and the Monitoring API and Prometheus API are operational. Additional details will be provided as they become available. If you have an urgent matter, please reach out to us via the support portal (support.instaclustr.com) or email ([email protected]).

We are still experiencing delays with provisioning clusters through the Cluster Management API and Management Console and we have also identified API causing delays when adding/deleting Kafka Integrations. As a result these API's may take longer than usual. Due to this Customer-initiated resize via the console and API have been temporarily disabled. If you require a resize of your cluster, please contact our support team. We have identified the cause and our team is working on resolving this problem. We apologise for any inconvenience caused. Existing customer clusters are unaffected, and the Monitoring API and Prometheus API are operational. Additional details will be provided as they become available. If you have an urgent matter, please reach out to us via the support portal (support.instaclustr.com) or email ([email protected]).

A fix has been implemented and we are monitoring the results. If you have an urgent matter, please reach out to us via the support portal (support.instaclustr.com) or email ([email protected]).

This incident has been resolved.

We are currently fixing the unavailability of Microsoft Social SSO login for our Management Console. Our team is actively working to resolve the issue and we apologise for any inconvenience caused. Other login methods for Management Console are still working. The Cluster Management API, Monitoring API, Prometheus API and Support Portal are still operational. We will provide updates as soon as they become available. If you have an urgent matter, please reach out to us via the support portal (support.instaclustr.com) or email ([email protected]).

Microsoft Social SSO login for Management Console has been fixed and customers can use the Sign in with Microsoft option again.