Instaclustr outage history

We are currently experiencing service degradation and potential disruptions in the Google Cloud Platform region us-west1. This incident is a result of an ongoing issue with the cloud provider's infrastructure. You will be unable to provision new clusters or nodes. Existing nodes in us-west1 may be unavailable. Cluster datacentres in other regions are unaffected, but are experiencing networking issues when interacting with nodes in us-west1. Individual impacted customers are being contacted via support tickets. We are actively monitoring the situation and will provide additional details as they become available. GCP status is tracked at https://status.cloud.google.com/. If you have an urgent matter, please reach out to us via the support portal (support.instaclustr.com) or email ([email protected]).

We are continuing to work on a fix for this issue.

Vertical scaling has been temporarily disabled in the GCP us-west1 region due to this cloud provider outage. We will re-enable this functionality when the cloud provider outage is resolved.

Google have completed mitigations and are reporting recovery across multiple products. We will continue monitoring the incident until the remaining products have recovered. Vertical scaling remains disabled in us-west1.

This incident has been resolved.

We are currently seeing compatibility issues between the Instaclustr Terraform provider and the Cluster Management API, affecting operations involving OpenSearch clusters. Standard customer clusters operations are unaffected. The Monitoring API and Prometheus API remain operational. Our team is actively working to resolve the issue and we apologise for any inconvenience. Further updates will be provided as more information becomes available. If you have an urgent matter, please contact us through the support portal at https://support.instaclustr.com or by email at [email protected].

This incident has been resolved.

We are currently experiencing issues with provisioning private network PostgreSQL clusters with 3 nodes through the Cluster Management API and Management Console. Our team is actively working on resolving this problem and we apologise for any inconvenience caused. Existing customer clusters are unaffected, and the Monitoring API and Prometheus API are operational. Additional details will be provided as they become available. If you have an urgent matter, please reach out to us via the support portal (support.instaclustr.com) or email ([email protected]).

We are continuing to work on a fix for this issue.

This incident has been resolved.

We are currently experiencing service degradation and potential disruptions in the GCP us-east4-b zone. GCP has not updated its status page yet, however initial investigation indicates an issue likely with the cloud provider's infrastructure in the us-east4-b zone. While we are seeing partial recovery, in the meantime provisioning new clusters or nodes in the concerned GCP zone will be impacted. Also existing nodes in the GCP us-east4-b zone can face intermittent issues and be offline. You should still be able to access your clusters if you are running a standard configuration. Individual impacted customers are being contacted via support tickets. We are actively monitoring the situation and will provide additional details as they become available. GCP status is tracked at https://status.cloud.google.com/. If you have an urgent matter, please reach out to us via the support portal (support.instaclustr.com) or email ([email protected]).

This incident has been resolved.

We are currently experiencing issues with provisioning Kafka Connect & Cadence clusters into existing VPCs through the Cluster Management API and Management Console. Our team is actively working on resolving this problem and we apologise for any inconvenience caused. Existing customer clusters are unaffected, and the Monitoring API and Prometheus API are operational. Additional details will be provided as they become available. If you have an urgent matter, please reach out to us via the support portal (support.instaclustr.com) or email ([email protected]).

The issue has been identified and a fix is being implemented.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

We are currently investigating an issue causing cluster level Prometheus monitoring query requests to be incorrectly blocked with a 403 error. Other Prometheus monitoring queries are unaffected. Our team is actively working to resolve the issue, and we apologise for any inconvenience caused. The Management Console, Cluster Management API and Monitoring API are still operational. We will provide updates as soon as they become available. If you have an urgent matter, please reach out to us via the support portal (support.instaclustr.com) or email ([email protected]).

The issue has been identified, and our team is actively working on implementing a fix. Cluster-level Prometheus monitoring query requests are being incorrectly blocked with a 403 error. Other Prometheus monitoring queries are unaffected. Existing customer clusters are unaffected. The Management Console and Cluster Management API are still operational. We will provide updates as soon as they become available. If you have an urgent matter, please reach out to us via the support portal (support.instaclustr.com) or email ([email protected]).

A fix is being rolled out and we are monitoring the results.

This incident has been resolved.

We are currently investigating elevated latencies and error rates with our Monitoring API. Our team is actively working to resolve the issue and we apologise for any inconvenience caused. Existing customer clusters are unaffected. The Management Console and Cluster Management API are still operational. We will provide updates as soon as they become available. If you have an urgent matter, please reach out to us via the support portal (support.instaclustr.com) or email ([email protected]).

This incident has been resolved.

We are currently experiencing service degradation and potential disruptions in the AWS region US-EAST-1. This incident is a result of an ongoing issue with the cloud provider's infrastructure. Customers using the AWS availability zone "use1-az4" are impacted, with nodes in this availability zone observed to become unavailable. Clusters are expected to be available overall, and not significantly impacted. We are actively monitoring the situation and will provide additional details as they become available. AWS status is tracked at https://health.aws.amazon.com/health/status. If you have an urgent matter, please reach out to us via the support portal (support.instaclustr.com) or email ([email protected]).

We are continuing to monitor the issue.

This incident has been resolved.

We are currently investigating elevated latencies and error rates with our Monitoring API. Our team is actively working to resolve the issue and we apologise for any inconvenience caused. Existing customer clusters are unaffected. The Management Console and Cluster Management API are still operational. We will provide updates as soon as they become available. If you have an urgent matter, please reach out to us via the support portal (support.instaclustr.com) or email ([email protected]).

This incident has been resolved.

We are currently experiencing issues with provisioning Cadence and Kafka Connect clusters on AWS when provisioning into a Private Network through the Cluster Management API and Management Console. Our team is actively working on resolving this problem and we apologise for any inconvenience caused. Existing customer clusters are unaffected, and the Monitoring API and Prometheus API are operational.  Additional details will be provided as they become available. If you have an urgent matter, please reach out to us via the support portal (support.instaclustr.com) or email ([email protected]).

A fix for this issue has been deployed.

We are currently experiencing issues with provisioning clusters through the Cluster Management API and Management Console. Our team is actively working on resolving this problem and we apologise for any inconvenience caused. In addition, operations on existing clusters such as replacing nodes and restores aren't functional, and we are investigating elevated latencies with the Monitoring API. Additional details will be provided as they become available. If you have an urgent matter, please reach out to us via the support portal (support.instaclustr.com) or email ([email protected]).

We are further seeing elevated latencies and error rates with our Monitoring API and Cluster Management API and additionally seeing issues logging into our Management Console. Our team is actively working to resolve the issue and we apologise for any inconvenience caused. Existing customer clusters are unaffected. The Management Console and Cluster Management API are partially degraded at this time. We will provide updates as soon as they become available. If you have an urgent matter, please reach out to us via the support portal (support.instaclustr.com) or email ([email protected]).

The issue has been identified and a fix is being implemented.

A fix has been implemented and operations such as provisioning clusters, replacing nodes and restores are now functional as-well. The Management Console and Cluster Management API are now fully operational. We are still seeing some elevated latencies with the Monitoring API and have implemented a fix for this. We will provide more updates as soon as they become available. If you have an urgent matter, please reach out to us via the support portal (support.instaclustr.com) or email ([email protected]).

This incident has been resolved.

We are currently experiencing issues with provisioning Cadence and Kafka Connect clusters on AWS when provisioning into an existing VPC through the Cluster Management API and Management Console. Provisioning a Cadence cluster into the same VPC as its linked Cassandra cluster or a Kafka Connect cluster into the same VPC as its linked Kafka cluster will fail. Our team is actively working on resolving this problem and we apologise for any inconvenience caused. Existing customer clusters are unaffected, and the Monitoring API and Prometheus API are operational. Additional details will be provided as they become available. If you have an urgent matter, please reach out to us via the support portal (support.instaclustr.com) or email ([email protected]).

This incident has been resolved.

We are currently experiencing issues with provisioning OpenSearch clusters on all providers through the Cluster Management API and Management Console. Our team is actively working on resolving this problem and we apologise for any inconvenience caused. Existing customer clusters are unaffected, and the Monitoring API and Prometheus API are operational. Additional details will be provided as they become available. If you have an urgent matter, please reach out to us via the support portal (support.instaclustr.com) or email ([email protected]).

The issue has been identified and a fix is being implemented.

A fix has been deployed and the ability to provision OpenSearch clusters has been restored.

We are currently investigating increased latencies and error rates affecting our Cluster Management API. Our team is actively working to resolve the issue, and we apologize for any inconvenience this may cause. During this time, you may experience slower load times on the Management Console and delayed API responses. Existing cluster operations are unaffected. We will provide updates as soon as they become available. If you have an urgent matter, please reach out to us via the support portal (support.instaclustr.com) or email ([email protected]).

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

We are currently experiencing delays with applying configuration and Kafka Integration changes to Clickhouse clusters via Terraform, the Cluster Management API and Management Console. During this time, making configuration changes to Clickhouse clusters may take longer than usual. Our team is actively working on resolving this problem and we apologise for any inconvenience caused. Existing customer clusters are unaffected, and the Monitoring API and Prometheus API are operational. Additional details will be provided as they become available. If you have an urgent matter, please reach out to us via the support portal (support.instaclustr.com) or email ([email protected]).

The issue has been identified and a fix is being implemented.

We have additionally identified that making changes to Kafka User via Terraform, the Cluster Management API, and the Management Console is also experiencing delays. We have implemented a fix that has significantly reduced the delays experienced when applying configuration and Kafka integration changes to Clickhouse clusters and also reduced the delays when making changes to Kafka Users on Kafka Clusters via Terraform, the Cluster Management API, and the Management Console. We will continue to monitor the results of the fix and provide additional updates as they become available. If you have an urgent matter, please reach out to us via the support portal (support.instaclustr.com) or email ([email protected]).

This incident has been resolved.

We are currently experiencing delays with provisioning clusters through the Cluster Management API and Management Console. During this time, cluster provisioning (and other related operations such as adding and removing nodes) may take longer than usual. Our team is actively working on resolving this problem and we apologise for any inconvenience caused. Existing customer clusters are unaffected, and the Monitoring API and Prometheus API are operational. Additional details will be provided as they become available. If you have an urgent matter, please reach out to us via the support portal (support.instaclustr.com) or email ([email protected]).

We are still experiencing delays with provisioning clusters through the Cluster Management API and Management Console and we have also identified API causing delays when adding/deleting Kafka Integrations. As a result these API's may take longer than usual. Due to this Customer-initiated resize via the console and API have been temporarily disabled. If you require a resize of your cluster, please contact our support team. We have identified the cause and our team is working on resolving this problem. We apologise for any inconvenience caused. Existing customer clusters are unaffected, and the Monitoring API and Prometheus API are operational. Additional details will be provided as they become available. If you have an urgent matter, please reach out to us via the support portal (support.instaclustr.com) or email ([email protected]).

A fix has been implemented and we are monitoring the results. If you have an urgent matter, please reach out to us via the support portal (support.instaclustr.com) or email ([email protected]).

This incident has been resolved.

We are currently fixing the unavailability of Microsoft Social SSO login for our Management Console. Our team is actively working to resolve the issue and we apologise for any inconvenience caused. Other login methods for Management Console are still working. The Cluster Management API, Monitoring API, Prometheus API and Support Portal are still operational. We will provide updates as soon as they become available. If you have an urgent matter, please reach out to us via the support portal (support.instaclustr.com) or email ([email protected]).

Microsoft Social SSO login for Management Console has been fixed and customers can use the Sign in with Microsoft option again.

We are currently investigating some delayed aggregated metrics on our Monitoring API. Our team is actively working to resolve the issue and we apologise for any inconvenience caused. This issue means that the "last 3 hours" and "last 6 hours" view for metrics on the console (and the monitoring API) are not working. Realtime metrics, and viewing metrics over a longer duration still works. We will provide updates as soon as they become available. If you have an urgent matter, please reach out to us via the support portal (support.instaclustr.com) or email ([email protected]).

We've identified the issue and have implemented a fix. We have monitored a decrease in the delay for these aggregated metrics. We will continue to monitor the issue while the backlog of metrics are processed.

We have monitored a further decrease in the delay for these aggregated metrics. We will continue to monitor the issue while more of the backlog of metrics are processed. As a result the "last 3 hours" and "last 6 hours" view for metrics on the console (and the monitoring API) are now showing a limited set of metrics. Realtime metrics, and viewing metrics over a longer duration still works. We will provide more updates as soon as they become available. If you have an urgent matter, please reach out to us via the support portal (support.instaclustr.com) or email ([email protected]).

We have monitored an even further decrease in the delay for these aggregated metrics. Most of the backlog of metrics have been processed. As a result the "last 3 hours" view for metrics on the console (and the monitoring API) may not show some metrics. Realtime metrics, and viewing metrics over a longer duration still works. We will provide more updates as soon as they become available. If you have an urgent matter, please reach out to us via the support portal (support.instaclustr.com) or email ([email protected]).

This incident has been resolved.

Multiple Instaclustr systems have been affected by an incident which is affecting multiple AWS services in us-east-1 During this incident, cluster provisioning operations will not work. Existing clusters and our monitoring systems are unaffected. We have disabled some platform functionality including customer-initated resizes to limit the impact of this issue. We are continuing to monitor the incident and will restore disabled functionality when the issue is resolved. For more information, refer to https://status.instaclustr.com or contact [email protected]

Multiple Instaclustr systems have been affected by an incident which is affecting AWS DynamoDB and related services in us-east-1. Many systems are showing signs of recovery, however we are continuing to see some issues affecting provisioning operations. We are continuing to investigate these issues Cluster resizes and node replaces remain disabled. For more information, refer to https://status.instaclustr.com or contact [email protected]

We are continuing to see issues with provisioning operations in AWS us-east-1. Operations in other regions appear to be restored, but resizes and replaces will remain disabled until the issues in us-east-1 are confirmed to be resolved. We are continuing to monitor these issues. For more information, refer to https://status.instaclustr.com or contact [email protected]

We are continuing to see issues with provisioning operations in AWS us-east-1. Operations in other regions appear to be restored, but resizes and replaces will remain disabled until the issues in us-east-1 are confirmed to be resolved. We are continuing to monitor these issues and will re-evaluate when operations in the region become more reliable. For more information, refer to https://status.instaclustr.com or contact [email protected]

We are continuing to see issues with provisioning operations in AWS us-east-1. Operations in other regions appear to be restored, but resizes and replaces will remain disabled until the issues in us-east-1 are confirmed to be resolved. At this time AWS is still reporting a large number of impacted services, and EC2 operations are being rate-limited. We are continuing to monitor these issues and will re-evaluate when operations in the region become more reliable. For more information, refer to https://status.instaclustr.com or contact [email protected]

Update - We are seeing recovery for provisioning operations in AWS us-east-1. Operations in other regions appear to be restored, but resizes and replaces will remain disabled until the issues in us-east-1 are confirmed to be resolved. At this time AWS is still reporting some throttling of EC2 instance launches, though it has been reduced. We are continuing to monitor these issues and will re-evaluate when operations in the region become more reliable. For more information, refer to https://status.instaclustr.com or contact [email protected]

Update - We have observed stable provisioning operations in AWS us-east-1, and AWS has reported all AWS services have returned to normal operations. Resize and replace operations have been restored on the Instaclustr Managed Platform, and no additional impact is remaining from this incident.

Update - We have observed stable provisioning operations in AWS us-east-1, and AWS has reported all AWS services have returned to normal operations. Resize and replace operations have been restored on the Instaclustr Managed Platform, and no additional impact is remaining from this incident.

We are experiencing issues with downloading certificates for Kafka clusters via the Instaclustr Console and API. This affects both new and existing Kafka-only clusters downloads. Other application clusters remain unaffected by this issue Our team is actively investigating the problem and working on a resolution. We will provide updates as soon as more information becomes available.If you have an urgent matter, please reach out to us via the support portal (support.instaclustr.com) or email ([email protected]).

The issue has been identified and a fix is being implemented.

We are continuing to investigate this issue.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.