Aiven outage history

We are investigating an ongoing Google Cloud VPN incident affecting the europe-west2 and africa-south1 regions. This is impacting network connectivity between nodes and may affect the availability and performance of clustered Aiven services in the affected regions. Our team is investigating the impact and monitoring affected services. We will provide further updates as more information becomes available.

We are investigating an ongoing Google Cloud networking incident. Google Cloud engineers have confirmed an issue with Virtual Private Cloud (VPC) networking affecting communication between virtual machines, including communication over internal IP addresses, IPv6, and other networking paths. ``` The incident is currently reported to affect the following regions: Africa: africa-south1 Europe: europe-central2, europe-north1, europe-north2, europe-southwest1, europe-west1, europe-west2, europe-west3, europe-west4, europe-west6, europe-west8, europe-west9, europe-west10, and europe-west12 ``` This issue is impacting network connectivity between nodes and may affect the availability and performance of clustered Aiven services in the affected regions. Our team is investigating the impact and monitoring affected services. We will provide further updates as more information becomes available.

The latest information identifies europe-west2 and the europe-west2-b zone as affected locations. The incident involves Virtual Private Cloud (VPC) networking issues that can disrupt communication between virtual machines. This may affect connectivity between nodes and the availability of clustered Aiven services in the affected location. Our team continues to investigate the impact on affected services and monitor their health. We will share further updates as more information becomes available.

The latest information identifies `europe-west2` and the `europe-west2-b` zone as affected locations. The incident involves Virtual Private Cloud (VPC) networking issues that are disrupting communication between virtual machines in the affected location, which is impacting the availability of clustered Aiven services — currently Aiven for Apache Kafka and Aiven for OpenSearch. We continue to monitor our services to identify any remaining impacted services and we are waiting for connectivity to be restored from Google side, and for the affected nodes to return. We will share further updates as more information becomes available.

We are continuing to respond to the Google Cloud incident affecting the europe-west2 region. The underlying Google Cloud issue — a power outage in the europe-west2-b zone has been mitigated by Google, and recovery of most workloads has been in progress since 15:10 EEST. Services running in the affected region may be unavailable or degraded while recovery is in progress. We are seeing the recovery of services and are prioritizing the recovery of existing workloads. Customers may not be able to yet create new workloads. We are actively recovering the affected services, including recycling affected Kafka nodes and working through the OpenSearch and PostgreSQL services that need additional steps. We are monitoring the situation closely and will continue until all affected services are healthy.

All PG and Kafka services known to be affected by the Google Cloud incident in europe-west2 have been recovered. We are currently recovering and fixing known Opensearch services. We are continuing to monitor all services in the region to confirm they remain healthy. If you are still experiencing issues with a service in europe-west2, please contact Aiven Support.

All Aiven services known to be affected by the Google Cloud incident in europe-west2 have been recovered. After Google mitigated the power outage in the europe-west2-b zone, we restored the affected Aiven for Apache Kafka, Aiven for OpenSearch and Aiven for PostgreSQL services by recovering nodes or replacing them. We are continuing to monitor all services in the region to confirm they remain healthy. If you are still experiencing issues with a service in europe-west2, please contact Aiven Support.

This incident has been resolved. A power outage in Google Cloud's europe-west2-b zone disrupted services running in the europe-west2 region. Nodes in the affected zone were rebooted when the zone recovered. Some affected Aiven for Apache Kafka, Aiven for OpenSearch, Aiven for PostgreSQL and Aiven for Valkey services needed extra recovery steps or node replacement before they returned to normal operation. All affected services have now been restored. We apologise for the disruption.

Creation and replacement of nodes in the **vultr-ewr** cloud region are experiencing issues and may take longer than normal to come up. Services that are already running are not affected. Our engineering team is actively working on this, and we will provide another update as soon as we have more information.

We have resolved the underlying issues of nodes creation and replacement in the **vultr-ewr** cloud region. Should any further issues be encountered, please reach out to support via the Aiven console. We apologise for the inconvenience caused by this issue, and commit to analysing and improving our platform to avoid future incidents similar to this one.

We are currently experiencing an issue with starting new service nodes for almost all service types. The running nodes are not affected. Our engineering team is actively working on a fix and we will provide another update as soon as we have more information.

A fix has been implemented and we are currently monitoring the results. New service nodes are able to start again, and running nodes were not affected at any point. We will provide further updates once we have confirmed the issue is fully resolved.

This incident has been resolved. New service nodes are able to start normally again, and running nodes were not affected at any point during the incident. We apologize for any inconvenience this may have caused.

We have issues in the google asia-southeast1 region where we cannot spin up nodes. This is due to an upstream cloud provider issue that we are working on. We are currently investigating and will update further details in the next 30 minutes.

We are investigating an issue preventing node creation in asia-southeast1. The issue is only affecting some users. Engineering is actively engaged with GCP support. Next update in 30 minutes.

We have identified the root cause of the issue preventing node creation in asia-southeast1. Our team is working with GCP to implement a fix. We will provide another update as soon as more information becomes available.

We have resolved the underlying cause of this issue in Google's asia-southest1 region. Should any further issues be encountered, please reach out to support via the Aiven console. We apologise for the inconvenience caused by this issue.

We are currently investigating an issue affecting Aiven for PostgreSQL services, where a major version upgrade can leave the service in an unusable state. We advise customers not to initiate a PostgreSQL major version upgrade at this time. If you have already started a major version upgrade and your service is not responding, please contact our support team at [email protected]. Our engineering teams are actively working to identify the root cause and restore affected services. We will provide further updates as soon as more information is available. We apologize for the inconvenience.

We have a mitigation that we are rolling out for the issue affecting Major version upgrade on Aiven for PostgreSQL services. Please continue to refrain from triggering major version upgrades for the moment. Affected services are being restored as the mitigation is deployed. We will provide further updates as the rollout progresses. We apologize for the inconvenience.

PostgreSQL major version upgrades can be performed again. A small number of services affected by an earlier failed upgrade are now being repaired. We will contact customers directly if any action is needed on their side, and we'll post another update here once the repairs are complete.

All customer services affected by this issue have been restored, and no further action is needed on your side. It is now safe to proceed with PostgreSQL major version upgrades. We apologise for the disruption this caused.

All customer services affected by this issue have been restored, and no further action is needed on your side. It is now safe to proceed with PostgreSQL major version upgrades. We apologise for the disruption this caused.

We are aware of an issue affecting the Aiven API. The Aiven Console, the Aiven CLI, the Terraform provider and any custom scripts or integrations all run through this same API, so customers may see errors across any of them. You may experience being unable to load or sign in to the Console, and failures when creating, modifying or querying services. Our engineering team is investigating and work to restore service is underway. We will provide a further update within 30 minutes, or sooner if there is news to share. We apologise for the disruption.

We have identified the cause of this issue and a fix is now being rolled out across our API nodes in stages. The Aiven Console and API are already working again for some customers. Until the rollout completes you may still see intermittent sign-in failures, or errors when creating, modifying or querying services via the Console, CLI, Terraform provider or direct API calls. Retrying will often succeed. We will confirm here once the rollout is complete. Thank you for your patience.

The rollout of the fix is still progressing across our API nodes. Most customers should now find the Aiven Console, API, CLI and Terraform provider working normally, and we expect any remaining errors to clear as the rollout completes. We will confirm here once it is complete. Thank you for your patience.

This incident is now resolved. Between approximately 00:10 and 02:05 UTC on 22 September, an issue affecting our public API endpoints caused sign-in failures and errors for customers using the Aiven Console, the Aiven API, the CLI, the Terraform provider and other API-based integrations. The issue was corrected and we have seen no further errors during our monitoring period. We are following up with additional changes to prevent this from recurring. We apologise for the disruption. If you are still seeing problems, please contact Aiven support.

This incident is now resolved. Earlier today a certificate problem affecting our public API endpoints caused sign-in failures and errors for customers using the Aiven Console, the Aiven API, the CLI, the Terraform provider and other API-based integrations. A corrected certificate has been in place across all API nodes since 02:05 UTC, and we have seen no further errors during our monitoring period. We are following up with a permanent fix to prevent this from recurring. We apologize for the disruption. If you are still seeing problems, please contact Aiven support at [[email protected]](mailto:[email protected])

We are currently investigating reports of users being unable to log in to the Aiven console. We will share further updates as soon as possible.

We are still investigating an issue affecting the Aiven Console. Customers are currently unable to create or submit new support tickets directly through the portal. If you require assistance with an urgent issue unrelated to the Aiven Console being down, please send an email to [[email protected]](mailto:[email protected]). Our support team will process your request directly.

A fix has been deployed for Aiven Console and portal functionality is restored. We are currently monitoring performance. If you encounter any issues, please email [[email protected]](mailto:[email protected])

This incident has been resolved. Aiven Console login and support ticket creation through the portal are fully functional again. We continued to monitor performance and have seen no further issues. If you encounter any problems, please email [[email protected]](mailto:[email protected]).

We are investigating issues with creating new nodes for services on the Aiven platform currently. The effect is slower creation time for new nodes. Any any event where a new node is created are impacted such as failovers, switchovers, upgrades and maintenance updates. Our engineers are currently investigating the root cause and working on a fix for the issue. We apologise for the inconvenience caused by this issue. We will provide a further update as our team continues on the solution for this situation.

We are monitoring our mitigating actions to issues with creating new nodes for services on the Aiven platform. Our engineers have done mitigating steps and we see improvements in the node creation, but we'll continue investigating and monitoring of the issue. We apologise for the inconvenience caused by this issue. We will provide a further update as our team continues monitoring and triaging this situation.

We have completed our mitigating actions and the creation of new nodes for services on the Aiven platform has returned to normal. Customers can now proceed with any operations that were delayed during this issue, including service creation, scaling, upgrades, maintenance updates and configuration changes. Operations that were left pending should continue automatically. Our engineers continue to monitor the platform to confirm the issue is fully resolved. We apologise for the inconvenience caused by this issue. We will provide a further update as our team completes monitoring of this situation.

We have resolved the underlying cause of the new node creation and new nodes are able to be created without issues. Should any further issues be encountered, please reach out to support via the Aiven console. We apologise for the inconvenience caused by this issue, and commit to analysing and improving our platform to avoid future incidents similar to this one.

We are currently experiencing delays in the creation of new virtual machines, which may affect the provisioning of new services and nodes. An ongoing incident with an upstream cloud provider (DigitalOcean) is preventing new VM creation in all DigitalOcean regions. Additionally, we are observing delays in new VM creation across all other cloud providers. **Impact**: **DigitalOcean**: New VM creation is currently unavailable in all DO regions. **All other clouds**: New VM creation may be delayed longer than usual. Existing services are not affected and continue to run normally. This issue impacts only the creation of new VMs (e.g., new service provisioning, node replacements, and scaling operations that require new VMs). Our engineering team is actively monitoring the situation and working to reduce the provisioning backlog. We are tracking the upstream provider's recovery and will restore normal provisioning times as soon as possible. We apologize for the inconvenience caused by this issue.

New VM creation across most cloud providers is now recovering, and provisioning times are slowly returning to normal. Services that were previously delayed should now be creating successfully soon. **Current status** • **All clouds except DigitalOcean:** New VM creations are slowly returning to normal pace. We are continuing to monitor to confirm provisioning times remain stable. • **DigitalOcean:** New VM creation remains unavailable in DigitalOcean regions due to the ongoing upstream provider incident. We are tracking the provider's recovery and will restore normal provisioning as soon as it is resolved. Our engineering team is actively monitoring provisioning performance across all clouds and continues to track the DigitalOcean provider's recovery. Thank you for your patience.

New VM creation has returned to normal across all cloud providers, including DigitalOcean. * VM provisioning is operating normally across all regions. * The upstream DigitalOcean incident has been resolved. * We are continuing to monitor the infrastructure to ensure stable provisioning. The incident is now resolved. Thank you for your patience.

We are investigating an issue affecting Dynamic Disk Sizing (DDS), where resize operations may fail for a small subset of newly created nodes. Mitigations are currently underway.

Issue has been identified and fixes are being applied. Thank you for your patience.

Our team has successfully applied the appropriate mitigation to all impacted nodes, and the fix has been deployed to prevent future occurrences.

We are currently investigating an issue affecting services running on DigitalOcean that is causing delays in node lifecycle management. As a result, any operations on DigitalOcean-hosted services that require creating, deleting, or replacing nodes will experience delays. Impacted operations (on DigitalOcean only) include: • Creating new services • Deleting existing services • Powering services up or down • Scheduled maintenance updates • Service scaling and plan upgrades Note: Existing, running services on DigitalOcean are currently unaffected and operating normally. Services running on other cloud providers (AWS, GCP, Azure, UpCloud) are completely unaffected. Our engineering team is actively working to resolve the issue, and we will provide further updates as soon as more information is available. We apologize for the inconvenience.

We have identified that the previously reported issue affecting node lifecycle management is not isolated to a single cloud provider, but is currently affecting services across all cloud providers. As a result, any operations that require creating, or replacing nodes will experience significant delays regardless of the underlying cloud hosting platform. We have stopped all pending maintenance updates to help mitigate the impact. Impacted operations across all providers include: - Creating new services - Powering services up - Scheduled maintenance updates - Service scaling and plan upgrades Note: Aiven for MySQL services are completely unaffected by this issue. For all other service types, existing, running services remain unaffected and continue to operate normally. Our engineering teams are actively working to resolve the root cause across our infrastructure, and we will provide further updates as the situation develops. We apologize for the continued inconvenience.

``` We have implemented a mitigation for the infrastructure issue affecting node lifecycle management, and we are now seeing service operations return to normal. We are closely monitoring the platform to ensure stability. Pending operations such as service creations, deletions, and maintenance updates are beginning to process, though it may take some time to clear the remaining backlog. ```

We have resolved the underlying cause of this issue. Should any further issues be encountered, please reach out to support via the Aiven console. We apologise for the inconvenience caused by this issue, and commit to analysing and improving our platform to avoid future incidents similar to this one.

We are currently investigating an issue affecting Terraform & Kubernetes operator users receiving `400` errors when using `aiven_pg_database`. This is impacting `aiven_pg_database` and potentially other service types. The root cause has been identified and a fix is in progress. We will provide an ETA as soon as it is available.

A fix has been applied and the issue is expected to be resolved. We are continuing to monitor closely as we work to restore full system operation. We apologise for the inconvenience caused by this issue.

The issue has been resolved. Terraform & Kubernetes operator users should no longer experience `400` errors when using `aiven_pg_database`. We apologize for any inconvenience caused.

We have paused Kafka 3.8 ->3.9 upgrades as we are investigating an issue with KRaft migration during the upgrade. Aiven is actively monitoring affected services and applying mitigations where required. Impact: Existing Kafka services continue to run. Customers attempting to upgrade Kafka services from version 3.8 to 3.9 will not be able to complete the upgrade while KRaft migrations are paused. Next update: We will provide an update once more information is available.

We have identified two separate problems: • An issue in migration code • An [upstream bug](https://github.com/apache/kafka/pull/19854) We are working on the fix for the migration issue and back-porting the fix for the upstream bug. If you have any issues, please contact support.

We have identified issues affecting KRaft migrations (Kafka 3.8 to 3.9 upgrades). We have paused all migrations while we work on a fix. If you are experiencing any issues, please contact support.

We have deployed fixes for the issues affecting KRaft migrations during Kafka 3.8 to 3.9 upgrades, and migrations have been re-enabled. We are continuing to monitor the situation to ensure services remain stable. If you are experiencing any issues, please contact support.

The issues affecting KRaft migrations during Kafka 3\.8 to 3\.9 upgrades have been resolved. Fixes have been deployed, migrations have been re-enabled, and services have remained stable during monitoring. If you are experiencing any issues, please contact support.

We have identified an issue affecting KRaft migrations during Kafka upgrades to version 3.9, and have temporarily paused those upgrades while we investigate. If you try to upgrade now, your request will be rejected with an error asking you to contact support. We will re-enable upgrades once the issue is resolved. **Impact:** Running Kafka services are not affected and continue to operate normally. Customers who attempt to upgrade a Kafka service to version 3.9 or later will receive an `HTTP 403` error and the upgrade will not proceed. No data or service availability is impacted.

We have a fix prepared, which is currently undergoing final verification. If validation is successful, we will proceed with deployment shortly. As a result, the incident is now moving into a monitoring phase while we observe system stability after the changes. We will re-enable Kafka upgrades to version 3.9 once we confirm the issue is fully resolved.

The issue has been resolved. All affected functionality is operating normally.

Aiven engineers are currently investigating a bug on Aiven for PostgreSQL that causes affected cluster fail to take backups and restore from backup. Known affected clouds are AWS and DigitalOcean. Thank you for your patience as we continue our work on this incident.

Our team has identified a potential root cause and is working to fix the issue on the affected services.

The fix has been applied to the affected services, and Aiven will continue to monitor to confirm that no further action is needed. Thank you for your patience.

Correction: Aiven for Apache Kafka services as well as Aiven for PostgreSQL services were affected. All services identified to have the issue have been fixed as per the previous status message.

The monitoring period has passed without further evidence of the issue occurring. Incident has been declared resolved.

We are experiencing an issue affecting service operations across Azure regions. While existing services remain up and running, operations such as service creation, plan changes, and scheduled maintenance are currently failing. We are actively working on a fix to restore full functionality as quickly as possible. We will continue to update here on the progress of this issue.

``` We have identified the root cause and are currently deploying a fix to resolve the issues affecting service operations across Azure regions.While existing services remain up and running, operations such as service creation, plan changes, and maintenance updates are currently failing. We are monitoring the deployment closely to ensure full functionality is restored as quickly as possible.We will continue to update here on the progress of this issue. ```

We have resolved the underlying cause of the issue affection service operations in our Azure regions. Should any further issues be encountered, please reach out to support via the Aiven console. We apologise for the inconvenience caused by this issue, and commit to analysing and improving our platform to avoid future incidents similar to this one.

``` We are experiencing performance degradation and node unresponsiveness affecting some Aiven services hosted in the DigitalOcean NYC3 region.This is tied to an underlying infrastructure issue currently being tracked by our cloud provider. You can monitor their infrastructure health directly on the DigitalOcean Status Page: https://status.digitalocean.com/incidents/spgd2scvttthOur engineering team is actively investigating the extent of the impact on our services and working to mitigate issues where possible. ```

This issue is now resolved. Should any further issues be encountered, please reach out to support via the Aiven console. We apologise for the inconvenience caused by this issue.

``` We are currently aware of an issue affecting Aiven for Kafka services. Our teams are currently investigating the root cause as a priority. We will continue to provide status updates as they become available. ```

We have identified the root cause of the issue affecting multiple Aiven for Apache Kafka services and are currently working to restore the affected services. Our teams are actively applying fixes and will continue to provide updates as progress is made.

``` We have applied a fix to the affected services and are seeing recovery progress. Our team is closely monitoring the platform and will continue to mitigate any remaining impact if needed. ```

Our team has seen the impacted services recover, and we are now closely monitoring for any remaining impact.

We have resolved the underlying cause of the kafka service disruption. Should any further issues be encountered, please reach out to support via the Aiven console. We apologise for the inconvenience caused by this issue, and commit to analysing and improving our platform to avoid future incidents similar to this one.

Amazon Web Services (AWS) currently has an outage that is impacting deployments of services in the ME-CENTRAL-1 Region (mec1-az2) specifically. Aiven provides the capability to let users move their workloads off this region, and we would recommend that users do so. For more information please see the AWS status page: https://health.aws.amazon.com/health/status

Amazon Web Services (AWS) currently has an outage that is impacting deployments of services in the ME-CENTRAL-1 Region. Aiven provides the capability to let users move their workloads off this region, and we would recommend that users do so. For more information please see the AWS status page: https://health.aws.amazon.com/health/status

Amazon Web Services (AWS) currently has an outage that is impacting deployments of services in the ME-CENTRAL-1 Region. We have disabled the creation of new nodes and services in this region. AWS has advised that users run their Disaster Recovery Plans for AWS products and services in that region and we are recommending the same. For more information please see the AWS status page: [https://health.aws.amazon.com/health/status](https://health.aws.amazon.com/health/status "https://health.aws.amazon.com/health/status")

Amazon Web Services (AWS) currently has an outage that is impacting deployments of services in the ME-CENTRAL-1 and ME-SOUTH-1 regions. We have disabled the creation of new nodes and services in both of these regions. AWS has advised that users run their Disaster Recovery Plans for AWS products and services in both of these regions and we are recommending the same. For more information please see the AWS status page: https://health.aws.amazon.com/health/status

The underlying issues with Amazon Web Services (AWS) in the **ME-CENTRAL-1** and **ME-SOUTH-1** regions are ongoing, and AWS continues to work on recovery. We have disabled the creation of new services and nodes in the affected regions to prevent further impact. We recommend our customers to migrate their services to a stable region as part of their disaster recovery plan. As there are no further actions we can take at this time, we are moving this incident to a resolved state on our status page. The recovery of these regions is fully dependent on AWS. We will make **ME-CENTRAL-1** and **ME-SOUTH-1** regions available again for Aiven services once AWS confirms they are fully restored and stable. We recommend customers continue to monitor the official AWS Health Status Page for updates. https://health.aws.amazon.com/health/status

We are currently experiencing issues with node creation on the UpCloud cloud platform. This may result in longer than usual start-up times for new services or replacement nodes in all UpCloud regions. Our engineering team is actively investigating the issue with UpCloud to resolve it as quickly as possible.

Node creation operations are now performing within normal parameters across all UpCloud regions. Our team is currently working through residual issues to ensure full stability. We will continue to monitor the situation closely and provide further updates.

The issue affecting node creation on the UpCloud cloud platform has been fully resolved. New services and replacement nodes are now provisioning successfully across all UpCloud regions. Our engineering team has completed all remediation steps and confirmed the stability of the platform.