Ιστορικό διακοπών του Cloud 66

1. 24 Αυγούστου 2026 στις 7:08 μ.μ. UTCidentified

   Το εύρος IPv6 του DigitalOcean φαίνεται να εμποδίζεται από την πρόσβαση στην αποθήκευση του Google Cloud. Αυτό σημαίνει ότι οι πελάτες CSv1 και CSv2 που έχουν διακομιστές με δημόσιες διευθύνσεις IPv6 αδυνατούν να τραβήξουν νέες εικόνες BuildGrid στους διακομιστές τους.

2. 24 Αυγούστου 2026 στις 7:24 μ.μ. UTCidentified

   Το GCS επιβεβαίωσε ότι ένα εσωτερικό ζήτημα προκαλεί την παρεμπόδιση της εμβέλειας του DigitalOcean IPv6 και εργάζεται επί του παρόντος για την επίλυση ενός ψηφίσματος.

3. 25 Αυγούστου 2026 στις 4:06 μ.μ. UTCidentified

   Ακόμη περιμένουμε την GCS για την επίλυση του ζητήματος.

Αυτόματη μετάφραση από την επίσημη ενημέρωση του συμβάντος.

1. 24 Αυγούστου 2026 στις 4:24 μ.μ. UTCidentified

   Το εύρος IPv6 του DigitalOcean φαίνεται να εμποδίζεται από την πρόσβαση στην αποθήκευση του Google Cloud. Αυτό σημαίνει ότι οι πελάτες CSv1 και CSv2 που έχουν διακομιστές με δημόσιες διευθύνσεις IPv6 αδυνατούν να τραβήξουν νέες εικόνες BuildGrid στους διακομιστές τους.

2. 24 Αυγούστου 2026 στις 7:06 μ.μ. UTCidentified

   Συνεχίζουμε να εργαζόμαστε πάνω σε μια λύση για αυτό το ζήτημα.

Αυτόματη μετάφραση από την επίσημη ενημέρωση του συμβάντος.

1. 12 Ιουνίου 2025 στις 7:09 μ.μ. UTCinvestigating

   Google authentication issues is having issues causing login and access issues for our systems and customers. We are investigating the impact and mitigation strategies.

2. 12 Ιουνίου 2025 στις 8:30 μ.μ. UTCresolved

   The underlying issue with GCP seems to be resolved, and we are not seeing further issues with the logins and other systems.

1. 2 Ιανουαρίου 2025 στις 12:24 μ.μ. UTCinvestigating

   We are aware of problems when scaling/creating Kubernetes clusters. This appears to be caused by new rate limits imposed by docker hub. The team are working on a solution.

2. 2 Ιανουαρίου 2025 στις 12:25 μ.μ. UTCidentified

   The issue has been identified and a fix is being implemented.

3. 2 Ιανουαρίου 2025 στις 12:25 μ.μ. UTCinvestigating

   We are currently investigating this issue.

4. 2 Ιανουαρίου 2025 στις 3:45 μ.μ. UTCresolved

   This incident has been resolved.

1. 28 Ιουνίου 2024 στις 1:35 π.μ. UTCinvestigating

   We are currently investigating a number of stuck deployments.

2. 28 Ιουνίου 2024 στις 1:50 π.μ. UTCmonitoring

   We have identified the issue causing the deployments to be stuck and fixed the issue. This was caused by a database failure at our cloud provider which we are investigating and monitoring.

3. 28 Ιουνίου 2024 στις 1:46 μ.μ. UTCresolved

   This incident has now been resolved.

1. 13 Μαΐου 2024 στις 8:27 π.μ. UTCinvestigating

   We are aware of issues with our CustomConfig git repositories. This issue impacts direct git access to the CustomConfig git repository. Direct access to CustomConfig pages via the web dashboard and API is not affected. We are investigating the root cause of the problem.

2. 14 Μαΐου 2024 στις 11:48 π.μ. UTCinvestigating

   Applications are able to be deployed, but the CustomConfig backend is not available for the moment.

3. 20 Μαΐου 2024 στις 3:07 μ.μ. UTCresolved

   The CustomConfig git backend migration has been completed

1. 19 Μαρτίου 2024 στις 10:07 π.μ. UTCinvestigating

   We are investigating issues affecting some production systems.

2. 19 Μαρτίου 2024 στις 10:09 π.μ. UTCinvestigating

   We are continuing to investigate this issue.

3. 19 Μαρτίου 2024 στις 10:13 π.μ. UTCresolved

   This incident has been resolved.

1. 9 Μαΐου 2023 στις 11:45 π.μ. UTCmonitoring

   Github is experiencing some API outages, deployments don't currently appear to be affected. We are monitoring (https://www.githubstatus.com/)

2. 9 Μαΐου 2023 στις 3:21 μ.μ. UTCresolved

   This incident has been resolved.

1. 26 Απριλίου 2023 στις 3:36 μ.μ. UTCinvestigating

   Water intrusion in a data center in europe-west9 has caused a multi-cluster failure and has led to a shutdown of multiple zones. We expect general unavailability of the europe-west9 region. There is no current ETA for recovery of operations in the europe-west9 region at this time, but it is expected to be an extended outage. Customers are advised to failover to other regions if they are impacted.

2. 26 Απριλίου 2023 στις 4:03 μ.μ. UTCinvestigating

   Due to a major outage in Google Cloud (https://status.cloud.google.com/regional/europe) - Kubernetes installations may fail on installations on servers located in the same region, regardless of the cloud, due to Kubernetes images themselves being hosted on Google Cloud

3. 26 Απριλίου 2023 στις 4:37 μ.μ. UTCmonitoring

   We are waiting on Google resolutions

4. 27 Απριλίου 2023 στις 8:22 π.μ. UTCresolved

   Maestro installations are working as expected again

1. 6 Οκτωβρίου 2022 στις 1:07 μ.μ. UTCinvestigating

   Currently, GitHub is having an issue with listing repositories while using the Cloud 66 Github app. As a workaround while this issue is ongoing. 1. Remove the Github installation. 2. Manually configure SSH key access to Github, please see the link below. https://help.cloud66.com/rails/how-to-guides/common-tools/access-your-code#manually-configuring-github-access

2. 6 Οκτωβρίου 2022 στις 2:52 μ.μ. UTCidentified

   The issue has been identified and a fix is being implemented and this should resolve the issue for most users. If the error is still persisting, please use the workaround provided: 1. Remove the Github installation. 2. Manually configure SSH key access to Github, please see the link below. https://help.cloud66.com/rails/how-to-guides/common-tools/access-your-code#manually-configuring-github-access

3. 7 Οκτωβρίου 2022 στις 10:26 μ.μ. UTCresolved

   Issue confirmed to be fixed by GitHub.

1. 14 Σεπτεμβρίου 2022 στις 8:15 μ.μ. UTCidentified

   The team is aware that some Buildgrid pushes are taking longer than normal, and are working on a mitigation strategy

2. 15 Σεπτεμβρίου 2022 στις 5:03 μ.μ. UTCidentified

   Our engineering team is continuing to look into mitigations for this issue.

3. 16 Σεπτεμβρίου 2022 στις 11:47 π.μ. UTCmonitoring

   The team has completed the first part of the mitigation strategy for intermittent slow Buildgrid pushes. We are now monitoring ongoing performance.

4. 6 Οκτωβρίου 2022 στις 2:50 μ.μ. UTCresolved

   This incident has been resolved.

1. 9 Σεπτεμβρίου 2022 στις 7:20 π.μ. UTCidentified

   The issue has been identified and a fix is being implemented.

2. 9 Σεπτεμβρίου 2022 στις 8:51 π.μ. UTCresolved

   This incident has been resolved.

3. 9 Σεπτεμβρίου 2022 στις 9:00 π.μ. UTCpostmortem

   At approximately 07:00 UTC we started to perform a minor system quality-of-life backend update. The update had unintended consequences on an older system component which hadn’t been changed in a while. The component was to do with agent server communications, and it ended up disabled after the update. Although the same component was unaffected during testing in dev/staging and passed UAT, it turned out that there was a necessarily different configuration around rewrites in production that caused the issue. The nature of the update meant that rolling back wasn’t straightforward, so the team resolved to fix the incompatibility in-place, and as such were unable to stop the erroneous “server down” notifications that then went out. The issue was resolved approximate 1 hour later. at 08:00 UTC - after which time servers would have started to appear as “online” again. Subsequent to this outage, the difference in configuration has been added to our operations processes, such that this will not occur again. Our apologies for the inconvenience and concern that this may have created!

1. 6 Σεπτεμβρίου 2022 στις 11:53 π.μ. UTCinvestigating

   GitHub is having an outage at this time. This may adversely affect Deployments.

2. 6 Σεπτεμβρίου 2022 στις 11:57 π.μ. UTCresolved

   This incident has been resolved.

1. 30 Αυγούστου 2022 στις 9:09 π.μ. UTCidentified

   From Azure (https://status.azure.com/en-gb/status) Starting at approximately 06:00 UTC on 30 Aug 2022, a number of customers running Ubuntu 18.04 (bionic) VMs recently upgraded to systemd version 237-3ubuntu10.54 reported experiencing DNS errors when trying to access their resources. Reports of this issue are confined to this single Ubuntu version.

2. 30 Αυγούστου 2022 στις 3:11 μ.μ. UTCidentified

   Azure is recommending that affected customers reboot their servers to obtain an updated DHCP lease to resolve this issue

3. 31 Αυγούστου 2022 στις 9:15 π.μ. UTCmonitoring

   We are continuing to monitor this incident

4. 5 Σεπτεμβρίου 2022 στις 8:21 π.μ. UTCresolved

   This incident has been resolved.