59 Upcloud incidents · август 2024 г. — official updates, affected components, duration and resolution details.
Мы изучаем проблему подключения к панели управления и API.
Было принято решение, и мы следим за ситуацией.
Этот инцидент был урегулирован.
Автоматический перевод официального обновления инцидента.
Мы выявили проблему деградации услуг, затрагивающую хранилище управляемых объектов в нашем центре обработки данных в Сингапуре (SG-SIN1). Проблема вызывает периодические всплески задержки и ошибки HTTP 503, которые длятся от 10 до 15 минут. Мы определили первопричину и активно работаем над решением вопроса и полным восстановлением сервиса. Мы приносим извинения за неудобства и предоставим дополнительные обновления как можно скорее. Если у вас есть какие-либо вопросы, пожалуйста, свяжитесь с нашей службой поддержки через живой чат или по электронной почте [email protected]
Мы определили первопричину деградации сервиса, влияющего на хранение управляемых объектов в нашем центре обработки данных в Сингапуре (SG-SIN1) и внедрили исправление. В настоящее время мы проводим мониторинг системы для обеспечения долгосрочной стабильности. Спасибо за ваше терпение, и мы приносим извинения за любые неудобства. Если у вас возникли какие-либо проблемы, пожалуйста, свяжитесь с нашей службой поддержки через живой чат или по адресу [email protected].
Этот инцидент был урегулирован.
Автоматический перевод официального обновления инцидента.
Мы изучаем проблему серверного хранилища файлов, которая затрагивает некоторые службы хранения файлов в нашем центре обработки данных FI-HEL2. Мы заметили некоторые нежелательные отказы для некоторых служб хранения файлов NFS, и в качестве меры безопасности мы выполняем еще несколько отказов для решения и изоляции оставшихся проблем и возвращения бэкэнда в нормальное состояние. Во время этих дополнительных отказов для некоторых из FI-HEL2 могут произойти службы NFS хранения файлов. Мы сожалеем о неудобствах.
Реализовано исправление, и мы отслеживаем результаты.
Этот инцидент был урегулирован.
Автоматический перевод официального обновления инцидента.
Мы изучаем проблему серверного хранилища, которая затрагивает некоторые ресурсы, включая облачные серверы в нашем центре обработки данных DE-FRA1. Мы активно работаем над тем, чтобы как можно быстрее решить этот вопрос. Мы сожалеем о любых неудобствах, которые это может вызвать, и благодарим вас за ваше терпение, поскольку мы работаем над этим. Если у вас есть какие-либо вопросы или проблемы, пожалуйста, не стесняйтесь обращаться в нашу службу поддержки 24/7 в нашей панели управления или по электронной почте [email protected].
Реализовано исправление, и мы отслеживаем результаты.
Этот инцидент был урегулирован.
Автоматический перевод официального обновления инцидента.
The issue has been identified, and we are working on fixing it. It affects all new deployments in all zones.
We have identified the cause of the issue affecting Cloud Server Deployments and are actively working on a fix. The issue may impact new Cloud Server Deployments. Please refrain from attempting management operations, including Start, Restart, Shut down, and Force Shutdown, to avoid unexpected complications while we work to bring all affected services back online as swiftly as possible. We apologize for the inconvenience and will provide further updates as more information becomes available. If you have any questions or concerns regarding your services, please reach out to our support team via live chat at or via email: [email protected]
A fix has been implemented, and new Cloud Server deployments are now proceeding without errors. We are currently cleaning up Cloud Servers that were left in an error state during the incident.
We thank you for your continued patience throughout this process and apologize for any impact. The incident is now resolved, and all affected services should be operational. For any further issues, please contact our Support team via live chat or email at [email protected]. We are compiling a detailed post-mortem to outline the sequence of events, the steps taken to rectify the situation, and the measures we are implementing to prevent such occurrences in the future. We understand the gravity of this disruption and extend our sincerest apologies to all affected customers.
Our team has confirmed the full resolution of the issue. All affected services should now be functioning normally. If you continue to experience any further issues, please contact our Support team via Live Chat or email [email protected] Thank you for your patience, and we apologize for any inconvenience caused.
В настоящее время мы испытываем технические проблемы с нашей системой Live Chat. Вы все еще можете связаться с нашей службой поддержки по электронной почте [email protected]. Мы приносим извинения за неудобства и предоставим обновление, как только будет восстановлен живой чат.
Этот инцидент был урегулирован.
Автоматический перевод официального обновления инцидента.
Мы выявили проблему с нашим хранилищем объектов Europe 2 в нашем центре обработки данных DE-FRA1. Проблема может привести к тому, что затронутые службы не смогут читать или писать в хранилище объекта. Мы работаем над тем, чтобы вернуть все затронутые сервисы в онлайн и решить ситуацию как можно скорее.
Реализовано исправление, и мы следим за этим.
Этот инцидент был урегулирован.
Автоматический перевод официального обновления инцидента.
We have resolved the DNS resolution issue that was causing increased response times. Impacting our control panel https://hub.upcloud.com/ and certain cloud services in the FI-HEL1 and FI-HEL2 data centres. All systems are now stable, and there should be no further visible impact. We are continuing to monitor performance closely while we investigate the root cause. We apologise for the inconvenience. If you have any questions, please don't hesitate to contact us via email [email protected] or via live chat https://hub.upcloud.com/.
We have resolved the DNS resolution issue that was causing increased response times. Impacting our control panel https://hub.upcloud.com/ and certain cloud services in the FI-HEL1 and FI-HEL2 data centres. After monitoring our systems for a some time, we can confirm now the situation is stable. We apologise for the inconvenience. If you have any questions, please don't hesitate to contact us via email [email protected] or via live chat https://hub.upcloud.com/.
Мы расследуем проблему, затрагивающую платежи по картам American Express, из-за которой мы наблюдаем увеличение числа отказов. Все остальные способы оплаты остаются полностью работоспособными. Мы рекомендуем тем, кто использует American Express для автоматизированных платежей, проверить, что они были успешными. В качестве обходного пути клиенты могут использовать альтернативный способ оплаты, пока мы работаем с нашим поставщиком платежей для решения проблемы. Доступные способы оплаты перечислены здесь: https://upcloud.com/docs/getting-started/billing/payment-metods/ Мы предоставим дополнительные обновления по мере их появления. Если у вас есть какие-либо вопросы или вам нужна дополнительная помощь, пожалуйста, свяжитесь с нами по электронной почте: [email protected] или Live Chat в нашей панели управления https://hub.upcloud.com
Проблема была выявлена, и мы работаем над ее решением. Особенно это касается платежей в евро, с картами American Express. В качестве обходного пути клиенты могут использовать альтернативный способ оплаты, пока мы работаем с нашим поставщиком платежей для решения проблемы. Доступные способы оплаты перечислены здесь: https://upcloud.com/docs/getting-started/billing/payment-metods/ Мы предоставим дополнительные обновления по мере их появления. Если у вас есть какие-либо вопросы или вам нужна дополнительная помощь, пожалуйста, свяжитесь с нами по электронной почте: [email protected] или Live Chat в нашей панели управления https://hub.upcloud.com
Реализовано исправление, и платежи в евро (EUR) с использованием карт American Express снова обрабатываются нормально. Мы продолжаем внимательно следить за ситуацией, чтобы обеспечить постоянную стабильность. Если у вас есть какие-либо вопросы или вам нужна дополнительная помощь, пожалуйста, свяжитесь с нами по электронной почте [email protected] или через Live Chat в нашей панели управления: https://hub.upcloud.com
Теперь это решено. Если у вас есть какие-либо вопросы или вам нужна дополнительная помощь, пожалуйста, свяжитесь с нами по электронной почте [email protected] или через Live Chat в нашей панели управления: https://hub.upcloud.com
Автоматический перевод официального обновления инцидента.
We have identified an issue with network connectivity in our SG-SIN1 data centre. The affected cloud servers may experience slow connections or become unreachable. We are working to restore connectivity and resolve the situation as soon as possible.
We are currently investigating this issue.
We have identified an issue with network connectivity in our SG-SIN1 data centre and AU-SYD1 🟢 Managed Kubernetes Cluster Workloads: Not Affected (100% Uptime) 🟢 Kubernetes API / kubectl: Not Affected 🔴 UpCloud Managed Kubernetes API & Control Panel Section: Temporarily Unavailable (Cannot create, view, delete, scale, or modify clusters) 🟢 Managed Object Storage Workloads: Not Affected (100% Uptime) 🔴 UpCloud Managed Object Storage API & Control Panel Section: Temporarily Unavailable (Cannot create, view, delete, scale, or modify clusters) We are actively working to resolve the incident. We sincerely apologise for any inconvenience this may cause to your operations. We will provide regular updates here as more information becomes available. If you have any questions or need further assistance, please reach out to us via email: [email protected] or Live Chat in our control panel https://hub.upcloud.com
We have restored all affected Managed Kubernetes and Managed Object Storage to normal operating status and will continue to monitor the situation. 🟢 Managed Kubernetes Cluster Workloads: Not Affected (100% Uptime) 🟢 Kubernetes API / kubectl: Not Affected 🟢 UpCloud Managed Kubernetes API & Control Panel Section: Temporarily Unavailable (Cannot create, view, delete, scale, or modify clusters) 🟢 Managed Object Storage Workloads: Not Affected (100% Uptime) 🟢UpCloud Managed Object Storage API & Control Panel Section: Temporarily Unavailable (Cannot create, view, delete, scale, or modify clusters) If your services is still not fully functional, please contact our support at [email protected]. We apologise for the incident.
We have resolved the root cause of the networking issue affecting our Managed Object Storage and our Managed Kubernetes service in our SG-SIN1 data center. All affected services should now be functioning normally. If you continue to experience any further issues, please contact our Support team via Live Chat or email [email protected]. Thank you for your patience and we apologize for any inconvenience caused.
We are currently investigating an issue affecting new cloud service deployments. We apologise for the inconvenience and will provide an update in 30 minutes with more information.
We have implemented the fix and isolated the issue. And we will keep monitoring the situation. We sincerely apologise for any inconvenience. And if you have any issues or questions, please don't hesitate to contact us via email: [email protected] or via live chat from our control panel.
This incident has been resolved.
We are currently investigating an issue affecting the Managed Kubernetes Service in our SG-SIN1 region. Users may find that their clusters are stuck in a Pending state. Our team is currently investigating the issue to identify the root cause. We will provide additional updates as more information becomes available. We apologise for any inconvenience this may cause.
We have identified the root cause of the issues affecting the Managed Kubernetes service in the SG-SIN1 region. Our team is currently working on a resolution to address the issue. We will provide further updates once the fix is applied.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We have identified an issue causing slowness with new logins to our control panel at https://hub.upcloud.com/ And we are currently working on fully stabilising the situation. This may also delay calls to our API, and dependent systems such as Terraform. Please note that this issue does not affect your active cloud services. If you experience any issues, please do not hesitate to contact our support team via email [email protected]. We apologise for the inconvenience.
We are continuing to work on a fix for this issue.
A fix has been implemented and we are monitoring the results.
This incident has been fully resolved. Please contact our support team via live chat or [email protected] if you have any further questions regarding the incident.
We have identified an issue causing latency and timeouts when accessing or managing Managed Object Storage in the Europe-1 region. Users may experience temporary timeouts or failure which management operations or to load bucket details. All other Managed Object Storage regions (Europe-2, Europe-3, US-1, APAC-1) are unaffected. If you have any questions or concerns regarding your services, please reach out to our support team via live chat at https://hub.upcloud.com/ or via email: [email protected]
We are continuing to work on a fix for this issue.
A fix has been implemented and we are monitoring the results.
Our team has confirmed the full resolution of the issue affecting our EUROPE-1 Managed Object Storage in our FI-HEL2 data center. All affected services should now be functioning normally. If you continue to experience any further issues, please contact our Support team via Live Chat or email [email protected] Thank you for your patience and we apologize for any inconvenience caused.
We have identified an issue with one of the compute nodes in our SG-SIN1 data centre. The problem may cause affected cloud servers to become slow or unresponsive. We are working to bring all affected servers back online and resolve the situation as soon as possible.
A fix has been implemented and we are monitoring the results.
Our team continues to be fully engaged and focused on finding a mitigation. To avoid unexpected complications during the recovery process, please refrain from attempting server management operations, including Start, Restart, Shut down, and Force Shutdown. We apologize for the inconvenience and appreciate your patience as we continue working to address the situation. If you have any questions or concerns regarding your services, please reach out to our Support team via Live Chat or via email [email protected]
This incident has been resolved.
We are currently investigating reports of issues affecting the deployment of new Cloud Servers in our US-SJO1 data center. We sincerely apologize for the inconvenience and will share more information as soon as it becomes available.
We have identified the cause of the issue and applied a fix. While we are seeing signs of new Cloud Server deployment operations recovering, we are continuing to work on a permanent resolution to prevent a recurrence. If you have any questions or concerns regarding your services, please reach out to our support team via Live Chat or email [email protected]
We have been monitoring the fix implemented earlier and all internal systems are performing as expected. All affected services have been fully restored. We will continue to monitor the situation until we can confirm the full resolution of the issue. If you continue to experience any persistent issues with your cloud services in the US-SJO1 region, please reach out to our support team immediately via Live Chat or email [email protected]
This incident has been resolved.
We are investigating a storage backend issue that is affecting some cloud servers in our US-NYC1 data centre.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We have identified a compute node issue affecting some Cloud Servers, Managed Load Balancers and Managed Kubernetes services in our FI-HEL1 data center. You may experience unresponsive servers, delays or failures in provisioning new resources, or errors when managing existing clusters and load balancers in this zone. We are working to bring all affected services back online and resolve the situation as soon as possible. If you have any questions or concerns regarding your services, please reach out to our support team via live chat or at [email protected].
Our team is working to ensure all services are fully restored and operating normally. We have deployed a fix for the issue affecting the managed resources. It may take a short time for these changes to take effect across all resources. If you have any questions or concerns regarding your services, please reach out to our support team via live chat at hub.upcloud.com or via email: [email protected]
We are continuing to work on this issue to ensure all services are fully restored to normal operation. The earlier fix is still finishing propagating to all managed resources.
We are continuing to work on this issue to ensure all services are fully restored to normal operation.
We are continuing to work on this issue to ensure all services are fully restored to normal operation. Normal operational status has been restored across the majority of our cloud infrastructure and managed services.
We have restored all affected cloud servers to normal operating status and continue to monitor the situation. If your server is still not fully functional, please contact our support at [email protected] or reach out to us via live chat at hub.upcloud.com We apologise for the incident.
We have been monitoring the situation closely over the 12 hours and can confirm that all affected services have now been fully restored. The Issue impacted some of our Managed Load Balancer instances within the FI-HEL1 Data Centre, which subsequently impacted some Managed Kubernetes clusters in that region. We sincerely apologise for this incident and we continue our internal investigation. If you have any remaining questions, please contact us via email at [email protected] or through our live chat https://hub.upcloud.com/
We are currently investigating an issue affecting Managed Object Storage in our APAC-1 location, specifically returning 503 errors during operations sporadically. We apologise for the inconvenience and will provide further updates as more information becomes available.
We thank you for your continued patience throughout this process and apologize for any impact. The incident is now resolved, and all affected services should be operational. For any further issues, please contact our Support team via live chat or email at [email protected] We are compiling a detailed post-mortem to outline the sequence of events, the steps taken to rectify the situation, and the measures we are implementing to prevent such occurrences in the future. We understand the gravity of this disruption and extend our sincerest apologies to all affected customers.
This incident has been resolved.
We are currently witnessing degraded performance with our Object Storage instances in our Europe-1 zone. We are actively working to resolve this issue as swiftly as possible. We regret any inconvenience this might be causing and thank you for your patience as we work to address this. If you have any questions or concerns, please feel free to contact our 24/7 live Support team in our control panel or by emailing [email protected].
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
