53 Hypernode incidents · september 2022 — official updates, affected components, duration and resolution details.
Dette spørgsmål er blevet identificeret, og et fix er ved at blive gennemført.
Et fix er blevet gennemført og overvåges
Denne hændelse er blevet løst.
Automatisk oversat fra den officielle hændelsesopdatering.
Vi oplever en DNS afbrydelse forårsaget af et DDoS angreb rettet mod vores DNS hosting partner infrastruktur. De diagnosticerer og mindsker angrebet. Vi er i tæt kontakt med deres team og overvåger situationen. Vi opdaterer, så snart tjenesten er genoprettet.
Holdet har afværget DDOS-angrebet og vil overvåge situationen nøje.
Denne hændelse er blevet løst.
Automatisk oversat fra den officielle hændelsesopdatering.
Vi oplever i øjeblikket en vis nedbrydning, når der foretages ændringer til Hypernoder (version ændringer, whitelists, SSH nøgler). Kontakt venligst support i hastetilfælde. Dette påvirker ikke den generelle tilgængelighed for din server eller applikation.
En rettelse er blevet rullet ud og mistede ændringer vil nu blive håndteret, skal du holde øje med 'hypernode-log' for eventuelle mistede handlinger.
Alle systemer er i drift. En synkronisering primitiv i vores automatiseringssystem blev fanget i en død tilstand. Vi gennemfører overvågningsregler for at identificere lignende problemer i fremtiden, mens vi holder overvågning af systemet for døde låse.
Denne hændelse er blevet løst.
Automatisk oversat fra den officielle hændelsesopdatering.
We are currently experiencing intermittent downtime affecting services under the *.hypernode.io domain. Our DNS provider is undergoing a large-scale DDoS attack, which is causing instability in domain name resolution. As a result, some requests may fail to resolve or experience delays. We want to emphasize that our platform and infrastructure remain operational; however, access is impacted due to external DNS issues. Our team is actively monitoring the situation and working with the provider to mitigate the impact. We are also exploring mitigation strategies to reduce dependency on the affected service. We will continue to provide updates as more information becomes available.
We experienced a brief period of instability affecting DNS resolution for the *.hypernode.io domain, which may have caused temporary downtime or failed requests. Service has since recovered, and all systems are currently operating normally. Initial indications suggested a possible DDoS attack at the DNS provider level; however, this has not been confirmed. We are continuing to monitor the situation closely and are in contact with the provider to better understand the root cause.
The earlier DNS resolution instability affecting the *.hypernode.io domain has now fully recovered. Stability has been restored and we are no longer observing any degraded performance or resolution failures over an extended period of time.
Part of our datacenter is currently under a DDoS attack, resulting in degraded performance across multiple nodes. We are actively investigating short-term mitigation options.
We have implemented anti-DDoS measures and rebooted the surrounding hypervisors, which has restored normal operations. We will keep monitoring the situation.
Since implementing these measures and closely monitoring the situation, we have not observed any further downtime or service disruptions.
We are currently experiencing issues with creating branchers, performing upgrades, and provisioning ordered Hypernodes. Our team is actively working to investigate and resolve the problem.
We have identified the issue and are currently working on a fix.
A fix has been implemented and we are monitoring the results.
The issue has been resolved. A fix was successfully implemented, and we are continuing to monitor the results to ensure stability.
We are currently experiencing degraded performance in the control panel. This issue has been caused by a portion of our non-critical infrastructure reaching resource limits. While the Hypernodes remain operational, users may notice slower or no responses when using the control panel. Our engineering team is actively working to resolve the capacity constraints and restore full performance. We expect operations to return to normal shortly.
A fix has been implemented and the problems in the control panel have been resolved, we will further monitor the situation.
We are continuing to monitor for any further issues.
Issues has been fixed we don't expect this to occur again
We are experience with our Object Storage service provider. Investigation with provider is ongoing.
The issue has been identified and a fix is being implemented.
This incident has been resolved by our Object Storage service provider.
We experienced a brief network hiccup in one of our datacenters, which may have caused intermittent connectivity issues for approximately 15 minutes. The issue has been resolved, and network stability has been fully restored. Our team monitored the situation closely to ensure no further impact. No further action is required at this time. We will continue to monitor the platform to ensure ongoing stability.
We are currently aware of an issue affecting the control panel on my.hypernode.com. Some users may experience problems loading the “My Hypernodes” page. Our team is investigating the issue and working to determine the cause. At this time, no other services appear to be impacted.
The issue affecting the "My Hypernodes” page has been isolated and resolved. All services are now operating normally.
We are currently experiencing an issue with our control panel on my.hypernode.com. The “My Hypernodes” page is not functioning correctly, and the list of Hypernodes is not loading. Our team is aware of the issue and is actively investigating the cause. At this time, no other services are impacted. We will provide an update as soon as more information becomes available.
The issue affecting the control panel on my.hypernode.com, where the “My Hypernodes” page was not loading correctly, has been resolved. All functionality is operating normally again, and no other services were impacted. We will continue to monitor the platform to ensure stability.
We are currently experiencing issues with several services due to a broader Cloudflare disruption. This outage is affecting our Support ticketing system, and our phone support is currently unavailable. Additionally, some customers may experience 500 errors on their websites as a result of this Cloudflare issue. Cloudflare is currently investigating problems impacting the Cloudflare Dashboard and related APIs. Customers using the Dashboard or API functionality may encounter failed requests or error messages.
Cloudflare has resolved the underlying issue, and all our services are fully operational again. We have confirmed that both our Support systems and phone availability are functioning normally.
Cloudflare is currently experiencing issues routing requests to some Hypernodes. Please note that this is not a Hypernode outage, and the issue must be resolved on Cloudflare’s side. We recommend keeping an eye on Cloudflare’s status page for updates. If this significantly impacts you, a temporary workaround is to disable the Cloudflare Proxy in your Cloudflare DNS settings until the issue is resolved.
The incident has been marked as resolved by Cloudflare and we're expecting no more inconvenience.
We're currently seeing extremely degraded performance across our platform, this seems to be related to our DNS resolving to hypernode.io , we are actively investigating and will update when possible.
We have identified our DNS server is being flooded with packets. We are activating our counter-measures and continue monitoring the situation.
Sadly as we're trying to resolve this as quick we can, an alternative temporary solution is to use the direct IP if you're using an external DNS service, you can get this with using dig with our alternative DNS server, make sure to point your A-record to the resulting IP-address: ``` dig @80.84.227.137 abc.hypernode.io A ``` We hope this won't be needed for long and we can return operations as soon as possible.
As the attack seems to calm down, operations are returning on our nodes and internal infrastructure. We continue working on increasing our stability to mitigate future incidents.
Our DNS infrastructure has been running stable for the past 12 hours. We'll continue to monitor and improve the infrastructure.
We're currently seeing degraded performance for our Cloudflare users running on at least our German dedicated datacenter, we are monitoring the situation and are contact with the relevant third parties
We're seeing no current problems on the datacenter/Hypernode side, and it appears to be an instability with Cloudflare. Please be advised to consider disabling the proxy temporarily and monitoring the situation closely.
The Hypernode cloud platform for the R405 room is currently experiencing degraded performance and may cause incididental downtime, we are aware of this and are monitoring the situation. As soon as we know more we will update our statuspage.
We are continuing to investigate this issue.
Our provider has identified the issue and is working on a solution
Operational activities are being restored across the nodes.
All affected nodes have fully resumed operations. Earlier, our cloud provider in Room R405 experienced a temporary issue with their network-storage layer, which caused downtime on these nodes. The provider quickly diagnosed and resolved the problem. We confirm that no data loss occurred.
We are aware of issues with the Cloud Hypernodes and are in close contact with the datacenter to determine the cause.
We identified a brief downtime in one of the OpenStack availability zones, which triggered our monitoring systems to automatically reboot some nodes. Platform stability is now improving and returning to normal.
A brief network interruption in one of our OpenStack availability zones caused some instances to automatically reboot. The issue has been resolved and platform stability has been restored.
We are currently experiencing DNS disruptions caused by a DDoS attack. This has resulted in intermittent service issues for some users.
We've taken measures on our side to help mitigate the impact of the DDoS attack on our DNS infrastructure. At this time, full performance has been restored, and we are closely monitoring the situation.
The issue has now been fully resolved, and all services are operating normally.
The issue has been identified and a fix is being implemented.
This incident has been resolved.
We are currently experiencing issues with creating branchers, xgrading and providing ordered Hypernodes. Our team is actively investigating and resolving this issue.
The issue has been identified and we are working on a fix.
The issue has been resolved and had to do with resource-limitations with our service partners, we will be monitoring the situation over the coming week but do not expect any downtime at this current moment.
