36 Endless Group incidents · Meurzh 2019 — official updates, affected components, duration and resolution details.
We are rebooting the kvm4 host node in order to access option menus and perform emergency system maintenance. This outage only affects systems hosted on that node, and does not affect the main shared DirectAdmin node.
We have verified there are no hardware problems present on kvm4 and all services have been restored.
Our network appears to be intermittently available. This may result in offline sites. We are working on resolving the problem now.
We have identified the issue as a flapping session with one of our transit providers. We have rerouted all traffic while we investigate.
We have reestablished the flapping connection and are monitoring for additional issues
The issue is continuing to occur. We are restarting our core router in an attempt to resolve.
We are continuing to see some issues with route convergence but all service should be accessible again.
This incident has been resolved.
Due to a crash of our DNS server, we have to reboot our core router. We expect this reboot to occur without any major issues.
The router has been restored however DNS is still nonfunctional. We are investigating this issue rapidly.
This incident has been resolved.
The previous maintenance tasks have not yet been completed.
We are experiencing a problem where one of our host machines is unable to successfully reboot following the upgrade. We are working on this issue as fast as possible.
We have identified the problem as a failing disk in one of our host machines. We are recovering the machine but as this may use a large portion of our in-network bandwidth, please expect degraded performance on your sites at this time.
We will be rebooting the remaining host system in order to finalize the update.
We are continuing to work on restoring the failed host. Most customer systems should be back online at this time.
All host systems have been successfully restored and confirmed to be operational. New drives were installed in the failing machine. Additionally, our new host system has been joined to the cluster. We are now monitoring to ensure that all components are operating normally. All customer systems should be online at this time. If you are experiencing an issue with your system, please contact us using our support channels.
We have been monitoring our host systems and have not observed any further issues. We consider this incident to be resolved. If you are still experiencing problems, please contact our support.
We are having a network outage that may continue for the next few days-- DirectAdmin customers: - Set your sites A records to 64.62.143.206 - We will make this change for you if you are using our DNS
DirectAdmin is functioning normally again. If you are having issues with anything, we can still be reached at [email protected] or via social media.
This incident has been resolved.
Our main DirectAdmin and VPS host server is down due to a community power outage on both the A+B (EDIT: see below!) power feeds at our datacenter. We are awaiting power recovery. Our email is not affected as this runs through Office 365. EDIT: The power outage may only be on one of the feeds and the access switch with our transit is on the dead feed.
This incident has been resolved.
We are investigating increased service error rates and outages.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
A power outage has affected our datacenter and caused some networking issues. Although our equipment did not specifically lose power, our transit providers did, so there was a brief period of time that our services were unavailable. Additionally, a reboot of our router was required in order to fully restore service, which has been completed.
We are currently experiencing an incoming network traffic issue caused by one of our partners, @RATELIMITEDME . A rogue VM owned by them is currently consuming all incoming traffic to our host. If you receive the message {"success":true,"message":"The File Handler is working normally.","node_info":{"version":2}} when attempting to visit your site, you are affected by this problem. We are currently working with them to resolve this issue and expect to have service restored soon.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
Quick Downtime to fix something
We are continuing to work on a fix for this issue.
This incident has been resolved.
We are experiencing an issue with our JIRA helpdesk that appears to be beyond our ability to fix. We have opened a ticket with Atlassian in order to get help on this issue. In the meantime, you can use Discord to contact us, as emailing our support email or opening a ticket in JIRA is currently not an option.
Thanks to Atlassian support for helping us with this issue! We're up and running again!
Our MySQL instance seems to have crashed and is not restarting as intended. We are investigating this issue and hope to be able to have it up and running again soon.
The issue has been identified to be a configuration that was reset without our knowledge during a recent system update. The configuration has been fixed and MySQL is now starting again. Please expect degraded performance as many sites are attempting to reconnect to the database now that is is online again.
This incident has been resolved.
At this time we are expierencing issues with the MySQL database on our production directadmin server. Please stand by while we investigate the issue.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
At this time we have been made aware of a problem affecting networking. Further investigation reveals that our BGP session with Vultr is flapping. We are currently investigating the cause of this but it appears unrelated to our end of the system.
The session with Vultr has returned to an established state and we are watching to ensure that the session does not flap again.
This incident has been resolved.
Sites that rely on database connections may not be functioning properly. We are investigating the issue.
We are continuing to investigate this issue.
This issue has been identified. Please expect up to two hours of downtime on database-related functions.
A fix has been implemented and we are now monitoring the results. Please notify us via support ticket or discord if your application is unable to connect to or use the database.
After testing that the database is operational, we consider this incident resolved. Please let us know if you have any issues after this point.
At this time we are entirely offline due to inclement weather causing a power outage at our main datacenter location. We do not have an ETA on service restoration, multiple trees have fallen across the power lines and must be handled.
We are continuing to work on a fix for this issue.
At this time services have been restored. Thanks for sticking with us.
Due to DirectAdmin licensing changes, our DirectAdmin license is being treated as expired on the server. This prevents users from logging into the DA website, but websites will remain online. We retain console access to the server. If there are mission-critical changes that must be made to your website during this time, please contact us and we will find a way to accommodate. We currently have no ETA as to when DA will be working again. We are at the whim of DA support to fix the licensing problem/activate our new license. Additionally, signups are currently not working as DA API has shutdown. If you create a signup, it will take a longer amount of time to get processed as we will have to wait until DA is working again.
DirectAdmin has gone ahead and fixed the issue with our license and we are back online as normal.
Our upstream provider is switching our BGP session to a new ASN. While they do this, we will not have uptime on our IP addresses, and therefore there will be a service outage.
Our upstreams have enabled our new session. We are now waiting for our prefix to become visible.
Please continue to monitor the ripe stat page for our IP block to watch the progress of our new announcements. https://stat.ripe.net/185.86.231.0%20-%20185.86.231.255?sourceapp=ripedb#tabId=routing
We are seeing at least 80% of the global network is able to see our address block. We will leave this incident open as we continue to monitor the situation but at this time the Endless services are back online.
As we are now seeing 100% visibility on RIPEStat, this incident is now resolved.
We are investigating an issue where there may be brief periods of packet loss within our network resulting in degraded performance of customer sites.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We are currently investigating issues that affect our Billing Portal (portal.theendlessweb.com) and our Jira/Confluence instances. We'll let you know once we have an update.
This incident has been resolved.