8 Clojars incidents · Cʼhwevrer 2016 — official updates, affected components, duration and resolution details.
Our hosting provider is experiencing network issues.
All Clojars services are currently up, but Linode is still working on the DDoS
The Linode DDoS seems to be subsiding, but we'll continue monitoring the situation.
The DDoS seems to be resolved for now, but we'll keep an eye on it.
Our hosting provider is experiencing network issues (https://status.linode.com).
Things seem to be fixed, but we'll monitor.
Looks like we're all better now
Working on restoring access after nginx upgrade.
Nginx problems appear to have been resolved (IPv6 issues)
This incident has been resolved.
## Summary There was approximately 25 minutes of server downtime from 2016-08-05 09:46 UTC to 2016-08-05 10:09 UTC while updating nginx. The root cause was a change in nginx default behaviour around IPv6 sockets which caused IPv4 connections to be ignored. ## Details The version of nginx installed on the server was quite old. Before upgrading, I looked through the release notes to look for changes. I copied them to a text file for easy reference if something went wrong. Nothing in the changelog looked like it would affect us (though crucially, I didn't fully understand all the mentioned changes...). I updated nginx while it was running (to prevent downtime being too long). Once the update had finished, I restarted nginx. It reloaded and everything seemed to be ok. I then realised that the old process was running, and the new version hadn't started. After killing the old process and starting the new one, clojars.org wasn't accessible. I searched for possible answers, and a change that was mentioned in the changelog came up: `ipv6only` was now set on by default for IPv6 sockets. This means that only IPv6 connections would be accepted and IPv4 would not connect. If you were connecting to Clojars over IPv6, you would have had no connectivity issues. I updated the nginx server config to turn off `ipv6only`, and service was restored. ## Learnings * When updating software, make sure that we understand all changes in the changelog, especially ones which change default behaviour. * If possible it would be good to test upgrades on a test server before applying it to the production server.
Linode appears to be suffering from partial network outages. We're monitoring the situation, but there isn't a lot we can do directly.
Linode issues appear to be resolved. Everything should be operational.
Linode is performing network maintenance that should only last a short while
This incident has been resolved.
We're experiencing network issues with our hosting provider. These appear to be transient.
It seems like everything is back to normal.
We've been alerted about network issues. We're investigating now.
Looks like it was just a temporary blip. Back to your regularly scheduled Clojure programming.
The Clojars team are aware of connectivity issues to Linode Atlanta, and are monitoring the situation. There isn't a lot we can do right now though, we're waiting on Linode to resolve the situation.
Linode has identified the issue, and are working to resolve it. In the meantime, you can always use our mirror to pull your dependencies: https://github.com/clojars/clojars-web/wiki/Mirrors
The site is back up, but Linode is still reporting that they are working on the issue, so we'll continue to monitor until they resolve.
It seems like everything is stable and back to normal, but we'll keep monitoring the situation.
