Elevio Status

Maintenance has been completed

Timeline \(in UTC\):19:10pm: The hosted KB becomes unavailable and service restores without intervention in less than 1 minute. 19:16pm: The hosted KB becomes unavailable and service restores without intervention within 2 minutes. 19:50pm - 20:30pm: Multiple events where the hosted KB becomes unavailable for 1-2 minutes and service restores without intervention. On-call engineer escalates the incident with the backend team. 20:30pm - 22:00pm: The service becomes unavailable and no longer restores. Restarting the servers restores service for short periods of time. 22:15pm: The issue is identified and a temporary fix is deployed. Service resumes as normal 23:30pm: A permanent fix is deployed. During this time, the temporary fix had to be momentarily reverted in order to deploy the new version which caused the service to become unavailable for < 5 mins.

This incident has been resolved.

On Monday 12 July 2021 at 10AM AEST, we made a change to our main database to improve its stability and performance. We closely monitored the metrics to make sure all systems that depend on it were performing normally, and they were. However, on Tuesday 13 July 2021 at around 12:05AM AEST, the database started degrading, leading to a partial outage. Our failover system kicked in within seconds, and it took 5-10 minutes for most of our services to fully recover and metrics to stabilise. Unfortunately the database gradually degraded again, leading to another partial outage at 4:20AM AEST. It successfully failed over to a backup instance like in the first outage. However, this time around it took our systems 15-20 minutes to fully recover from it. At around 4:50AM AEST everything went back to normal. After investigating the two incidents, we identified the root cause as the change we made on Monday morning. We've decided to roll the database back to its previous configuration and we're cautiously optimistic that the same issue will not happen again.

This incident has been resolved.

All systems are now operating normally.

Related to \(but independant from\) the KB downtime that was experienced on May 14, one of the KB servers experienced an issue whereby excessive caching filled the servers disk capacity causing flow on effects. The issue was confined to only one server, which meant the KB would appear down in some cases, but OK in others after a reloading of the page. To resolve this issue, caching in ares has been removed, as improvements made over time at the API level have rendered the speed improvements received with caching unnecessary and will remove the possibility of this issue from reoccurring in future.

Earlier this morning there was an issue with the knowledge base servers having intermittent troubles displaying. This was due to the servers filling up with log files over time, as well as cached views, and reached a tipping point. We’ve resolved the issues and cleaned up the servers, as well as moved the logging to an external service to prevent this situation from occurring again in the future.

This incident has been resolved.

After monitoring the KB servers for some time, we're confident the issue can be marked as resolved.

The scheduled maintenance has been completed.

The scheduled maintenance has been completed.

The scheduled maintenance has been completed.

The scheduled maintenance has been completed.

The scheduled maintenance has been completed.

The scheduled maintenance has been completed.

The scheduled maintenance has been completed.

The scheduled maintenance has been completed.

The scheduled maintenance has been completed.

The scheduled maintenance has been completed.