Mautic Cloud Status

This incident has been resolved.

This incident has been resolved.

There was a situation which resulted in intermittent 500 errors showing on the Community Handbook which has now been resolved.

The Mautic Community Portal was subjected to a Distributed Denial of Service attack starting on 9th November 2024, which despite having proactive measures in place to deflect such attacks, resulted in an extreme increase in network usage leading to our cloud-based infrastructure reaching its maximum limits for scaling, taking the portal offline. The situation was addressed and the attack was brought under control, with additional measures implemented to ensure any future attacks are proactively halted.

This incident has been resolved.

On Monday morning, a two hour outage was planned to transition our Decidim instance from using a private image over to using the Mautic GitHub repository and DockerHub image, with the end goal being able to have automated pipelines for deployment when a community PR is merged. During the process, one of the encryption keys which are essential for Decidim’s functioning - which was supposed to be backed up - unfortunately was overwritten and could not be recovered. We reached out to the hosting provider, Infomaniak, to request a rollback to backup, but it was after hours which meant we had to wait for the next morning. The next day, we called Infomaniak again to request the rollback and they assured us it would happen in the early evening. Upon calling to get a status update, we were informed that they did not have any backups from which to restore from. We therefore had to set about the process of removing all encrypted data and re-configuring it with new keys, which happened on Wednesday morning. ## Actions and learnings 1. In the process the current setup was removed and started again pulling from the new infrastructure, which resulted in the keys being lost. In the future we will take the longer route of making the production instance read-only and setting up a separate working area to do the update before switching to it. 2. We were planning to set up off-site backups rather than just relying on the hosting provider’s backup which we now know are non-existent. This has now been accelerated and we’ve got backups both of the encryption keys and the entire instance in several locations for future resilience. Pipelines will be set up to automatically push backups at regular intervals. 3. One of the people involved assumed the other had set up off-site backups already and didn’t think to question - in the future we’ll be more explicit about our backup integrity before making such major changes.

We are pleased to inform you that the incident we've been dealing with has been successfully resolved. We kindly thank you for your patience and cooperation during this time. Our team will continue monitoring the system to ensure ongoing stability.

This incident has now been resolved.

An unplanned outage on our documentation server.

This incident has been resolved.

The scheduled maintenance has been completed.

The scheduled maintenance has been completed.

The scheduled maintenance has been completed.

The scheduled maintenance has been completed.

The scheduled maintenance has been completed.

The scheduled maintenance has been completed.

The scheduled maintenance has been completed.

The scheduled maintenance has been completed.

The scheduled maintenance has been completed.

The scheduled maintenance has been completed.