10 Mautic Cloud incidents · January 2022 — official updates, affected components, duration and resolution details.
We are currently carrying out some essential maintenance to the website. We'll be back online soon!
We're just finishing the maintenance, and will monitor for the next 30 minutes.
We are continuing to monitor for any further issues.
This incident has been resolved.
We are currently investigating an outage on the mautic.org website.
Access to the website is restored and we'll continue to monitor the situation.
This incident has been resolved.
There was a situation which resulted in intermittent 500 errors showing on the Community Handbook which has now been resolved.
The Mautic Community Portal was subjected to a Distributed Denial of Service attack starting on 9th November 2024, which despite having proactive measures in place to deflect such attacks, resulted in an extreme increase in network usage leading to our cloud-based infrastructure reaching its maximum limits for scaling, taking the portal offline. The situation was addressed and the attack was brought under control, with additional measures implemented to ensure any future attacks are proactively halted.
Discourse has a service outage which is under investigation. Please follow the incident at https://status.discourse.org/. Incident Status Partial Service Disruption Components Discourse Basic, Standard, and Business Hosting, Discourse Enterprise Hosting Locations NA West (sea) 1, NA West (sea) 2 June 14, 2024 10:02AM UTC[Investigating] We are investigating reports of connectivity issues to some hosted sites June 14, 2024 10:32AM UTC[Identified] We have identified an issue with ipv4 connectivity for some users, and continue to investigate
This incident has been resolved.
We are working on finalising the deployment of the Community Portal which encountered some problems with broken images and links not working.
We are continuing to work on a fix for this issue.
We have identified the cause of the issues (broken links and not being able to log in) which is relating to a mismatch in encryption keys. We will be rolling back in the morning when the infrastructure providers' support is available, which will resolve the issue and allow us to get back up and running. Apologies for the inconvenience.
We have implemented a roll-back and are in the process of reconfiguring some aspects which have not been cleanly recovered. It is anticipated this will be resolved on Wednesday morning. At present login and some pages will be unavailable until this is resolved.
A fix has been implemented for the issues that were experienced, we will continue to monitor to ensure that all systems are working as expected.
This incident has now been resolved.
On Monday morning, a two hour outage was planned to transition our Decidim instance from using a private image over to using the Mautic GitHub repository and DockerHub image, with the end goal being able to have automated pipelines for deployment when a community PR is merged. During the process, one of the encryption keys which are essential for Decidim’s functioning - which was supposed to be backed up - unfortunately was overwritten and could not be recovered. We reached out to the hosting provider, Infomaniak, to request a rollback to backup, but it was after hours which meant we had to wait for the next morning. The next day, we called Infomaniak again to request the rollback and they assured us it would happen in the early evening. Upon calling to get a status update, we were informed that they did not have any backups from which to restore from. We therefore had to set about the process of removing all encrypted data and re-configuring it with new keys, which happened on Wednesday morning. ## Actions and learnings 1. In the process the current setup was removed and started again pulling from the new infrastructure, which resulted in the keys being lost. In the future we will take the longer route of making the production instance read-only and setting up a separate working area to do the update before switching to it. 2. We were planning to set up off-site backups rather than just relying on the hosting provider’s backup which we now know are non-existent. This has now been accelerated and we’ve got backups both of the encryption keys and the entire instance in several locations for future resilience. Pipelines will be set up to automatically push backups at regular intervals. 3. One of the people involved assumed the other had set up off-site backups already and didn’t think to question - in the future we’ll be more explicit about our backup integrity before making such major changes.
We are currently investigating a degradation of service on mautic.org.
We're pleased to inform you that the source of the recent issue has been identified. Our team has already initiated a remediation plan to rectify the problem. Issue: Degradation of service on mautic.org resulting in timeout errors. Cause: Discourse comments module causing excess load. Current Status: Remediation is in progress. We appreciate your understanding and patience as we work towards restoring full functionality. We will continue to provide regular updates until the situation is completely resolved.
Good news! The issue we've been experiencing has been successfully resolved. We would like to express our thanks to everyone for their patience and understanding during this process. Rest assured, our team will remain vigilant and will continue to ensure the stability of our systems. If you encounter any further problems, please do not hesitate to contact us.
We are pleased to inform you that the incident we've been dealing with has been successfully resolved. We kindly thank you for your patience and cooperation during this time. Our team will continue monitoring the system to ensure ongoing stability.
We are aware of intermittent errors on the Mautic Community Forums and are investigating.
Discourse have declared an incident and are investigating: https://status.discourse.org/pages/incident/5e2141ce30dc5c04b3ac32fc/62a9bab8cdf18612b051e284
Discourse have identified the problem and implemented a fix, we will continue to monitor the situation.
This incident has now been resolved.
An unplanned outage on our documentation server.
The issue is being investigated.
We are continuing to work on a fix for this issue.
A fix has been implemented and we are monitoring.
The issues have been resolved and the system is now fully operational.
This incident has been resolved.