55 recorded Fmx incidents since July 2018, with official updates, affected components, duration and resolution information.
Investigating reports of users not being able to login to the mobile application
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We are currently investigating this issue.
The issue has been identified and a fix is being implemented.
We are continuing to work on a fix for this issue.
We're adding additional hardware to increase capacity
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
The issue has been identified and a fix is being implemented.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
There have been reports of an elevated number of errors across the application.
We are currently investigating this issue.
The issue has been identified and a fix is being implemented.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We are currently investigating the issue.
The issue has been identified and we are working to remediate the issue.
We're currently rolling out a fix across the affected environments.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We are currently investigating this issue.
We are continuing to investigate this issue.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We're currently investigation an incident with one of our cloud providers.
We have implemented a workaround to the cloud provider incident and are monitoring the situation.
This incident has been resolved.
We are currently investigating this issue.
The issue has been identified and a fix is being implemented.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We are currently investigating the issue.
The issue has been identified and a solution is being implemented.
We are starting the deployment of our fix to the performance issue
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We deployed a critical hotfix to address an issue in our system. While this deployment was designed to occur without downtime, an internal misconfiguration inadvertently triggered maintenance mode. We have identified the issue and are actively working on remediation steps. We sincerely apologize for this interruption and any inconvenience it may have caused.
The main application should be available now and we are still working through the other services
A fix has been implemented and we are now monitoring.
This incident has been resolved. Again. We sincerely apologize for this interruption and any inconvenience this may have caused.
Earlier today, we deployed a hotfix to address a previously identified calendar-related issue. Inadvertently, additional changes were included in the deployment, which resulted in the calendar returning errors and being temporarily inaccessible for approximately 3-4 minutes. The issue was quickly identified and resolved, and full functionality has been restored. We apologize for any inconvenience this may have caused and are reviewing our deployment processes to prevent similar occurrences in the future.
Earlier today, we deployed a critical hotfix to address an issue in our system. While this deployment was designed to occur without downtime, an internal misconfiguration inadvertently triggered maintenance mode, resulting in approximately 5-10 minutes of unexpected downtime. We sincerely apologize for this interruption and any inconvenience it may have caused. We plan to implement measures to prevent this from happening in future deployments and are reviewing our processes to ensure greater reliability moving forward.
We're currently investigating reports of customers that are receiving a 502 error while logging into Mapping and Mobile.
We are continuing to investigate this issue.
The issue has been identified and a fix is being implemented.
We have implemented a fix and are continuing to monitor.
This incident has been resolved.
On 10/22/2024 at approximately 10:30 PM EDT FMX experienced a server failure that required manual intervention to restore it. On 10/23/2024 at 4:10 AM EDT we had successfully restored the affected databases and are continuing to investigate the extent of data that was not restored between 10 and 11 pm EDT
Through our investigation we have identified a subset of customers that were potentially affected by this issue. If you are one of the customers that we have identified, our Customer Success team will be reaching out to you via phone or email. What we know at this time is that we experienced multiple hardware failures that required us to roll back a subset of our databases to an earlier restore point. Any data that was added, edited, or deleted between 10:00pm EDT, and 11:00pm EDT on 10/22/2024 for these identified customers has been corrupted and our Customer Success team will be reaching out to you to answer any further questions. We are still in the process of investigating alternative remediation steps and will be conducting an internal postmortem to better understand and prioritize future infrastructure changes and processes to remove the possibility of this ever happening again. We know that any amount of data corruption can be a problem, and we apologize for any inconvenience this issue has caused.
We have received reports that some users can't access their site. Microsoft has reported that there is an issue between their cloud and AT&T customers. They're currently looking into the issue. You can follow their status page https://azure.status.microsoft/en-us/status for more details. If available, you can try switching over to a mobile hot spot.
The incident has been marked as resolved by Microsoft.
Our Cloud Service Provider is experiencing an outage. We will update this message as we have more details. We apologize for the inconvenience.
We are continuing to monitor and follow our cloud providers status pages and will provide more updates as they become available.
According to our cloud provider they've `determined the underlying cause and are currently applying mitigation through multiple workstreams. ` We're continuing to monitor the situation and will provide updates as they become available.
We are seeing sites come back online. We're continuing to monitor the incident to ensure our cloud service provider's mitigation is successful.
Our cloud provider has deployed a fix that appears to have resolved the current incident. We'll continue to monitor the situation and provide updates as they happen.
This incident has been resolved by our cloud service provider. We will continue to keep an eye out for any further issues. Again, our sincere apologies for any service disruption.
We want to thank you for using FMX to manage your operations. We understand that FMX is often a critical component of running your organization and therefore we take any service disruptions seriously. This postmortem report will help you to better understand what caused the interruption in service as well as how we plan to avoid issues like these in the future. **The root cause of the outage** On the evening of July 18th, we discovered that the FMX platform was unavailable. After investigating we determined that the cause of this was due to an entire region outage at the Microsoft Azure datacenters across the entire US-Central region. You can read more about it [here](https://azure.status.microsoft/en-us/status/history/) from the Azure team. Tracking ID: `1K80-N_8` **Solution** To solve the problem, we simply monitored the situation and waited for the Azure remediations to be implemented. **Future mitigations** On July 30th, we conducted our own internal postmortem discussing future mitigation tactics that we can employ to reduce our reliance on an individual region or cloud provider, improve communication, and improve our monitoring capabilities. This discussion aligned very closely with the [Azure Well-Architected Framework](https://learn.microsoft.com/en-us/azure/well-architected/). Regards, FMX Team
We are currently investigating degraded site performance and errors when generating PDFs. We will update our status page with additional details as they become available. We apologize for any inconvenience.
A fix has been implemented to resolve PDF generation errors. We are continuing to monitor site performance.
This incident has been resolved.
We have identified an issue with our reporting dashboards. We are investigating solutions for a fix. We anticipate that reporting dashboards should be up in a few hours. We'll update this message as we progress on this issue. We apologize for the inconvenience.
We have identified a fix for this issue. We are working on testing the fix now. We'll update this message once our fix passes testing and is out for release. We still anticipate that reporting dashboards should be up in a couple of hours.
A fix has been implemented, and we're monitoring the results.
This incident has been resolved.
We are currently investigating an issue related to our servers. We will report back when we have more information. We apologize for the inconvenience.
We are continuing to investigate an issue related to our servers. We will report back when we have more information. We apologize for the inconvenience.
The application is back online. However, we may see degraded performance until our system can recover from the issue. We will report back when our application performance is back to normal. We apologize for the inconvenience.
This incident has been resolved. Application performance should be back to normal.
