Fmx 장애 기록

1. 2026년 9월 16일 오후 2:18 UTCinvestigating

   모바일 애플리케이션에 로그인할 수 없는 사용자의 조사 보고서

2. 2026년 9월 16일 오후 2:37 UTCmonitoring

   수정이 구현되어 결과를 모니터링하고 있습니다.

3. 2026년 9월 16일 오후 2:56 UTCresolved

   이 사건은 해결되었습니다.

공식 인시던트 업데이트를 자동 번역했습니다.

1. 2026년 9월 14일 오후 1:17 UTCinvestigating

   우리는 현재이 문제를 조사하고 있습니다.

2. 2026년 9월 14일 오후 1:47 UTCidentified

   문제는 식별되고 수정이 구현됩니다.

3. 2026년 9월 14일 오후 2:06 UTCidentified

   이 문제를 해결하기 위해 계속 노력하고 있습니다.

4. 2026년 9월 14일 오후 2:50 UTCidentified

   용량을 늘리는 추가 하드웨어

5. 2026년 9월 14일 오후 3:15 UTCmonitoring

   수정이 구현되어 결과를 모니터링하고 있습니다.

6. 2026년 9월 15일 오후 1:18 UTCresolved

   이 사건은 해결되었습니다.

공식 인시던트 업데이트를 자동 번역했습니다.

1. 2026년 9월 4일 오후 5:50 UTCidentified

   문제는 식별되고 수정이 구현됩니다.

2. 2026년 9월 4일 오후 5:58 UTCmonitoring

   수정이 구현되어 결과를 모니터링하고 있습니다.

3. 2026년 9월 4일 오후 6:18 UTCresolved

   이 사건은 해결되었습니다.

공식 인시던트 업데이트를 자동 번역했습니다.

1. 2026년 9월 4일 오후 1:04 UTCresolved

   응용 프로그램에서 높은 수준의 오류의 보고서가 있습니다.

공식 인시던트 업데이트를 자동 번역했습니다.

1. 2026년 8월 5일 오후 5:15 UTCinvestigating

   We are currently investigating this issue.

2. 2026년 8월 5일 오후 5:35 UTCidentified

   The issue has been identified and a fix is being implemented.

3. 2026년 8월 5일 오후 5:54 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

4. 2026년 8월 5일 오후 6:24 UTCresolved

   This incident has been resolved.

1. 2026년 8월 2일 오후 4:40 UTCinvestigating

   현재 이 문제를 조사하고 있습니다.

2. 2026년 8월 2일 오후 5:16 UTCidentified

   문제가 확인되었습니다. 우리는 문제를 재조정하기 위해 노력하고 있습니다.

3. 2026년 8월 2일 오후 7:45 UTCidentified

   우리는 현재 영향을받는 환경에 대한 수정을 압연하고 있습니다.

4. 2026년 8월 2일 오후 8:33 UTCmonitoring

   수정이 구현되어 결과를 모니터링하고 있습니다.

5. 2026년 8월 2일 오후 8:41 UTCresolved

   이 사건은 해결되었습니다.

공식 인시던트 업데이트를 자동 번역했습니다.

1. 2025년 10월 30일 오후 5:59 UTCinvestigating

   We are currently investigating this issue.

2. 2025년 10월 30일 오후 6:00 UTCinvestigating

   We are continuing to investigate this issue.

3. 2025년 10월 30일 오후 6:30 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

4. 2025년 10월 30일 오후 6:37 UTCresolved

   This incident has been resolved.

1. 2025년 10월 29일 오후 4:24 UTCinvestigating

   We're currently investigation an incident with one of our cloud providers.

2. 2025년 10월 29일 오후 4:48 UTCmonitoring

   We have implemented a workaround to the cloud provider incident and are monitoring the situation.

3. 2025년 10월 30일 오후 1:28 UTCresolved

   This incident has been resolved.

1. 2025년 10월 2일 오후 2:35 UTCinvestigating

   We are currently investigating this issue.

2. 2025년 10월 2일 오후 2:47 UTCidentified

   The issue has been identified and a fix is being implemented.

3. 2025년 10월 2일 오후 3:02 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

4. 2025년 10월 2일 오후 3:19 UTCresolved

   This incident has been resolved.

1. 2025년 9월 29일 오후 12:59 UTCinvestigating

   We are currently investigating the issue.

2. 2025년 9월 29일 오후 1:03 UTCidentified

   The issue has been identified and a solution is being implemented.

3. 2025년 9월 29일 오후 2:19 UTCidentified

   We are starting the deployment of our fix to the performance issue

4. 2025년 9월 29일 오후 2:46 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

5. 2025년 9월 29일 오후 3:01 UTCresolved

   This incident has been resolved.

1. 2025년 4월 14일 오후 3:57 UTCidentified

   We deployed a critical hotfix to address an issue in our system. While this deployment was designed to occur without downtime, an internal misconfiguration inadvertently triggered maintenance mode. We have identified the issue and are actively working on remediation steps. We sincerely apologize for this interruption and any inconvenience it may have caused.

2. 2025년 4월 14일 오후 4:00 UTCidentified

   The main application should be available now and we are still working through the other services

3. 2025년 4월 14일 오후 4:11 UTCmonitoring

   A fix has been implemented and we are now monitoring.

4. 2025년 4월 14일 오후 4:38 UTCresolved

   This incident has been resolved. Again. We sincerely apologize for this interruption and any inconvenience this may have caused.

1. 2025년 3월 10일 오후 3:02 UTCresolved

   Earlier today, we deployed a hotfix to address a previously identified calendar-related issue. Inadvertently, additional changes were included in the deployment, which resulted in the calendar returning errors and being temporarily inaccessible for approximately 3-4 minutes. The issue was quickly identified and resolved, and full functionality has been restored. We apologize for any inconvenience this may have caused and are reviewing our deployment processes to prevent similar occurrences in the future.

1. 2025년 3월 3일 오후 9:10 UTCresolved

   Earlier today, we deployed a critical hotfix to address an issue in our system. While this deployment was designed to occur without downtime, an internal misconfiguration inadvertently triggered maintenance mode, resulting in approximately 5-10 minutes of unexpected downtime. We sincerely apologize for this interruption and any inconvenience it may have caused. We plan to implement measures to prevent this from happening in future deployments and are reviewing our processes to ensure greater reliability moving forward.

1. 2024년 11월 19일 오후 7:06 UTCinvestigating

   We're currently investigating reports of customers that are receiving a 502 error while logging into Mapping and Mobile.

2. 2024년 11월 19일 오후 7:09 UTCinvestigating

   We are continuing to investigate this issue.

3. 2024년 11월 19일 오후 7:13 UTCidentified

   The issue has been identified and a fix is being implemented.

4. 2024년 11월 19일 오후 7:18 UTCmonitoring

   We have implemented a fix and are continuing to monitor.

5. 2024년 11월 19일 오후 7:25 UTCresolved

   This incident has been resolved.

1. 2024년 10월 23일 오후 1:57 UTCinvestigating

   On 10/22/2024 at approximately 10:30 PM EDT FMX experienced a server failure that required manual intervention to restore it. On 10/23/2024 at 4:10 AM EDT we had successfully restored the affected databases and are continuing to investigate the extent of data that was not restored between 10 and 11 pm EDT

2. 2024년 10월 24일 오후 1:11 UTCresolved

   Through our investigation we have identified a subset of customers that were potentially affected by this issue. If you are one of the customers that we have identified, our Customer Success team will be reaching out to you via phone or email. What we know at this time is that we experienced multiple hardware failures that required us to roll back a subset of our databases to an earlier restore point. Any data that was added, edited, or deleted between 10:00pm EDT, and 11:00pm EDT on 10/22/2024 for these identified customers has been corrupted and our Customer Success team will be reaching out to you to answer any further questions. We are still in the process of investigating alternative remediation steps and will be conducting an internal postmortem to better understand and prioritize future infrastructure changes and processes to remove the possibility of this ever happening again. We know that any amount of data corruption can be a problem, and we apologize for any inconvenience this issue has caused.

1. 2024년 9월 12일 오후 1:11 UTCinvestigating

   We have received reports that some users can't access their site. Microsoft has reported that there is an issue between their cloud and AT&T customers. They're currently looking into the issue. You can follow their status page https://azure.status.microsoft/en-us/status for more details. If available, you can try switching over to a mobile hot spot.

2. 2024년 9월 12일 오후 2:33 UTCresolved

   The incident has been marked as resolved by Microsoft.

1. 2024년 7월 18일 오후 10:52 UTCinvestigating

   Our Cloud Service Provider is experiencing an outage. We will update this message as we have more details. We apologize for the inconvenience.

2. 2024년 7월 19일 오전 12:56 UTCinvestigating

   We are continuing to monitor and follow our cloud providers status pages and will provide more updates as they become available.

3. 2024년 7월 19일 오전 1:38 UTCinvestigating

   According to our cloud provider they've `determined the underlying cause and are currently applying mitigation through multiple workstreams. ` We're continuing to monitor the situation and will provide updates as they become available.

4. 2024년 7월 19일 오전 1:43 UTCinvestigating

   We are seeing sites come back online. We're continuing to monitor the incident to ensure our cloud service provider's mitigation is successful.

5. 2024년 7월 19일 오전 2:47 UTCmonitoring

   Our cloud provider has deployed a fix that appears to have resolved the current incident. We'll continue to monitor the situation and provide updates as they happen.

6. 2024년 7월 19일 오전 5:02 UTCresolved

   This incident has been resolved by our cloud service provider. We will continue to keep an eye out for any further issues. Again, our sincere apologies for any service disruption.

7. 2024년 7월 31일 오후 8:29 UTCpostmortem

   We want to thank you for using FMX to manage your operations. We understand that FMX is often a critical component of running your organization and therefore we take any service disruptions seriously. This postmortem report will help you to better understand what caused the interruption in service as well as how we plan to avoid issues like these in the future. **The root cause of the outage** On the evening of July 18th, we discovered that the FMX platform was unavailable. After investigating we determined that the cause of this was due to an entire region outage at the Microsoft Azure datacenters across the entire US-Central region. You can read more about it [here](https://azure.status.microsoft/en-us/status/history/) from the Azure team. Tracking ID: `1K80-N_8` **Solution** To solve the problem, we simply monitored the situation and waited for the Azure remediations to be implemented. **Future mitigations** On July 30th, we conducted our own internal postmortem discussing future mitigation tactics that we can employ to reduce our reliance on an individual region or cloud provider, improve communication, and improve our monitoring capabilities. This discussion aligned very closely with the [Azure Well-Architected Framework](https://learn.microsoft.com/en-us/azure/well-architected/). ‌ Regards, FMX Team

1. 2023년 10월 2일 오후 12:38 UTCinvestigating

   We are currently investigating degraded site performance and errors when generating PDFs. We will update our status page with additional details as they become available. We apologize for any inconvenience.

2. 2023년 10월 2일 오후 1:09 UTCmonitoring

   A fix has been implemented to resolve PDF generation errors. We are continuing to monitor site performance.

3. 2023년 10월 2일 오후 3:05 UTCresolved

   This incident has been resolved.

1. 2023년 4월 24일 오후 12:16 UTCinvestigating

   We have identified an issue with our reporting dashboards. We are investigating solutions for a fix. We anticipate that reporting dashboards should be up in a few hours. We'll update this message as we progress on this issue. We apologize for the inconvenience.

2. 2023년 4월 24일 오후 1:56 UTCidentified

   We have identified a fix for this issue. We are working on testing the fix now. We'll update this message once our fix passes testing and is out for release. We still anticipate that reporting dashboards should be up in a couple of hours.

3. 2023년 4월 24일 오후 4:01 UTCmonitoring

   A fix has been implemented, and we're monitoring the results.

4. 2023년 4월 24일 오후 5:36 UTCresolved

   This incident has been resolved.

1. 2023년 3월 10일 오후 5:03 UTCinvestigating

   We are currently investigating an issue related to our servers. We will report back when we have more information. We apologize for the inconvenience.

2. 2023년 3월 10일 오후 5:19 UTCinvestigating

   We are continuing to investigate an issue related to our servers. We will report back when we have more information. We apologize for the inconvenience.

3. 2023년 3월 10일 오후 5:32 UTCidentified

   The application is back online. However, we may see degraded performance until our system can recover from the issue. We will report back when our application performance is back to normal. We apologize for the inconvenience.

4. 2023년 3월 10일 오후 5:44 UTCresolved

   This incident has been resolved. Application performance should be back to normal.