Front 장애 기록

1. 2026년 8월 15일 오후 8:11 UTCinvestigating

   우리는 현재이 문제를 조사하고 있습니다.

2. 2026년 8월 15일 오후 8:12 UTCinvestigating

   일부 고객 (cell 14)은 시스템 오류로 인해 감사 로그에 누락 된 리소스 변경 이벤트를 볼 수 있습니다. 고객 메시지 데이터는 영향을받지 않고 감사 로그 만합니다.

3. 2026년 8월 15일 오후 8:44 UTCresolved

   이 사건은 해결되었습니다.

공식 인시던트 업데이트를 자동 번역했습니다.

1. 2026년 5월 28일 오후 6:24 UTCinvestigating

   We are currently investing an issue where a subset of customers are seeing offline O365 channels.

2. 2026년 5월 28일 오후 6:57 UTCidentified

   We are attempting a mitigation step to get impacted channels back online.

3. 2026년 5월 28일 오후 7:35 UTCidentified

   Front continues to investigate paths to restore access for affected Office365 channels. Office reports internal errors, which we are trying to work around.

4. 2026년 5월 28일 오후 11:36 UTCidentified

   Front is partnering with Microsoft to investigate the channel sync failures and will continue to update here as we make progress. Manually re-authorizing the channel in Front's channel settings has worked in some cases, but not all. Please contact [email protected] for assistance.

5. 2026년 5월 29일 오전 2:05 UTCidentified

   Front is continuing to partner with Microsoft to investigate the channel sync failures. The next update will be within the next 8 hours. Manually re-authorizing the channel in Front's channel settings has worked in some cases, but not all. Please contact [email protected] for assistance.

6. 2026년 5월 29일 오후 11:40 UTCidentified

   Front is continuing to partner with Microsoft to investigate the channel sync failures. The next update will be within the next 6 hours. Manually re-authorizing the channel in Front's channel settings has worked in some cases, but not all. Please contact [email protected] for assistance.

7. 2026년 5월 30일 오후 4:04 UTCidentified

   Front is continuing to partner with Microsoft to investigate the decreasing volume of channel sync failures. The next update will be within the next 12 hours. Manually re-authorizing the channel in Front's channel settings has worked in most cases, but not all. Please contact [email protected] for assistance.

8. 2026년 5월 31일 오후 3:22 UTCidentified

   Front is continuing to partner with Microsoft to investigate the channel sync failures. The next update will be within the next 12 hours. Manually re-authorizing the channel in Front's channel settings has worked in most cases, but not all. Please contact [email protected] for assistance.

9. 2026년 6월 1일 오후 3:30 UTCidentified

   Front is continuing to partner with Microsoft to investigate the channel sync failures. The next update will be within the next 12 hours. Manually re-authorizing the channel in Front's channel settings has worked in majority of cases, but not all. Please contact [email protected] for assistance.

10. 2026년 6월 1일 오후 11:35 UTCresolved

    Front is continuing to partner with Microsoft on root cause of remaining channel sync failures. We added the ability to automatically bring these channels back online to prevent further customer impact.

1. 2026년 4월 25일 오후 6:37 UTCinvestigating

   Counters (numbers in the navigation bar of the app) are updating with a delay for some customers [us-west-1]

2. 2026년 4월 25일 오후 7:09 UTCresolved

   This incident has been resolved.

1. 2026년 4월 14일 오후 5:03 UTCmonitoring

   There was a delay in message delivery of up to an hour. Messages are now being delivered without delay. We are continuing to monitor

2. 2026년 4월 14일 오후 8:00 UTCresolved

   This incident has been resolved.

1. 2026년 2월 23일 오후 3:17 UTCinvestigating

   Some customers are experiencing delays in inbound messages being received from Gmail and Office channels (typically 2-4 minutes of delay). We are currently scaling up our systems to address this.

2. 2026년 2월 23일 오후 3:51 UTCinvestigating

   Delays have been reduced, but remain slightly elevated. We are continuing to investigate.

3. 2026년 2월 23일 오후 4:36 UTCinvestigating

   Office delays have been resolved. Gmail delays have reduced further (to approximately 1 minute).

4. 2026년 2월 23일 오후 4:48 UTCmonitoring

   Delays are resolved for both Gmail and Outlook channels. We're continuing to monitor to ensure we're at a stable steady-state.

5. 2026년 2월 23일 오후 6:10 UTCresolved

   This incident has been resolved.

1. 2026년 2월 18일 오후 2:59 UTCinvestigating

   We are currently investigating this issue.

2. 2026년 2월 18일 오후 3:14 UTCidentified

   The issue has been identified and a fix is being implemented.

3. 2026년 2월 18일 오후 4:01 UTCidentified

   We are continuing to work on a fix for this issue.

4. 2026년 2월 18일 오후 4:50 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

5. 2026년 2월 18일 오후 5:08 UTCresolved

   This incident has been resolved.

1. 2026년 2월 7일 오전 12:03 UTCinvestigating

   We are currently investigating an issue that is causing delayed message import. Impacting all channels all regions

2. 2026년 2월 7일 오전 12:26 UTCresolved

   This incident has been resolved.

1. 2026년 1월 23일 오후 9:51 UTCidentified

   Front is currently inaccessible. We have identified the cause and are working on a resolution.

2. 2026년 1월 23일 오후 9:53 UTCidentified

   We are continuing to work on a fix for this issue.

3. 2026년 1월 23일 오후 10:28 UTCmonitoring

   Application and API access have been restored and we are continuing to monitor the system.

4. 2026년 1월 24일 오전 1:40 UTCresolved

   This incident has been resolved. A fix has been implemented.

1. 2026년 1월 22일 오후 8:49 UTCidentified

   Microsoft Office365 is experiencing an outage affecting some Front channels: "Users may be seeing degraded service functionality or be unable to access multiple Microsoft 365 services." Front is unable to send or receive messages for affected customers. Their status page is available at https://status.cloud.microsoft/.

2. 2026년 1월 23일 오전 3:28 UTCmonitoring

   Microsoft appears to have recovered, and we're no longer seeing any impact on Office 365 channels. All pending messages have been delivered. We're continuing to monitor the situation closely.

3. 2026년 1월 23일 오전 5:36 UTCresolved

   Microsoft is still reporting service degradation, but it's not impacting Front. We'll continue monitoring the situation in case it changes.

1. 2025년 12월 19일 오후 3:13 UTCinvestigating

   We are currently investigating the issue. [us-west-1], [us-west-2]

2. 2025년 12월 19일 오후 4:06 UTCinvestigating

   We are continuing to investigate this issue.

3. 2025년 12월 19일 오후 4:47 UTCidentified

   The issue has been identified and a fix is being implemented.

4. 2025년 12월 19일 오후 5:29 UTCmonitoring

   A fix has been implemented and we are monitoring the results. We are seeing recovery but continuing to close out remaining recovery items.

5. 2025년 12월 19일 오후 5:57 UTCmonitoring

   We are continuing to monitor and close our our remaining recovery items for [us-west-2] customers. Other regions are recovered.

6. 2025년 12월 19일 오후 6:23 UTCmonitoring

   Front is operational for all customers. We are continuing to backfill any missed messages and application webhooks.

7. 2025년 12월 19일 오후 9:23 UTCresolved

   All backfills complete

8. 2025년 12월 19일 오후 11:21 UTCpostmortem

   On Friday, Dec 19, at 14:50 UTC \(6:50am PST\), customers backed in our US-West-2 data center experienced dramatically increased API latency, resulting in the website failing to load and messages being queued in the backend. This continued until 18:10 UTC \(10:10am PST\). During this time no messages were lost, though there may have been a significant delay for messages to appear in customer inboxes. All queued messages were delivered by 21:00 \(1:00pm PST\). ‌ Customers based in Front’s EU-West-1 and US-West-1 datacenters may have experienced some delays during this time, as some systems are interdependent, but this impact was intermittent and uncommon. ‌ The root cause of this issue was the failure of a caching system. There are several database systems that support the Front application, which are supported by caching to improve performance. A recent change increased the size of some objects in the cache layer. This is not inherently wrong, and did not have any immediate impact. On Friday the 19th the caching layer in US-West-2 crossed a new threshold of data volume which triggered a large number of evictions, particularly of other data that is necessary for most application activity. Besides putting additional load on the databases, there was simply not enough room in the cache for all the data we needed to store there. This caused a high amount of thrashing that significantly increased latency for all systems.

1. 2025년 12월 10일 오전 1:06 UTCinvestigating

   We are currently investigating an issue that is causing errors loading conversations. We believe it is limited in scope to a subset of customers who use our slack integration

2. 2025년 12월 10일 오전 1:11 UTCidentified

   We believe we have identified the issue, currently working on implementing a fix

3. 2025년 12월 10일 오전 1:23 UTCidentified

   We are rolling out the fix now

4. 2025년 12월 10일 오전 1:53 UTCmonitoring

   We've rolled out a fix to the issue, all conversations should now load. Please reach out to support if you continue to experience any issues

5. 2025년 12월 10일 오후 4:00 UTCresolved

   This incident has been resolved.

1. 2025년 12월 1일 오후 10:18 UTCidentified

   We have identified an issue where delegated inboxes are not appearing in the navigation panel. We are working on a fix for this issue and will provide another update once there is an ETA for the resolution.

2. 2025년 12월 1일 오후 10:25 UTCidentified

   We are continuing to work on a fix for this issue.

3. 2025년 12월 1일 오후 10:26 UTCresolved

   We have restored delegated inboxes. If delegated inboxes are not currently visible, please restart the application.

1. 2025년 11월 21일 오후 7:21 UTCinvestigating

   We are currently investigating an outage.

2. 2025년 11월 21일 오후 7:48 UTCmonitoring

   A rollback has started recovering the failures in us-west-1. We are monitoring and further investigating root cause

3. 2025년 11월 21일 오후 7:57 UTCmonitoring

   Recovering messages for all regions but restoring back to operational state. Continuing to monitor

4. 2025년 11월 21일 오후 8:15 UTCmonitoring

   Fully recovered for us-west-1 and eu-west-1 regions. Continued recovery for us-west-2 customers

5. 2025년 11월 21일 오후 8:18 UTCmonitoring

   Us-West-2 customers are now recovered. We are continuing to monitor

6. 2025년 11월 21일 오후 8:40 UTCmonitoring

   Investigating continued outages for [by_company_00017] customers only.

7. 2025년 11월 21일 오후 9:15 UTCmonitoring

   Continuing investigation for [by_company_00017] customers only.

8. 2025년 11월 21일 오후 9:45 UTCmonitoring

   Recovering app for [by_company_00017] customers, delay for rules and calendar remains.

9. 2025년 11월 21일 오후 10:18 UTCresolved

   This incident has been resolved.

10. 2025년 11월 21일 오후 11:59 UTCpostmortem

    At 19:05 UTC \(11:05 am PST\), Front began a routine application deployment. Front typically deploys new versions of the application multiple times per day as part of normal operation. This app release included a change in the way we connect to an internal caching system. The change passed all our testing requirements, but due to an environment-specific configuration on the caching system it immediately caused a problem in our customer environment. ‌ Front was alerted within minutes of the increased error rate and immediately started investigating. At 19:20 UTC we initiated a rollback of the deployment across all regions. Because of the progressive nature of the rollback, some customers saw the application recover quickly after while for others it took up to 30 minutes. By 19:50 all rollbacks were complete and most customers were back to normal. ‌ However, for one slice of Front customers the rollback triggered a second issue that kept the application from recovering. For clarity, Front divides customers into one of 40 “cells” that isolate them from one another. This architecture is designed to limit the scope of certain failures, like this one. But for the 2% of customers on the affected cell we had to take additional steps to recover. ‌ This second issue was triggered by a spike in database load caused by the stampede of traffic from customers returning to the application. In this cell a particular workload caused high database contention that ultimately timed out and then started over, which prevented us from escaping the problem. Traffic continued to back up and we were unable to progress through all the messages. ‌ Once we identified that the problematic workload was in the evaluation of rules, we were able to block all rules from processing. This allowed the database and the application to immediately recover, coming back online for users at 21:39 UTC. Rule evaluations continued to back up, and we were able to slowly restart processing to make sure the database didn’t reenter a bad state. All rule evaluations were caught up by 22:20 UTC. ‌ We would like to apologize for the disruption this outage has caused and for the unusually long duration. In the immediate aftermath of the incident we have identified a number of actions we can take to prevent this kind of event from occurring again. ‌ The first step is to ensure we have a test environment that appropriately represents the configuration of the production customer environment for the caching system. This issue should have been detected before we ever deployed it. The second task is to investigate the rollback system to see if we can safely improve the speed, so that if we have a similar issue in the future we can recover even faster.

1. 2025년 10월 9일 오후 9:29 UTCinvestigating

   Some Canadian IP addresses are experiencing app slowness and/or degraded email message performance.

2. 2025년 10월 9일 오후 10:14 UTCinvestigating

   We are continuing to investigate this issue.

3. 2025년 10월 9일 오후 11:19 UTCinvestigating

   We are still investigating the issue and searching for the root cause.

4. 2025년 10월 10일 오전 2:15 UTCinvestigating

   We are continuing to investigate this issue, and are yet to identify a root cause Some impacted users have been able to workaround the issue by using a VPN to re-route their traffic, or by tethering to a mobile device. We will update this page as soon as we have more information

5. 2025년 10월 10일 오후 5:39 UTCmonitoring

   We've received reports this morning that users are no longer encountering issues. We are continuing to monitor, please reach out to support if you are experiencing degraded performance

6. 2025년 10월 11일 오전 12:20 UTCresolved

   We've had no further reports of issues. Please reach out to Front support if you experience degradation in the app

1. 2025년 9월 26일 오후 4:47 UTCinvestigating

   App is not loading in specific regions and timeouts have been spiking for some API calls

2. 2025년 9월 26일 오후 4:56 UTCresolved

   A cache node was lost and automatically recovered in 6 minutes. All systems should now be operational.

1. 2025년 9월 19일 오후 6:40 UTCinvestigating

   We've have received reports of the google sign in flow failing to redirect back to the mobile app successfully, and have been able to reproduce the issue We are currently investigating the problem

2. 2025년 9월 19일 오후 8:08 UTCinvestigating

   We have isolated the issue to iOS devices, Android customers should be unaffected Continuing to investigate

3. 2025년 9월 19일 오후 10:51 UTCidentified

   We've identified the issue, and are still working on a fix. In the meantime we have identified a workaround that appears to be working on some clients - by navigating directly to https://app.frontapp.com/oauth/google-signin?client_type=universalLink in Safari on iOS and signing you should get redirected to the app in a signed in state

4. 2025년 9월 23일 오전 2:13 UTCidentified

   We've released a new version of the app (5.69.1) that fixes the behavior in some cases, but does not fully address the issue. We are in progress on a full fix. For clients still unable to login we have identified a workaround that appears to be working on some clients - by navigating directly to https://app.frontapp.com/oauth/google-signin?client_type=universalLink in Safari on iOS and signing you should get redirected to the app in a signed in state

5. 2025년 9월 23일 오전 6:11 UTCmonitoring

   We've implemented a fix - you can download the latest version from the app store (5.69.2) to test. Please let us know if you experience any issues on this version

6. 2025년 9월 23일 오전 9:48 UTCresolved

   We have confirmed reports that the latest version of the iOS app (5.69.2) addresses the issue. If you run into login issues, ensure you've updated to the latest version, and reach out if the problem persists.

1. 2025년 9월 16일 오후 10:02 UTCinvestigating

   Front is currently investigating an issue in which trying to load some conversations results in an error message.

2. 2025년 9월 16일 오후 10:26 UTCidentified

   Conversations that leveraged Suggest Reply between 21:30 and 21:50 UTC (2:30pm – 2:50pm PDT) may cause an error when loading. Front has identified the root cause and is deploying a fix that will allow these messages to display.

3. 2025년 9월 16일 오후 10:52 UTCmonitoring

   A fix for the affected conversations has now been deployed, and initial tests show all conversations should now be visible. Please make sure to refresh your browser or app if you don't see this right away.

4. 2025년 9월 16일 오후 11:23 UTCresolved

   Front has now completed the fix and cleanup associated with this incident. If you continue to see errors loading conversations, please make sure you've done a browser or app refresh, and if the issue persists please contact support. Summary: Conversations for which Front attempted to provide a Suggested Reply between 21:30 – 21:50 UTC may have been unreadable until the fix was completed at 22:49 UTC. During that initial time, suggested replies were generated with an additional field that caused an unrecoverable error in the frontend. No data or messages were lost or corrupted during the incident. Suggested replies for conversations that arrived during that time may need to be regenerated.

1. 2025년 9월 12일 오후 10:32 UTCidentified

   We have an ongoing issue causing delays importing new messages from all gmail channels. We have identified the issue and are currently working on a fix

2. 2025년 9월 12일 오후 10:44 UTCmonitoring

   Fixes are currently rolling out, we are monitoring backlog of unimported messages

3. 2025년 9월 12일 오후 11:31 UTCmonitoring

   Continuing to monitor our backlog of messages, the majority of channels have recovered - but still have a small segment still catching up

4. 2025년 9월 13일 오전 12:00 UTCresolved

   All channels should now be recovered

1. 2025년 9월 5일 오후 1:41 UTCinvestigating

   We are investigating reports of Google SSO and Office SSO login issues on our mobile apps (iOS and Android).

2. 2025년 9월 5일 오후 2:41 UTCinvestigating

   We are continuing to investigate this issue.

3. 2025년 9월 5일 오후 4:41 UTCidentified

   The issue has been identified and a fix is being implemented.

4. 2025년 9월 5일 오후 5:09 UTCresolved

   This incident has been resolved.

1. 2025년 9월 1일 오후 2:05 UTCinvestigating

   We are currently investigating reports of increased latency affecting a subset of customers in the EU region. Impact: Some customers in EU may experience slower response times or intermittent delays when using our app. Not all customers in this region are affected.

2. 2025년 9월 1일 오후 2:05 UTCinvestigating

   We are continuing to investigate this issue.

3. 2025년 9월 1일 오후 2:53 UTCidentified

   The issue has been identified and a fix is being implemented.

4. 2025년 9월 1일 오후 3:02 UTCmonitoring

   Response times are stabilizing, though we continue to actively monitor the situation to ensure the issue is fully resolved.

5. 2025년 9월 1일 오후 5:52 UTCresolved

   This incident has been resolved.