Flatfile 장애 기록

1. 2026년 9월 15일 오후 2:51 UTCinvestigating

   우리는 현재이 문제를 조사하고 있습니다.

2. 2026년 9월 15일 오후 10:55 UTCresolved

   이 사건은 해결되었습니다.

공식 인시던트 업데이트를 자동 번역했습니다.

1. 2026년 9월 14일 오후 3:35 UTCinvestigating

   Existing workbooks는 저장에서 복원 할 때 느리고 degraded. 우리는 뿌리 원인을 조사하고 있습니다.

2. 2026년 9월 14일 오후 8:41 UTCmonitoring

   수정이 구현되어 결과를 모니터링하고 있습니다.

3. 2026년 9월 14일 오후 10:38 UTCresolved

   이 사건은 해결되었습니다.

공식 인시던트 업데이트를 자동 번역했습니다.

1. 2026년 9월 2일 오후 2:54 UTCinvestigating

   Mapping 및 기타 서비스는 다운됩니다. 우리는 문제를 조사하고 있습니다.

2. 2026년 9월 2일 오후 3:29 UTCmonitoring

   수정이 구현되어 결과를 모니터링하고 있습니다.

3. 2026년 9월 2일 오후 3:39 UTCmonitoring

   우리는 더 많은 문제를 모니터링하고 있습니다.

4. 2026년 9월 2일 오후 3:48 UTCresolved

   이 사건은 해결되었습니다.

공식 인시던트 업데이트를 자동 번역했습니다.

1. 2026년 6월 10일 오후 5:28 UTCresolved

   Timestamp: 16:46 UTC to 17:25 UTC

1. 2026년 6월 10일 오후 5:00 UTCresolved

   Retroactive: Occurred Tue Jun 9, 2026 16:48:14 UTC to Wed Jun 10, 2026 12:02 UTC , Resolved

1. 2026년 5월 29일 오후 5:10 UTCresolved

   2026-05-29 13:45 UTC to 14:12 UTC Customers experienced intermittent failures when uploading and processing workbook files. The issue was traced to elevated resource utilization on internal processing workers, which caused a backlog in our job queue and prevented files from completing extraction. Our team identified the problem, restarted the affected services, and confirmed full restoration of functionality. We are implementing infrastructure improvements to prevent recurrence under similar load conditions.

1. 2026년 5월 29일 오후 5:06 UTCresolved

   2026-05-28 23:14 UTC to 2026-05-29 00:19 UTC Customers experienced intermittent failures when uploading and processing workbook files. The issue was traced to elevated resource utilization on internal processing workers, which caused a backlog in our job queue and prevented files from completing extraction. Our team identified the problem, restarted the affected services, and confirmed full restoration of functionality. We are implementing infrastructure improvements to prevent recurrence under similar load conditions.

1. 2026년 4월 10일 오후 5:24 UTCidentified

   The issue has been identified and a fix is being implemented.

2. 2026년 4월 10일 오후 8:17 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

3. 2026년 4월 10일 오후 8:24 UTCresolved

   This incident has been resolved.

1. 2026년 2월 2일 오후 3:07 UTCinvestigating

   We are currently investigating reports of degraded performance with mapping in spaces.

2. 2026년 2월 2일 오후 4:32 UTCresolved

   This incident has been resolved.

1. 2025년 12월 18일 오후 4:31 UTCidentified

   The issue has been identified and a fix is being implemented.

2. 2025년 12월 18일 오후 4:54 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

3. 2025년 12월 18일 오후 5:44 UTCresolved

   This incident has been resolved.

1. 2025년 12월 18일 오후 1:32 UTCinvestigating

   We are investigating reports of space initialization and creation issues

2. 2025년 12월 18일 오후 1:38 UTCidentified

   We have identified the root cause and a fix is currently being rolled out

3. 2025년 12월 18일 오후 2:01 UTCidentified

   A fix has been implemented and we are monitoring the results.

4. 2025년 12월 18일 오후 2:01 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

5. 2025년 12월 18일 오후 2:54 UTCresolved

   The incident has been resolved.

1. 2025년 11월 18일 오후 2:06 UTCinvestigating

   An issue with one of our third-party service providers

2. 2025년 11월 18일 오후 3:12 UTCmonitoring

   A fix has been implemented and we are monitoring the recovery.

3. 2025년 11월 18일 오후 3:25 UTCresolved

   This incident has been resolved.

1. 2025년 11월 11일 오후 7:52 UTCinvestigating

   We are currently investigating this issue.

2. 2025년 11월 11일 오후 7:56 UTCidentified

   The issue has been identified and a fix is being implemented.

3. 2025년 11월 11일 오후 8:25 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

4. 2025년 11월 11일 오후 9:25 UTCresolved

   This incident has been resolved.

1. 2025년 11월 6일 오후 3:44 UTCinvestigating

   We are currently investigating an issue where some users are seeing an increased number of 400 errors from the API with the message "remaining connection slots are reserved for roles with the SUPERUSER attribute".

2. 2025년 11월 6일 오후 4:24 UTCinvestigating

   We are no longer seeing these errors however we are continuing to investigate the cause.

3. 2025년 11월 6일 오후 4:44 UTCidentified

   The issue has been identified and a fix is being implemented.

4. 2025년 11월 6일 오후 4:49 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

5. 2025년 11월 6일 오후 6:53 UTCresolved

   This incident has been resolved.

1. 2025년 11월 4일 오후 4:15 UTCinvestigating

   We are currently investigating this issue.

2. 2025년 11월 4일 오후 4:51 UTCidentified

   The issue has been identified and a fix is being implemented.

3. 2025년 11월 4일 오후 4:57 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

4. 2025년 11월 4일 오후 5:33 UTCmonitoring

   We are continuing to monitor for any further issues.

5. 2025년 11월 4일 오후 6:51 UTCresolved

   This incident has been resolved.

6. 2025년 11월 7일 오후 4:17 UTCpostmortem

   # Incident Overview **Date and Time of Incident:** November 4 08:15 PST - 10:51 PST **Nature of Incident:** Exhausted Database Connections due to Thawing Expired Workbooks **Services Affected:** Flatfile Platform US Region ## Details of the Incident On November 4 a daily event expired a large number of workbooks. Many of the expired workbooks had been backed up and moved to cold storage. The expiry event caused these workbooks to first thaw before being marked as expired and purged. This led to elevated database connections growing for a period of time that ultimately exhausted database resources at approximately 8:15 PST. This caused intermittent service degradations across the platform that manifested as API requests randomly failing when the API was unable to open a database connection. Flatfile engineering took steps to short circuit the thawing process for expired workbooks and hardened the thaw process so that a large number of workbooks moving from cold storage to “hot” storage would not spike database resources. ## Impact Assessment During the incident window, some API requests would randomly fail leading to unpredictable behavior across the Platform. ## Root Cause The data retention policy feature allows a customer to automatically purge workbooks that have not been active in a set amount of days. This feature would run on a cron-like basis to identify and purge workbooks. It was discovered that this conflicted with the cold storage system for workbooks where any workbook that was backed up and moved to “cold” storage \(S3 backup\) would first be “thawed” and moved into “hot” storage. In the early morning of the 4th, a large number of workbooks in cold storage were identified for expiry and several thousand workbooks entered the queue to await thaw. Over the course of several hours, the database connection pool became saturated causing API requests to intermittently fail while also causing workbooks in the thaw process to fail and be re-enqueued which exacerbated the connection pool problem. ## Resolution Flatfile engineering took steps to implement a short circuit for expired workbooks to resolve the immediate symptoms. The engineering team followed up with a series of steps to harden the queue, the database connection pool, and the thaw mechanism as well. ## Security and Data Integrity Please be assured that this incident did not compromise the security or integrity of your data. Our commitment to data protection remains a top priority.

1. 2025년 10월 23일 오전 10:13 UTCinvestigating

   We are currently investigating reports of a service degredation on our EU platform region

2. 2025년 10월 23일 오전 10:22 UTCidentified

   We have identified the issue and are rolling out a fix

3. 2025년 10월 23일 오전 10:51 UTCresolved

   A fix has

1. 2025년 9월 12일 오후 3:55 UTCinvestigating

   We are currently investigating an issue where some users are seeing an "Unknown Error" when launching a space. We are investigating this issue however refreshing the page seems to resolve this error.

2. 2025년 9월 12일 오후 4:10 UTCidentified

   The issue has been identified and a fix is being implemented.

3. 2025년 9월 12일 오후 4:12 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

4. 2025년 9월 12일 오후 4:19 UTCresolved

   This incident has been resolved.

1. 2025년 8월 28일 오후 6:10 UTCidentified

   The issue has been identified and a fix is being implemented.

2. 2025년 8월 28일 오후 6:18 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

3. 2025년 8월 28일 오후 6:22 UTCresolved

   This incident has been resolved.

1. 2025년 8월 22일 오전 3:58 UTCidentified

   The issue has been identified and a fix is being implemented.

2. 2025년 8월 22일 오전 4:32 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

3. 2025년 8월 22일 오전 4:36 UTCresolved

   This incident has been resolved.

1. 2025년 8월 21일 오후 6:18 UTCidentified

   Our login vendor is experiencing issues. Some dashboards will be effected for US users. Thank you for your patience while this is resolved.

2. 2025년 8월 21일 오후 6:34 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

3. 2025년 8월 21일 오후 7:05 UTCresolved

   This incident has been resolved.