Files 장애 기록

We resolved an incident on the Files.com platform that caused SFTP logins using ed25519 SSH keys for authentication to fail. This incident impacted a small portion of users on our platform. Only SFTP, and only key-based logins using ed25519 keys were impacted by this incident. No other services or authentication types were effected. This incident occurred between from 17:13 UTC on 6/4/26 and 15:12 UTC 6/5/26, and was fully resolved by a code deploy. We will provide a detailed postmortem and RCA for this incident.

Between 17:13 UTC on June 4, 2026 and 15:12 UTC on June 5, 2026, [Files.com](http://Files.com) customers using SFTP with ed25519 SSH key-based authentication experienced authentication failures. This incident was limited to SFTP connections using ed25519 public keys. All other authentication methods — including password-based SFTP authentication, RSA key authentication, and all other [Files.com](http://Files.com) protocols and services — continued to operate normally throughout this period. We deployed a change to our SFTP server infrastructure intended to improve host key handling, but it inadvertently broke the authentication path for clients using ed25519 public keys. We failed to catch this regression before it reached production. We use automated testing to avoid this sort of regression, but there was a gap around specific combinations of ciphers and authentication flows. We have expanded our test suite to ensure every possible SFTP authentication path is tested. In addition, we did not manually verify ed25519 authentication during pre-deploy validation. This was a failure in our testing process and we are updating our guidelines around similar changes going forward. We also failed to detect this regression through our own monitoring. A latent bug in our internal compliance logging pipeline caused log events from the affected authentication sessions to be silently dropped rather than stored. Because these failed login attempts were not surfacing in our monitoring systems, we did not detect the incident internally—we learned of it from customer reports beginning on the morning of June 5, 2026. We have resolved the logging bug, and added alerting to this part of the logging pipeline to ensure future failures are surfaced immediately. We resolved the incident at 15:12 UTC on June 5, 2026 by rolling back the SFTP change and restarting all SFTP services across all regions. We confirmed the fix across affected customer accounts in all regions. As soon as we completely understood the situation, we posted an update to our status page. The root cause of this incident was [Files.com](http://Files.com)'s insufficient test coverage for all possible authentication paths, combined with a monitoring failure that prevented early detection. We have corrected both. Our customers trust us with their most sensitive and time-critical workflows, and we understand the disruption this caused. We are sorry. Our entire engineering team is committed to the improvements needed to prevent this type of incident from occurring again. If you need additional assistance or continue to experience issues, please contact our Customer Support team.

Between 13:00 and 13:39 UTC, a platform-wide issue occurred, resulting in all uploads, as well as create, update, and delete operations to any resource to fail. Logins and listings were unaffected. The issue has been resolved. We apologize for the inconvenience. A full postmortem will be published here once available.

We have resolved an SSL certificate configuration issue affecting a subset of servers in our USA region. This incident occurred between the times of 17:33 UTC and 19:14 UTC and resulted in elevated error rates during this time window. Most traffic did operate successfully during this time window. We are still in the process of determining the exact scope of impact.

From 17:32 UTC until 19:14 UTC on April 1, 2026, a subset of servers in our primary US region returned TLS errors that produced elevated failure rates across the [Files.com](http://Files.com) Web UI, API, FTP/S, SFTP, and WebDAV. Customers whose traffic was routed to unaffected hosts experienced no issues; others saw login timeouts or "invalid password" messages. **What Happened** While adding an additional domain to our primary wildcard SSL certificate, a bug in our internal certificate management software generated a certificate that omitted the wildcard entries for many of our domains. Our automated certificate rotation process then installed that faulty certificate on a portion of the fleet, causing those hosts to reject connections. Because traffic is load-balanced evenly, some customer sessions failed while others succeeded, making the issue difficult to detect on global dashboards. We identified the certificate issue and generated a corrected certificate within 10 minutes. However, the incorrect certificate remained deployed for an additional 92 minutes. This second delay is the most frustrating part of this incident. Recent modifications to our certificate rotation system had not been reflected in our internal documentation. As a result, our on-call engineers were working from outdated instructions for manually forcing a certificate rotation. It took additional time to uncover the correct process for performing the manual rotation. **What We Have Done To Mitigate This In The Future** We have updated our certificate generation to avoid the accidental omission of wildcard entries through additional validation. We have added an additional monitoring check that which continuously verifies deployed certificates on every public endpoint for the correct wildcard entries. We have documented the certificate-rotation mechanism and emergency override procedure in our internal documentation. We know our customers rely on [Files.com](http://Files.com) for mission-critical workflows, and any service interruption is unacceptable. We apologize for the disruption and appreciate your patience as we improve our safeguards to ensure this does not recur.

We have resolved an incident causing failures to load the Files.com Editor in our web interface. This incident occurred between the times of approximately 18:20 to 19:23 UTC on February 10th. Office documents can once again be previewed and edited in the built-in Files.com Editor. We apologize for the inconvenience that this incident caused.  We will perform a full postmortem on this incident and publish the report here when it is available.

From approximately 18:20 UTC to 19:23 UTC on 10 February 2026, customers could not preview or edit Office documents in the [Files.com](http://Files.com) Editor.  All other [Files.com](http://Files.com) services remained fully available.   **What happened** Our document-editing service relies on an internal message broker \(Amazon MQ / RabbitMQ\).  A routine broker maintenance reboot caused our editor software to stop accepting new editing sessions but continued to report itself as healthy.  Because our health check logic was flawed and yielding that incorrect status, our load balancer kept sending traffic to the failing servers, resulting in the spinning “Loading…” message you observed.   **What we found** * The editor’s own health endpoint did accurately represent its real status after the broker reboot but that was not known because the failure response was a HTTP 200 OK with the body of “false”. * Our Consul monitoring treated any HTTP 200 as healthy, so no automatic alert fired. * This same chain of events occurred on 20 January but was not fully remediated.   **What we are doing** 1. Updating the Consul health check to validate the response body and fail closed when the editor reports “false”. 2. Enabling CloudWatch logs and metrics for Amazon MQ and alerting on broker restarts and AMQP channel errors. 3. Adding an integration test that continuously opens and saves a document in production and pages engineering if it stalls. 4. We are working to replicate the exact error scenario in staging by creating a full production-like cluster. Once replicated, we will use that data to craft a solution to prevent this error from reoccurring.   We failed to detect the problem promptly and allowed it to recur.  We know you rely on the [Files.com](http://Files.com) Editor for time-critical collaborative work, and we are sorry for the disruption.  The actions above are already in progress, and we will publish further updates on our status page as milestones are reached.  Thank you for your continued trust in [Files.com](http://Files.com).

We are investigating an incident causing failures to load the Files.com Editor in our web interface. This incident only impacts the built-in Files.com Editor for previewing and editing Office documents. This incident does not impact the previewing of images, PDFs, or plaintext files, and this incident does not impact customers using the Microsoft Office editor. Customers who need to urgently view or edit Office documents can still download those documents locally. We will post updates to this incident as we have them.

We have resolved an incident causing failures to load the Files.com Editor in our web interface. This incident was resolved at 19:45 UTC on January 20th. Office documents can once again be previewed and edited in the built-in Files.com Editor. We apologize for the inconvenience that this incident caused. We will perform a full postmortem on this incident and publish the report here when it is available.

We have resolved an issue which caused upload and download failures for customers using both Custom Domains and Remote Server Mounts. From 16:14 to 16:28 UTC, Customers using both Custom Domains and Remote Server Mounts experienced failures in uploads and downloads to those folders on a Remote Server Mount. These failures manifested as Internal Server Error messages or similar messages on other protocols. Additionally from 21:45 to 22:55 UTC a smaller subset of customers using both Custom Domains and Remote Server Mounts experienced failures in uploads (but not downloads). This incident only affected remote server mounts, and only affected sites with Custom Domains. We understand that many customers rely on Remote Server Mounts for time-critical workflows, and even a brief interruption is unacceptable. We apologize for the disruption and appreciate your continued trust in Files.com. If you have any questions, please reach out to our Support team at any time.

From 21:45 UTC to 22:55 UTC on 9 January 2026, a subset of customers with custom domains enabled experienced elevated “500 – Internal Server Error” responses when uploading \(and in some cases downloading\) files through certain Remote Server Mount–backed transfer paths, including [Files.com](http://Files.com) Agent- backed connections. A code change in the [Files.com](http://Files.com) API control plane, intended to correct a bug in custom-domain uploads, introduced an unintended error condition affecting uploads routed through remote server mounts for sites with custom domains. Specifically, the API returned an invalid \(blank\) upload URL in its response, which caused subsequent upload steps to fail and resulted in HTTP 500 errors. These errors were immediately generated and logged by our application. Under normal conditions, this error volume would have triggered an automated PagerDuty alert to our on-call engineering team. However, a misconfiguration in our error-tracking system \(Sentry\) caused the majority of these error events to be suppressed. As a result, the expected alerts were not generated, and the issue was not immediately paged to on-call staff. Although our CI and production canary testing do exercise Agent-backed and Remote Server Mount upload workflows, those tests are not currently executed against customer sites using custom domains. Because this defect only manifested when custom domains were combined with remote server mounts, this edge case was not detected during pre-production testing. The issue was ultimately identified through customer reports, which prompted immediate investigation and remediation. A patch was verified in staging at 22:48 UTC and deployed to production at 22:55 UTC, fully restoring normal operation. We corrected the Sentry configuration and deployed new, version-controlled Terraform configuration to ensure that critical production errors always generate complete telemetry and alerts. We will also expand automated tests and deployment canaries to explicitly cover custom domain configurations so that this class of issue is detected during pre- production testing. This incident represents our most significant service disruption in the past year. While the issue affected a relatively small portion of our customer base, it had a material impact on those customers, with disruptions lasting more than an hour for some workflows. We are acutely aware that for the customers affected, the scope of the impact matters far more than the size of the cohort. We are extremely embarrassed by this failure. At the same time, we believe it is important to provide full transparency: even with this incident, [Files.com](http://Files.com) continued to meet and exceed its contractual SLA commitments for the month. That fact does not lessen the seriousness of the disruption, but it does highlight the overall resilience of the platform and our ongoing investment in reliability. We recognize that many customers depend on Remote Server Mounts for time- critical and business-critical workflows, and that any interruption is unacceptable. We sincerely apologize for the disruption this incident caused and for the impact to your operations. We take this failure seriously and are committed to maintaining the level of reliability and trust you expect from [Files.com](http://Files.com). If you have any questions or would like to discuss this incident further, our Support team stands ready to assist.

We have resolved an issue which caused elevated error rates in outbound connections from Files.com to Sharepoint and OneDrive remote servers. The impact was limited to these two remote server integrations and did not impact any inbound connections to Files.com that don’t involve Onedrive or SharePoint remote servers. This incident occurred between the times of 1:23PM and 2:58PM UTC.

We’ve received reports from customers that users accessing their site through Google Chrome, Mozilla Firefox, Safari, or other browsers that rely on Google’s Safe Browsing are erroneously seeing a "Dangerous Site" warning. This issue specifically affects the Files.com login page when it is accessed directly, at a URL formatted like domain.files.com/login. This usually only occurs when the login page is bookmarked. Users can navigate directly to their domain (for example, by accessing domain.files.com instead of domain.files.com/login) without receiving this warning. We believe that this may have been caused by one of our customers falsely reporting to Google that their Files.com site was unsanctioned. We are attempting to work with Google to have them review the report. There is no security concern with the Files.com application, and you should disregard this error. Please help us by submitting a report to Google that this page is safe: https://safebrowsing.google.com/safebrowsing/report_error/?url=https:%2F%2Ffiles.com%2Flogin&hl=en-US When writing your report, please fill in your custom subdomain in the URL box. We will provide additional details as they become available. Customers with urgent questions are encouraged to contact our Customer Support team by email. Thank you for your patience.

This issue specifically affects the Files.com login page when it is accessed directly, at a URL formatted like domain.files.com/login. This usually only occurs when the login page is bookmarked. Users can navigate directly to their domain (for example, by accessing domain.files.com instead of domain.files.com/login) without receiving this warning. We will update this page again only when new information becomes available.

We are continuing to investigate this issue.

Google has reviewed our appeal of this issue and acknowledged that this was a false positive. There was never any danger to any users connecting to any Files.com site. This issue is now resolved. Starting this morning, we began to receive reports from customers that users accessing Files.com through Google Chrome, Mozilla Firefox, Safari, or other browsers that rely on Google’s Safe Browsing were erroneously seeing a "Dangerous Site" warning. Google has removed Files.com from its list of "Dangerous Sites" and our testing indicates that browsers are no longer displaying this erroneous warning. If you or your users are still seeing a warning when navigating to your Files.com site, you likely need to wait a little longer for your browsers to receive an update from Google.

From 10:03 UTC to 19:02 UTC on August 4, 2025, some users attempting to sign in to [Files.com](http://Files.com) via URLs ending in /login \(e.g., [acme.files.com/login](http://acme.files.com/login)\) encountered a red warning stating “Deceptive site ahead.” This warning appeared in Chrome, Safari, Firefox, and other browsers that rely on Google Safe Browsing. Affected access paths included: * Bookmarked links to /login * Direct navigation to /login * The "Sign In" button on our marketing website All other product surfaces—including APIs, desktop and CLI apps, agents, mobile apps, custom domains, and already-loaded web app sessions—remained fully functional and unaffected. **Root Cause** The Chrome, Safari, and Firefox browsers all use a service from Google called Google Safe Browsing to identify potentially dangerous sites and apply a full page warning when a user tries to visit any site that has been flagged by Google Safe Browsing. The [Files.com](http://Files.com) subdomain for a [Files.com](http://Files.com) customer was incorrectly reported to Google Safe Browsing as a phishing site by a third-party security vendor affiliated with that customer. Google accepted the report without independent validation and—unexpectedly— applied a warning to any login page on the entire [files.com](http://files.com) domain, not just that specific customer subdomain. To be clear: * No malware was present. * No compromise occurred. * No policy violation took place on the [Files.com](http://Files.com) platform. We have identified both the customer and the third-party vendor responsible for the report and have taken action to prevent any recurrence. **Response Timeline** At 10:09 UTC, our team convened an incident bridge within six minutes of detection and posted a status page update. Shortly after: * We disabled the affected customer’s subdomain. * We submitted a formal appeal to Google via Search Console, providing logs and proof that the report was a false positive. * We pushed a change to reroute new login sessions from /login to /newsession, restoring access for all customers except those with bookmarks to the previous URL. * We requested “This site is safe” confirmations from affected customers to help expedite Google’s manual review. At 19:02 UTC Google completed its review and removed the incorrect classification. Browser warnings began disappearing shortly thereafter as Safe Browsing lists refreshed \(typically within 30–60 minutes\). **Our Perspective** We understand the critical nature of [Files.com](http://Files.com) in your workflows, and we consider this incident wholly unacceptable—not because of any fault in our systems, but because access to our login page was unfairly blocked, creating real-world disruption for our customers and reputational damage for both you and us. We are deeply frustrated by how this situation was handled by Google: * An erroneous third-party report—submitted via a poorly governed “security” company system—was accepted by Google without any review. * Worse, the impact of that report was applied far beyond its scope, extending the warning to our entire domain, rather than the single customer subdomain in question. * Despite escalating through multiple internal and public channels, it took Google over 9 hours to act, during which time countless users encountered misleading and damaging warnings when trying to access [Files.com](http://Files.com). This incident reveals a deeply troubling reality: Google Safe Browsing wields extraordinary power over the web, but operates with little transparency, no meaningful appeals process, and effectively zero accountability. There’s no SLA, no phone number, and no obligation to fix their own errors promptly—if at all. We believe this system is broken. A single bad report—no matter how inaccurate— can cause cascading damage to legitimate businesses without recourse. That’s not just frustrating. It’s dangerous. We’re currently working to establish a more reliable escalation path within Google Safe Browsing, but the truth is: no business should have to navigate this kind of opaque and unilateral system to protect its integrity. We sincerely apologize for the disruption and appreciate your patience and trust. If you have any further questions or concerns, please reach out to our Support team.

We are aware of an issue preventing certain logins to the FTP services in all regions. This incident only impacts unencrypted FTP connections to the Files.com platform. Encrypted FTPS connections are not affected at all by this incident. Unencrypted FTP is not supported by default on the Files.com platform. This incident only affects customers who have enabled unencrypted FTP on their sites. We are currently working to resolve this incident and we will post an update here when more information is available.

We have resolved an issue that prevented certain unencrypted logins to FTP services in all regions. This incident only impacted a portion of unencrypted FTP connections to the Files.com platform. Unencrypted FTP is not supported by default on the Files.com platform, so this incident caused issues for only a small number of customers. Encrypted FTPS connections were not affected at all by this incident. The incident affected different users at different times between 16:27 UTC and 17:39 UTC.

From 16:27 UTC to 17:39 UTC on 24 July 2025, a subset of customers who rely on unencrypted \(plain\) FTP experienced login failures via FTP. Encrypted FTPS/SFTP services, API calls, automations, and all other platform functions operated normally. At 16:27 UTC we released a security enhancement to our FTP services related to our sunset of legacy 3DES ciphers \(which were previously only used by a very small number of customers who had explicitly opted in to this\). The new validation contained a logic error which mistakenly applied to plain FTP sessions as well. Because plain FTP contains no relevant cipher, every attempt was rejected with an “insecure cipher” error. Users who had existing cached sessions continued to operate, which masked the problem for roughly 20 minutes, delaying detection by our automated monitors. Once the cache expired, customers relying on unencrypted FTP began to fail authentication consistently. We identified the faulty code, fixed the logic, redeployed the service, and cleared all caches. The last observed customer error occurred at 17:39 UTC. Plain FTP is disabled by default on the [Files.com](http://Files.com) platform; only customers who had explicitly enabled it were at risk. All encrypted protocols \(FTPS, SFTP, HTTPS, AS2, WebDAV, API\) remained fully operational. We know customers depend on [Files.com](http://Files.com) for critical workflows and we failed to uphold that trust for those affected. We apologize for the disruption and appreciate your continued confidence in our service.

We have resolved elevated error rates in outbound connections from Files.com to remote servers. The impact was limited to remote server integrations and did not impact inbound connections to Files.com that don’t involve remote servers. This incident occurred between the times of 21:39 UTC to 21:56 UTC.

From 21:39 UTC to 21:56 UTC on 19 July 2025, [Files.com](http://Files.com) customers experienced elevated error rates on certain Remote Server Integrations \(Sync and Mount\). All other platform services, including the [Files.com](http://Files.com) Agent, continued to operate normally. At 21:39 UTC, we released an update intended to simplify how our remote server integration engine chooses a backend server through which to route the connection. However, the update contained a defect that caused certain jobs to fail to select a backend server at all. At 21:56 UTC, we rolled back the change, immediately restoring normal performance. Full validation confirmed no residual impact. We understand that many customers rely on Remote Server Integrations for time-critical workflows, and even a brief interruption is unacceptable. We apologize for the disruption and appreciate your continued trust in [Files.com](http://Files.com). If you have any questions, please reach out to our Support team at any time.

We have resolved an intermittent issue that caused the Upload Files button in the web interface to become unresponsive. This issue was limited to the button functionality only—file uploads still worked correctly via drag-and-drop in the web interface, as well as through all other supported protocols. This incident occurred between 22:55 UTC on July 7 and 10:48 UTC on July 8. Thank you for your patience while we investigated and addressed the problem.

From 22:55 UTC on 7 July 2025 until 12:48 UTC on 8 July 2025 the “Upload Files” and “Upload Folder” buttons in the [Files.com](http://Files.com) web interface did not respond when clicked. Drag-and-drop uploads, API uploads, and all other protocols continued to operate normally. We deployed an update to our web interface intended to clean up an internal warning message. Unfortunately, that update removed the event handler that activates the file-picker dialog, rendering the Upload button inert. Because our automated tests and monitors did not exercise that specific UI path, the regression was not detected before release or by our monitoring systems. After customers reported the problem, we immediately rolled back the change, restoring normal operation. We apologize for the inconvenience and frustration this incident caused. Our customers rely on [Files.com](http://Files.com) for mission-critical workflows, and we did not meet that standard during this window. We are committed to learning from this event and preventing similar issues in the future.

FTP and SFTP only: We are investigating elevated error rates on the FTP and SFTP services on Files.com in all regions. This incident does not impact other network services such as API, WebDAV, AS2, and others.

We have resolved an incident which caused elevated error rates on the SFTP and FTP services on Files.com in all regions. This incident began at 19:27 UTC, and was resolved at 19:49 UTC. This incident did not impact other network services such as API, WebDAV, AS2, and others. We will post an incident report, including a full postmortem, as soon as one is available.

FTP and SFTP only: We are investigating reports of intermittent degraded performance on FTP and SFTP services on Files.com in our primary USA region. This incident appears to only impact a small number of customers. This incident does not impact other network services such as API, WebDAV, AS2, and others. If you are experiencing degraded performance on FTP or SFTP transfers and you have an urgent need to access Files.com, you should be able to immediately connect (and access your existing files and account) using the hostname of our Canada region, which is app-ca-central-1.files.com. We will provide additional details as they become available. Customers with urgent questions are encouraged to contact our Customer Support team by email. Thank you for your patience.

We are continuing to investigate this issue.

We have resolved an incident causing degraded performance on FTP and SFTP transfers in our USA region, and transfer performance is now back to normal. The incident began to affect customer transfer performance as early as 6am Pacific Time on January 21st, and was resolved at 9:53 AM Pacific Time on January 24th. This issue was intermittent and affected a small set of customers. We are still investigating the root cause of this incident. When we have more information to share we will post a detailed postmortem here.

London only: We are investigating a partial network outage on Files.com affecting Files.com services in the London region. This outage is only affecting our network gateway networking in the London region, and our Core Services are running correctly. At this time, we believe that all network services are currently up in our other regional locations. If you have an urgent need to access Files.com via FTP, SFTP, or WebDAV, you should be able to immediately connect (and access your existing files and account) using the hostname of our USA region, which is app-us-east-1.files.com. We will provide additional details as they become available. Customers with urgent questions are encouraged to contact our Customer Support team by email. Thank you for your patience.

All services have been restored and are operating normally. London only: We have resolved a performance degradation on Files.com affecting Files.com services in the London region. This incident occurred between the times of 6:07 a.m. PST to 7:07 a.m. PST, but only in the London region. If you previously moved any workloads to another region in response to this incident, you are cleared to move those regional workloads back to the London region. We will follow up with an Incident Report within ten (10) business days including the root cause and steps taken to address the root cause. If you need additional support, please do not hesitate to contact our Customer Support team by email or phone. Thanks for your support while we resolved this issue.

We have resolved an incident causing elevated error rates on the FTP service on Files.com in the USA and Germany regions. From 3:05pm to 3:15pm Pacific Time, some FTP uploads and downloads failed to transfer. This incident only impacted a portion of new connections that were initiated between 3:05pm and 3:15pm. FTP connections established before the start of the incident were not impacted. This incident only impacted FTP, and none of our other interfaces (API, Web, SFTP, AS2, etc). This incident only impacted the USA and Germany regions. The incident is fully resolved. We will post a full RCA to this Status Page when it is available. If you have any questions, please contact our Support department at support@files.com or call (800) 286-8372 x 2

Singapore FTP only: We are investigating a major outage of the FTP service on Files.com in the Singapore region. This incident does not impact other network services such as API, SFTP, WebDAV, AS2, and others, nor does it impact regions other than Singapore. At this time, we believe that all network services are currently up in our other regional locations. If you have an urgent need to access Files.com, we recommend using SFTP in lieu of FTP. If you must connect via FTP, you should be able to immediately connect (and access your existing files and account) using the hostname of our USA region, which is app-us-east-1.files.com. We will provide additional details as they become available. Customers with urgent questions are encouraged to contact our Customer Support team by email. Thank you for your patience.

All services have been restored and are operating normally. We have resolved a major outage of the FTP service on Files.com in the Singapore region. This incident did not impact other network services such as API, SFTP, WebDAV, AS2, and others. The FTP service was down from 10:59 PM PST on November 24th to 12:46 AM PST November 25th, with a total downtime of 1 hour 47 minutes, but only in the Singapore region. If you previously moved any workloads to another region in response to this incident, you are cleared to move those regional workloads back to the Singapore region. We will follow up with an Incident Report within ten (10) business days including the root cause and steps taken to address the root cause. If you need additional support, please do not hesitate to contact our Customer Support team by email or phone. Thanks for your support while we resolved this issue.

We have resolved elevated error rates on the FTP service on Files.com in all regions. This incident did not impact other network services such as API, SFTP, WebDAV, AS2, and others. This incident occurred between the times of 8:49am PST and 10:30am PST. We are compiling a Root Cause Analysis for this incident, which we will post here.

On November 19 from 8:49am to 10:30am PST we experienced an issue with our FTP services that resulted in elevated error rates and inability to connect entirely to FTP for some, but not all, customers. This issue did not affect any other network services at [Files.com](http://Files.com), such as SFTP, WebDAV, AS2, API, etc. It was specific to FTP. This issue affected approximately 12.5% \(1/8th\) of all source IP addresses connecting to [Files.com](http://Files.com) FTP. To explain how this issue affected only 1/8th of source IP addresses, and to give a sense of the scale we operate at regarding FTP, some background on [Files.com](http://Files.com)’s FTP services is helpful. [Files.com](http://Files.com) serves over 4,000 customers across a variety of file transfer protocols and paradigms from a globally distributed multi-tenant architecture. Each file transfer service is implemented and deployed individually so that they can be updated, monitored, and restarted on an individual basis without affecting other network services. For example, our FTP service is not co-located in any way with our SFTP service. For FTP specifically, we operate a total of 16 server machines in 7 global regions, and on each server machine we operate two different FTP daemons in a Blue/Green configuration. This allows us to deploy software changes to FTP without disrupting existing connections. And we do in fact regularly deploy software changes to FTP, most of which go totally unnoticed to our customers. In front of those 32 FTP servers \(16 machines x 2 instances per machine\), we also operate 20 front-end proxy servers which serve as the termination point for the nearly 2,000 dedicated IP addresses that we host on behalf of our customers. **Background** This incident affected only certain combinations of connectivity between the front- end proxy servers and the back-end FTP servers, and that’s why it only affected 12.5% of source IP addresses. To explain this more, first you need to understand a little bit about the FTP protocol. The FTP protocol, originally developed in 1971, is an old and legacy protocol. It was designed way before the idea of load balancing even existed, and therefore its design is such that it is difficult to load balance in a reliable way. This is one of the main reasons we generally recommend against the use of FTP entirely. Instead, we recommend that you prefer modern connectivity methods such as our CLI, SDKs, API, [Files.com](http://Files.com) apps, and direct integrations with cloud providers such as Boomi, MuleSoft, Zapier, and more. Nevertheless, we understand that a lot of customers have legacy business processes that are built on legacy technologies such as FTP, and therefore, we do our best to support it. The specific reason that FTP is hard to load balance is that FTP does not use a single encrypted TCP connection between the client and server. Instead, FTP uniquely creates a control connection, which is one TCP connection between the client and server, and then it also creates individual data connections for each file being transferred. Incidentally, this design also hurts the performance of FTP because it results in separate TLS and TCP connection initiations for every single file in a batch of transfers. More modern protocols like the ones I mentioned before are able to multiplex multiple transfers over the same TCP connection. That FTP maintains separate data connections is what also creates a major challenge for load balancing. That's because all of those data connections need to end up at the same backend FTP server, which means there needs to be a load balancing strategy that ensures that these data connections all end up at the same backend server. Most load balancers don’t do this by default, because it isn’t what you’d usually want in a load balancing arrangement. We implement FTP load balancing using a capability called balance\_source in the popular HAProxy software. It computes a hash of your source IP address and uses that hash to target FTP requests to specific backend servers that are maintained in a list of backends. We have a sophisticated health check apparatus that maintains that list of backend servers using health metrics about each backend server. This ensures that our load balancers generally only send traffic to FTP servers which are healthy. **Explanation of this Incident** In this incident, one eighth of our backend FTP servers were marked as healthy when, in fact, they were not healthy. This caused about 1/8 of FTP traffic to be directed to those servers. Unlike on other protocols where this would have applied to 1/8th of all requests from all customers, due to the load balancing design of FTP this ended up applying to all FTP connections from 1/8th of source IP addresses. How did the processes mistakenly report as healthy when they were not? This was a software bug in our health check software, related to an invalid assumption about which FTP server was running in the “most recent” deployment position of the Blue/Green deployment. When we wrote the software, we made the mistaken assumption in the code that in Linux, server processes with higher process ID numbers would always be newer than processes with lower process ID numbers. In fact, that's not true. A Linux system only has about 65,000 process ID numbers available, and in a long-running system such as our servers, which often stay up for months at a time, the process IDs can wrap back around. So our health check software misinterpreted which of the two FTP services on each server was the newest, and this occurred on 4 out of our 32 FTP servers. We corrected this problem in our health check software and we do not expect this issue to recur. **Monitoring and Incident Resolution Time** This incident occurred for one hour and 41 minutes. The selective impact of this issue unfortunately also caused it to escape our monitoring systems. Because our monitoring systems experienced 100% success during this time, we did not detect that one out of eight source IP addresses to FTP were unsuccessful at connecting. It is not clear how we could have better detected this situation through monitoring. We ultimately became aware of this issue through our regular customer support channel. Unfortunately, it took our customer support team over 30 minutes to confirm and escalate this incident after first being alerted of it, and this increased the time to resolution of the issue. **Similarity to the Incident from November 13** Although this incident seems similar to the incident on November 13, which also affected FTP, this incident has a different root cause. Please read that incident’s Root Cause Analysis document to learn more. **Stability and Uptime, Generally** Any time we have two separate incidents affecting the same service within a short period of time, we are often asked by customers whether there's some sort of systemic problem with our business or our infrastructure or our software or our processes that should be considered unstable. One of the reasons that we write such long root cause analysis reports is to hopefully convince you that these two incidents were not caused by the same root cause. Additionally, we hope that these reports also help to convince you that neither root cause has at its root a culture or systemic problem at [Files.com](http://Files.com). When comparing [Files.com](http://Files.com) to other systems, it’s important to compare apples to apples. [Files.com](http://Files.com) is a continuously updated multi-tenant cloud service that does not ever schedule downtime. We are designed for 24/7/365 operation all while performing regular performance, feature, and security updates. It is not a fair comparison to compare the uptime of [Files.com](http://Files.com) to the uptime of an on-premise system where updates are never installed. We do our own independent monitoring of many of the cloud providers for file transfer and according to our statistics, [Files.com](http://Files.com) exceeds these other providers in many of the metrics that we measure as it relates to uptime and speed and performance. We greatly appreciate your patience and understanding as we resolved this issue. If you need additional assistance or continue to experience issues, please contact our Customer Support team.

FTP only: We are investigating elevated error rates on the FTP service on Files.com in our primary USA region. This incident does not impact other network services such as API, SFTP, WebDAV, AS2, and others. This incident does not impact FTP in any other regions. If you are affected by this incident and have an urgent need to access Files.com, we recommend using SFTP in lieu of FTP. If you must connect via FTP, you should be able to immediately connect (and access your existing files and account) using the hostname of our Canada region, which is app-ca-central-1.files.com.

We have resolved elevated error rates on the FTP service on Files.com in our primary USA region. This incident did not impact other network services such as API, SFTP, WebDAV, AS2, and others. This incident occurred between the times of 7:25am PST and 8:12am PST. If you previously moved any workloads to another region in response to this incident, you are cleared to move those regional workloads back to the USA region. We are compiling a Root Cause Analysis for this incident, which we will post here.

On November 13 from 7:25am to 8:10am PST we experienced an issue with our FTP services that resulted in elevated error rates and inability to connect entirely to FTP for some, but not all, customers. This issue did not affect any other network services at [Files.com](http://Files.com), such as SFTP, WebDAV, AS2, API, etc. It was specific to FTP in our USA region. According to our logs, this issue affected approximately half of all traffic connecting to FTP in our USA region during the impacted time window. All other regions were unaffected. This incident related to a deployment of a critical software update to our production proxy servers in the USA region. We have 20 worldwide proxy servers, and they are the toughest devices in our fleet to update. These servers often need to be updated in place because they route traffic for all of our network services. Of those 20, 8 are located in our USA region. In this incident, 4 of the 8 proxy servers failed to restart the FTP proxy service after the critical software update. Because we experienced more than one proxy server failure at once, the incident caused total failure of FTP to some customers. The impact of this incident was mainly caused by a failure in our deployment process to detect the failure after the first proxy server failed, leading to a larger failure. We have revised our deployment process to reduce the chance of this sort of cascade in the future. We greatly appreciate your patience and understanding as we resolved this issue. If you need additional assistance or continue to experience issues, please contact our Customer Support team.

Files.com has resolved an incident that caused elevated errors on the web interface. No other services were impacted by this incident. Certain pages in the web interface failed to load between 11:37am and 1:24pm Pacific time. We are still investigating this incident and compiling a Root Cause Analysis, which we will post here.

From 11:37 AM PST to 1:24PM PST, a small number of [Files.com](http://Files.com) customers experienced errors in the web interface when viewing some pages.  This was caused by a JavaScript bug introduced during a routine update, related to rendering pages with “tab-like” navigation.  Because the buggy behavior impacted only a small number of pages, it was not detected during our routine testing. Our monitoring systems did record an increase in frontend error rates, however the increase in errors was not high enough to raise an alert and instead we were alerted to this bug through our customer support channel. As a result of this incident, we have reduced the alerting threshold for similar front end errors. We expect that we will now detect and resolve similar incidents in the future.

Files.com is aware of elevated error rates on the FTP, SFTP, and WebDAV services on Files.com in all regions from 10:29am to 10:36am Pacific Time. We believe this incident impacted less than half of total traffic across these services during this time period. This incident did not impact any other network services such as API, AS2, HTTP, or others. We are compiling a Root Cause Analysis for this incident, which we will post here as soon as it is ready.

On October 29th from 10:29 am PST until 10:36am PST, [Files.com](http://Files.com) customers experienced considerably higher error rates for FTP, SFTP, and WebDAV requests. No other [Files.com](http://Files.com) services, including the [Files.com](http://Files.com) API, were impacted during this window. This incident began when we deployed a new version of the software package we use internally that contains our FTP, SFTP, and WebDAV software. The root cause of this incident was that the internal tool that we use for deploying software had an edge case bug that resulted in deployment failure if a large number of deployments were performed during the same day. We quickly identified the issue and deployed a fix. We have added additional testing around this scenario and do not expect this issue to recur. We greatly appreciate your patience and understanding as we resolved this issue. If you need additional assistance or continue to experience issues, please contact our Customer Support team.