Documo 장애 기록

We have identified an issue causing errors when a subset of user's attempt to send a fax using our email to fax feature. A fix is currently being implemented. Updates to come.

This incident has been resolved.

We have identified an issue causing errors when a subset of user's attempt to send a fax using our email to fax feature. A fix is currently being implemented. Updates to come.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

We are currently experiencing outbound fax delays due to an ongoing issue affecting Google Cloud global networking services. Our platform relies on these resources, and the disruption may result in increased outbound fax processing times. Our team is actively monitoring the situation and working with our infrastructure providers to minimize customer impact. We will continue to provide updates as more information becomes available.

We are seeing improvements in outbound fax processing and queue times as upstream Google Cloud services continue to recover. At this time, we have moved the incident to a monitoring state and will continue to closely observe system performance until full stability is confirmed. Please note that a backlog of outbound faxes accumulated during the disruption, and our systems will continue working through this queue as processing returns to normal.

This incident has been resolved.

We are currently investigating an issue causing disruptions with our API and fax services for a subset of customers. Our team is working diligently to resolve. Updates to follow.

The issue has been identified and a fix is being implemented. Updates to follow.

A fix has been implemented and we are monitoring the results.

We are continuing to monitor for further issues. Updates to follow.

This incident has been resolved.

We are currently investigating an issue causing disruptions with our API, portal, and fax services. Our team is working diligently to resolve. Updates to follow.

We are continuing to investigate this issue.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

We are currently investigating an issue causing a subset of users to receive large amounts of email suppression notifications. Updates to follow.

The issue has been identified and a fix is being implemented. Updates to follow.

This incident has been resolved.

We are currently investigating an issue causing UUID errors when attempting to open faxes within the web portal. Updates to follow.

A fix has been implemented and we are monitoring the results. Updates to follow.

This incident has been resolved.

We are currently investigating an issue causing all FaxBridge devices to appear offline, FMCs are operating as normal. This is effecting both inbound and outbound traffic. More updates to follow.

The issue has been identified and a fix is being implemented.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

We are investigating an issue causing inbound and outbound faxing delays. More updates to follow.

A fix has been implemented and we are monitoring the results. Updates to follow.

This incident has been resolved.

We are currently investigating an issue causing disruptions with our portal and fax services. Our team is working diligently to resolve. Updates to follow.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

We are currently investigating an issue causing disruptions with our portal and fax services. Our team is working diligently to resolve. Updates to follow.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

We are currently investigating an issue causing all FaxBridge devices to appear offline (FMCs are functioning as normal). This is effecting both inbound and outbound traffic. This is not impacting telecom and inbound faxes will print as expected once resolved. Our teams are actively engaged and working on a resolution. Updates to follow.

We are continuing to investigate this issue and will provide updates as we have them.

A fix has been implemented and we are monitoring results.

This incident has been resolved.

We are currently investigating an issue affecting outbound faxing. More updates to follow.

The issue has been identified and a fix is being implemented.

A fix has been implemented and we are monitoring the results. Please expect a delay in delivery as we are working through the backlogged faxes. Additional updates to follow.

This incident has been resolved and the backlog has been cleared.

We are currently investigating an issue where a subset of customers are experiencing outbound fax delays. More updates to follow.

We have identified the issue causing outbound fax delays and are currently working on a fix. More updates to follow.

A fix has been implemented and we are monitoring the results. We will be working through the backlog of faxes through the next few hours and will update here once the backlog is clear.

This incident has been resolved.

We are currently investigating an issue where a subset of FMC devices are experiencing delivery issues, including both inbound and outbound faxing. Updates to follow.

The issue has been identified and a fix is being implemented. Updates to follow.

We are continuing to work on a fix for this issue. More updates to follow.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

We are currently investigating an issue causing FMC devices to appear offline. This is affecting functionality on both inbound and outbound faxing. Updates to follow.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

We are currently investigating an issue that is causing FMC devices to be disconnected from our network. Updates to follow.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

We are currently investigating an issue causing a subset of users to receive an error when attempting to sign-in through office 365 that reads 'Token not issued for this application'. Updates to follow.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

We have identified an issue when dialing a subset of our inbound numbers resulting in faxes failing with a 'no fax tone detected' error. One of our telecom providers is experiencing intermittent failures which is the underlying cause. We will provide more updates as we have them.

We are continuing to work with our telecom vendor on a fix for this issue. More updates to follow as we have them.

A fix has been implemented and we have seen successful faxes being received from the affected numbers. We are continuing to monitor to ensure things are working as expected.

This incident has been resolved.

We are currently investigating an issue where customers are experiencing an 'internal server error' when attempting to view, print, and send outbound faxes. Updates to follow.

We are currently investigating an issue due to a global GCP outage. This is causing customers to receive an 'internal server error' when attempting to view, print, or send faxes. Our team is continuing to monitor for updates and will provide them as we have them.

We are aware of an outage that is affecting GCP across all regions globally and is affecting many other applications. For the Documo app, it is causing users to receive an 'internal server error' when attempting to view, print, or send faxes. Our team has been in contact with representatives from Google, and understand that their engineers have identified the root cause and have begun to apply the appropriate fixes. It will take some time (exact ETA unknown) for this to propagate across all regions. We will post updates here as soon as we know more.

As of 4:31 PM EDT, we are seeing evidence of recovery and restoration of the flow of faxes through our systems. Enqueued faxes may take some time to be completed, and the update of fax statuses in the portal may be somewhat delayed. Our contacts at Google have advised us that they are not yet 100% certain that the systems are fully recovered and stable. We will provide further updates when we have been provided the all-clear from Google engineers.

Our internal fax queues have been recovering well over the last hour and we look to be in a stable state. Our contacts at Google advise us that full recovery will be around 6 PM EDT. More updates to follow.

This incident has been resolved.