Mailprotector outage history

We are investigating reports of connection timeouts or rate limiting with some domains using Shield. This is not affecting all mail flow.

We are continuing to investigate this issue.

It appears that a reduction in the number of Microsoft servers communicating with Shield this morning contributed to the per-server volume spike. The team has mitigated the connection timeouts and rate limits, and the queues are draining. We do not have an exact time for when the queues will be emptied, but it appears it should be within the hour, most likely sooner. We are continuing to investigate with a priority of restoring stable and expected mail flow.

The queues are caught up, and we are observing normal mail flow at this time. The rate limiting experienced this morning was a response to abusive mail volume that was exacerbated by the reduction in Microsoft servers communicating with Shield. We will continue to monitor our queues through the business day to ensure processing is stable.

This incident has been resolved.

At approximately 1:00 PM ET, we were alerted to slow message processing in Shield. The symptoms were slow message updates and some delivery delays. The team has identified the bottleneck and resolved the issue. The queues are catching up, and performance appears to be normalizing. We will continue to monitor the queue for full resolution.

The queues have caught up, and performance is operating at expected levels.

Email log data for CloudFilter is not visible in the Console. The issue appears to have started approximately 9:00 AM ET this morning. The team is investigating the issue, and we will provide more information as soon as we can.

Email log data stopped shipping to the storage layer. The team has resolved the log shipping, and the logs are recovering. Log backfill will continue well into the afternoon, and we will update this notice once the logs have caught up.

Email logs for CloudFilter have caught up, and current logs are displaying as expected.

We are currently experiencing an issue with our Microsoft 365 tenant that is preventing us from receiving email replies to our support address. We are working to resolve this. If you need to reach us, please log into the support portal (support.mailprotector.com) and submit or update your request there. Portal submissions are working normally and will reach our team. We will update this notice once email replies are restored.

Support tickets and replies from Mailprotector will temporarily originate from a zendesk.com email address rather than our standard mailprotector.com domain. This is an intentional change we made to restore email reply functionality to our ticketing system. Emails arriving from zendesk.com on behalf of Mailprotector Support are legitimate. If your email filtering flags these messages as possible impersonation, they are safe to release and trust. We will update this notice when we return to our standard domain.

Support tickets and repllies are back to originating from mailprotector.com. No changes or actions need to be taken. Support emails will resume their normal behavior.

Microsoft is experiencing intermittent issues with mail flow. Please see their notice for reference. https://admin.cloud.microsoft/#/servicehealth/:/alerts/EX1331830 Mailprotector is monitoring the situation. However, mail flow with CloudFilter and Shield appears to be normal. Please check logs to identify bounces or delays from Microsoft hosts to confirm potential effects from their systems.

Microsoft Defender is restricting connectors due to false-positive reports. Please go to Microsoft Defender > Restricted Entities and unblock the listed connectors. (https://security.microsoft.com/restrictedentities) You will need to monitor and check on this restriction for the time being. We have observed that Microsoft restricts the connector after a short time, and the unblock request must be made again. We are continuing to monitor and investigate possible mitigation options.

Earlier today, Microsoft identified some of Shield's services as suspicious or compromised. This caused emails to bounce or be delayed. As of approximately 2:30 PM ET, the status began to lift, and we observed a return to expected mail flow. A support request has been opened with Microsoft to help us determine how and why this incident happened. We have not received a response yet. We are not closing this issue as resolved. Our intention is to get to the root of the incident. Our first priority was to find a way to resume normal mail flow, and it appears we have done so. We are continuing to monitor the situation, including the Microsoft notice regarding mail flow issues in Exchange Online, which appear to have been resolved.

We have begun reprocessing and redelivering emails that bounced during the affected time period. Here is what to expect: Most emails should be delivered, but not all of them. Some bounces during the incident may have been legitimate, and those messages will not be recoverable through reprocessing. Duplicates are possible. If a sender resent their email after the incident and the original message is also redelivered through our reprocessing, recipients may receive both. This is expected behavior and not a cause for concern. Reprocessing takes time. We expect this to be completed by midnight ET tonight, though it may finish sooner.

This incident has been resolved.

This morning, a third-party package repository used by our virus scanning infrastructure experienced an outage following an attack on its systems. Because our virus scanning instances rely on this repository at startup to pull the latest files and configurations, affected instances were unable to start. This reduced the capacity of our virus-scanning cluster below demand. When a virus scan cannot complete, our system quarantines the email as a precaution. As a result, emails processed during this window were quarantined rather than delivered to inboxes. No emails were lost. Affected emails can be released from quarantine by users or admins. Resolution The virus scanning cluster has been reconfigured to pull packages directly from the source repository, eliminating the dependency on the third-party intermediary. Monitoring and alerting thresholds have also been updated to surface this class of issue more quickly in the future. The issue is fully resolved.

We have been made aware that emails sent from Google Workspace accounts via CloudFilter/SafeSend are failing intermittently. There may also be delays in delivering inbound email due to an issue Google is experiencing. If you have a domain affected by this incident, we recommend temporarily disabling the outbound mail routes for Gmail. You may check the status of Google's incident at https://support.cloud.google.com/portal/system-status?product=WORKSPACE&cssp=true

This incident has been resolved.

An update to improve brand-related impersonation attacks was aggressively holding outgoing messages and sending inbound emails to the Junk E-mail folder. The update has been rolled back, and the held outgoing emails have been released. Emails delivered to the Junk E-mail folder must be moved out by users. The aggressive protection was active from approximately 2:00 PM ET to 3:00 PM ET.

Emails that were delivered to the Jail due to the aggressive behavior of the update have been released. The emails may still end up in the Junk E-mail folder if the risk level is medium or low from a new sender.

An update released at approximately 2:00 PM ET today was designed to improve protection against brand impersonation. It behaved more aggressively than intended, holding and jailing legitimate emails, particularly those with signatures containing social media links. The update was rolled back at approximately 2:50 PM ET. This issue is resolved. Actions taken to remediate after rollback: - Inbound emails delivered to Junk have been reprocessed to ensure appropriate risk and moved to the Inbox. - Released jailed emails; they may still land in Junk if the sender is new or carries a medium risk level. - Outgoing emails that were held have been released. No action needed.

Microsoft has announced a service issue affecting Exchange Online in M365. We are seeing email delivery delays to M365 domains with temporary error notifications. We will continue to monitor the Microsoft report. Mailprotector's CloudFilter and Shield services are operational.

Microsoft's status has not changed. For more information, please visit https://status.cloud.microsoft/

We will continue to monitor the status Microsoft provides at https://status.cloud.microsoft/. It appears Microsoft is making progress in some of their affected zones. Logs shows some emails are beginning to deliver, but without Microsoft's confirmation of a resolution, we expect deferrals to continue.

This incident has been resolved.

The team has identified an issue with email log ingestion and backend storage. A fix is being prepared. Logs will get backfilled once the issue is resolved. The log issue does not impact mail flow. Email delivery is performing as expected.

The log ingestion issue is resolved, and logs have been backfilled.

An issue with API notifications from Microsoft was discovered, and a fix is already being deployed. Mail flow has not been interrupted; however, the movement of emails to a folder based on sender preferences or risk level without mailbox rules was impacted. The fix is being deployed.

The fix has been deployed. New emails are being placed in the appropriate location as expected per X-ray log details. Emails that were not moved prior to the fix will remain in their current location. Moving the message will create an opportunity for Shield to act on the message, if needed.

An underlying issue at AWS is preventing an essential Radar task from completing. Radar will not complete the round-trip test and return results. We continue to monitor AWS's investigation into the failed service.

We are continuing to monitor the situation. AWS is continuing to investigate the issue.

Radar is operating as expected. However, AWS has not confirmed that the affected service has been fully resolved. We will continue to monitor the situation.

This incident has been resolved.

We are investigating errors connecting to CloudMail via webmail and email clients.

Connections to CloudMail are operating as expected. We will continue to monitor the status.

This incident has been resolved.

AWS is experiencing service issues that are impacting CloudFilter's quarantine. Emails destinated for the quarantine are queued. Emails passing filtering are being delivered to the destination recipients. Only quarantined emails are affected. You can follow the AWS updates at https://health.aws.amazon.com/health/status

AWS has reported progress but quarantine performance effects remain. We are continuing to monitor and work with AWS.

The Mailprotector Console is currently unavailable due to the AWS issues. Mail flow for good emails continues to operate as expected. To reiterate, good emails are being delivered to recipient servers. The Console and quarantine are currently affected.

The Mailprotector Console is back online. Quarantined emails continue to be queued and "good" emails are delivering to recipient servers.

Quarantine services are beginning to recover. We continue to monitor the progress reported by AWS and the recovery of queues and services affected by the outage.

Quarantine queues are continuing recovery and services appear to be operating as expected. We will monitor the AWS status through the morning.

We are currently investigating the issue. Email is being accepted in mailboxes; however, access via webmail and IMAP is interrupted.

A server service was unstable. The team has resolved the issue. Webmail and IMAP access is working as expected. We are continuing to monitor the service.

This incident has been resolved.

A permissions change caused the Mailprotector Console to lose connection to a status database. The result was a false report of mail flow errors in the Console and email notifications to admin accounts configured for the alerts. The permissions have been corrected, and the mail flow status in the Console for CloudFilter shows accurate information. There was no mail flow interruption.

We are investigating an inbound delivery delay. Delays of approximately 20 minutes have been observed.

Inbound delivery is performing as expected. We are continuing to monitor the flow.

Inbound delivery has continued to perform as expected. This incident is resolved.

A modification to handling ARC seal authentication was deployed late yesterday evening to handle an outlier situation observed over the past few months. The modification created a mail loop in specific messages that were missing some ARC seal authentication information in the headers. This specific situation caused the emails to look up the MX record for delivery, and those domains with a Shield MX record caused the email to loop. A fix for this situation was deployed shortly after the mail loop was detected. The fix now accounts for the rare circumstances involving the ARC seal and will gracefully handle mail delivery.

We are investigating an issue causing email delivery performance.

The issue has been isolated, and resources are being reallocated to empty the email delivery queue as quickly as possible.

The team has resolved email delivery performance, and inbound messages have been arriving as expected since about 1:00 PM ET. However, the issue is still monitored as API-related actions (sender training, message moves, etc.) and logs catch up. Please be aware that emails may sit in a folder a little longer than expected, and Spotlight logs may display an incorrect email location while the database catches up on activity. We expect the issue to be resolved entirely this evening.

Email delivery and API activity are operating as expected.

Mailprotector is seeing an increase in User Sync Errors affecting some Google Workspace customers due to a recent Google update. The error occurs when authenticating with Google. The error may be displayed as "Data too long for column". Resolution (required for each affected domain): 1. Delete the User Source under the User Sync tab in your console 2. Add a new User Source and reconnect to Google Workspace Please reach out to support with any additional questions or concerns.

We are continuing to monitor.

User sync errors affecting Google Workspace customers have been resolved. If an error still exists for a domain, please delete and recreate the user sync in the Mailprotector Console or open a request with the support team for guidance.