51 Mailprotector incidents · 2023年3月 — official updates, affected components, duration and resolution details.
午後1時頃に、シールドでメッセージ処理を遅くするという警告がありました。 症状は、メッセージの更新を遅くし、配送遅延が生じる。 チームはボトルネックを識別し、問題を解決しました。 キューがキャッチアップされ、パフォーマンスが正規化されるように見えます。 引き続き、全解像度のキューを監視します.
キューを巻き上げ、性能が期待されるレベルで動作します.
公式のインシデント更新を自動翻訳しています。
CloudFilter のメールログデータはコンソールで表示されません。 朝9時頃からスタートしたのが問題です。 チームは課題を調査し、出来る限り迅速に情報を提供します.
メールログデータは保存層に出荷を停止します。 チームはログの配送を解決し、ログは回復しています。 ログのバックフィルは午後にうまくいきます。また、ログが起きたらこの通知を更新します.
CloudFilter のメールログが立ち上がり、現在のログが期待通り表示されます.
公式のインシデント更新を自動翻訳しています。
We are currently experiencing an issue with our Microsoft 365 tenant that is preventing us from receiving email replies to our support address. We are working to resolve this. If you need to reach us, please log into the support portal (support.mailprotector.com) and submit or update your request there. Portal submissions are working normally and will reach our team. We will update this notice once email replies are restored.
Support tickets and replies from Mailprotector will temporarily originate from a zendesk.com email address rather than our standard mailprotector.com domain. This is an intentional change we made to restore email reply functionality to our ticketing system. Emails arriving from zendesk.com on behalf of Mailprotector Support are legitimate. If your email filtering flags these messages as possible impersonation, they are safe to release and trust. We will update this notice when we return to our standard domain.
Support tickets and repllies are back to originating from mailprotector.com. No changes or actions need to be taken. Support emails will resume their normal behavior.
Microsoft is experiencing intermittent issues with mail flow. Please see their notice for reference. https://admin.cloud.microsoft/#/servicehealth/:/alerts/EX1331830 Mailprotector is monitoring the situation. However, mail flow with CloudFilter and Shield appears to be normal. Please check logs to identify bounces or delays from Microsoft hosts to confirm potential effects from their systems.
Microsoft Defender is restricting connectors due to false-positive reports. Please go to Microsoft Defender > Restricted Entities and unblock the listed connectors. (https://security.microsoft.com/restrictedentities) You will need to monitor and check on this restriction for the time being. We have observed that Microsoft restricts the connector after a short time, and the unblock request must be made again. We are continuing to monitor and investigate possible mitigation options.
Earlier today, Microsoft identified some of Shield's services as suspicious or compromised. This caused emails to bounce or be delayed. As of approximately 2:30 PM ET, the status began to lift, and we observed a return to expected mail flow. A support request has been opened with Microsoft to help us determine how and why this incident happened. We have not received a response yet. We are not closing this issue as resolved. Our intention is to get to the root of the incident. Our first priority was to find a way to resume normal mail flow, and it appears we have done so. We are continuing to monitor the situation, including the Microsoft notice regarding mail flow issues in Exchange Online, which appear to have been resolved.
We have begun reprocessing and redelivering emails that bounced during the affected time period. Here is what to expect: Most emails should be delivered, but not all of them. Some bounces during the incident may have been legitimate, and those messages will not be recoverable through reprocessing. Duplicates are possible. If a sender resent their email after the incident and the original message is also redelivered through our reprocessing, recipients may receive both. This is expected behavior and not a cause for concern. Reprocessing takes time. We expect this to be completed by midnight ET tonight, though it may finish sooner.
This incident has been resolved.
This morning, a third-party package repository used by our virus scanning infrastructure experienced an outage following an attack on its systems. Because our virus scanning instances rely on this repository at startup to pull the latest files and configurations, affected instances were unable to start. This reduced the capacity of our virus-scanning cluster below demand. When a virus scan cannot complete, our system quarantines the email as a precaution. As a result, emails processed during this window were quarantined rather than delivered to inboxes. No emails were lost. Affected emails can be released from quarantine by users or admins. Resolution The virus scanning cluster has been reconfigured to pull packages directly from the source repository, eliminating the dependency on the third-party intermediary. Monitoring and alerting thresholds have also been updated to surface this class of issue more quickly in the future. The issue is fully resolved.
We have been made aware that emails sent from Google Workspace accounts via CloudFilter/SafeSend are failing intermittently. There may also be delays in delivering inbound email due to an issue Google is experiencing. If you have a domain affected by this incident, we recommend temporarily disabling the outbound mail routes for Gmail. You may check the status of Google's incident at https://support.cloud.google.com/portal/system-status?product=WORKSPACE&cssp=true
This incident has been resolved.
An update to improve brand-related impersonation attacks was aggressively holding outgoing messages and sending inbound emails to the Junk E-mail folder. The update has been rolled back, and the held outgoing emails have been released. Emails delivered to the Junk E-mail folder must be moved out by users. The aggressive protection was active from approximately 2:00 PM ET to 3:00 PM ET.
Emails that were delivered to the Jail due to the aggressive behavior of the update have been released. The emails may still end up in the Junk E-mail folder if the risk level is medium or low from a new sender.
An update released at approximately 2:00 PM ET today was designed to improve protection against brand impersonation. It behaved more aggressively than intended, holding and jailing legitimate emails, particularly those with signatures containing social media links. The update was rolled back at approximately 2:50 PM ET. This issue is resolved. Actions taken to remediate after rollback: - Inbound emails delivered to Junk have been reprocessed to ensure appropriate risk and moved to the Inbox. - Released jailed emails; they may still land in Junk if the sender is new or carries a medium risk level. - Outgoing emails that were held have been released. No action needed.
Microsoft has announced a service issue affecting Exchange Online in M365. We are seeing email delivery delays to M365 domains with temporary error notifications. We will continue to monitor the Microsoft report. Mailprotector's CloudFilter and Shield services are operational.
Microsoft's status has not changed. For more information, please visit https://status.cloud.microsoft/
We will continue to monitor the status Microsoft provides at https://status.cloud.microsoft/. It appears Microsoft is making progress in some of their affected zones. Logs shows some emails are beginning to deliver, but without Microsoft's confirmation of a resolution, we expect deferrals to continue.
This incident has been resolved.
The team has identified an issue with email log ingestion and backend storage. A fix is being prepared. Logs will get backfilled once the issue is resolved. The log issue does not impact mail flow. Email delivery is performing as expected.
The log ingestion issue is resolved, and logs have been backfilled.
An issue with API notifications from Microsoft was discovered, and a fix is already being deployed. Mail flow has not been interrupted; however, the movement of emails to a folder based on sender preferences or risk level without mailbox rules was impacted. The fix is being deployed.
The fix has been deployed. New emails are being placed in the appropriate location as expected per X-ray log details. Emails that were not moved prior to the fix will remain in their current location. Moving the message will create an opportunity for Shield to act on the message, if needed.
An underlying issue at AWS is preventing an essential Radar task from completing. Radar will not complete the round-trip test and return results. We continue to monitor AWS's investigation into the failed service.
We are continuing to monitor the situation. AWS is continuing to investigate the issue.
Radar is operating as expected. However, AWS has not confirmed that the affected service has been fully resolved. We will continue to monitor the situation.
This incident has been resolved.
We are investigating errors connecting to CloudMail via webmail and email clients.
Connections to CloudMail are operating as expected. We will continue to monitor the status.
This incident has been resolved.
AWS is experiencing service issues that are impacting CloudFilter's quarantine. Emails destinated for the quarantine are queued. Emails passing filtering are being delivered to the destination recipients. Only quarantined emails are affected. You can follow the AWS updates at https://health.aws.amazon.com/health/status
AWS has reported progress but quarantine performance effects remain. We are continuing to monitor and work with AWS.
The Mailprotector Console is currently unavailable due to the AWS issues. Mail flow for good emails continues to operate as expected. To reiterate, good emails are being delivered to recipient servers. The Console and quarantine are currently affected.
The Mailprotector Console is back online. Quarantined emails continue to be queued and "good" emails are delivering to recipient servers.
Quarantine services are beginning to recover. We continue to monitor the progress reported by AWS and the recovery of queues and services affected by the outage.
Quarantine queues are continuing recovery and services appear to be operating as expected. We will monitor the AWS status through the morning.
We are currently investigating the issue. Email is being accepted in mailboxes; however, access via webmail and IMAP is interrupted.
A server service was unstable. The team has resolved the issue. Webmail and IMAP access is working as expected. We are continuing to monitor the service.
This incident has been resolved.
A permissions change caused the Mailprotector Console to lose connection to a status database. The result was a false report of mail flow errors in the Console and email notifications to admin accounts configured for the alerts. The permissions have been corrected, and the mail flow status in the Console for CloudFilter shows accurate information. There was no mail flow interruption.
We are investigating an inbound delivery delay. Delays of approximately 20 minutes have been observed.
Inbound delivery is performing as expected. We are continuing to monitor the flow.
Inbound delivery has continued to perform as expected. This incident is resolved.
A modification to handling ARC seal authentication was deployed late yesterday evening to handle an outlier situation observed over the past few months. The modification created a mail loop in specific messages that were missing some ARC seal authentication information in the headers. This specific situation caused the emails to look up the MX record for delivery, and those domains with a Shield MX record caused the email to loop. A fix for this situation was deployed shortly after the mail loop was detected. The fix now accounts for the rare circumstances involving the ARC seal and will gracefully handle mail delivery.
We are investigating an issue causing email delivery performance.
The issue has been isolated, and resources are being reallocated to empty the email delivery queue as quickly as possible.
The team has resolved email delivery performance, and inbound messages have been arriving as expected since about 1:00 PM ET. However, the issue is still monitored as API-related actions (sender training, message moves, etc.) and logs catch up. Please be aware that emails may sit in a folder a little longer than expected, and Spotlight logs may display an incorrect email location while the database catches up on activity. We expect the issue to be resolved entirely this evening.
Email delivery and API activity are operating as expected.
Mailprotector is seeing an increase in User Sync Errors affecting some Google Workspace customers due to a recent Google update. The error occurs when authenticating with Google. The error may be displayed as "Data too long for column". Resolution (required for each affected domain): 1. Delete the User Source under the User Sync tab in your console 2. Add a new User Source and reconnect to Google Workspace Please reach out to support with any additional questions or concerns.
We are continuing to monitor.
User sync errors affecting Google Workspace customers have been resolved. If an error still exists for a domain, please delete and recreate the user sync in the Mailprotector Console or open a request with the support team for guidance.
Mailprotector is aware of intermittent bounces of inbound emails to Microsoft 365 domains. The bounces reference one of two reasons: "banned sending IP" or "blocked using Spamhaus." In both cases, Microsoft and Spamhaus do not indicate that the IP addresses are blocked when a delist request is made. Mailprotector has two open and active support requests with Microsoft to explain and resolve the situation. Mailprotector has been attempting to resolve the issue with Microsoft for over a week. Unfortunately, we have seen increased bounces in the last 24 hours. Partners and users may click on the delist links in the bounce notification. However, the delist will reply that the IP addresses are not listed. It is important to note that this is not an outbound reputation issue. Microsoft is randomly rejecting emails from IP addresses that only relay inbound emails after being filtered. The IP addresses are added to the recommended inbound connector for M365 deployments.
Bounces over the last 7 days have subsided, but an increase has been observed today, April 29. The number of bounces is tiny; however, we know a single bounce can frustrate a customer. Communication with Microsoft was escalated last week, and we await more information.
Since April 30, little to no bounces have been observed. We are continuing to monitor the situation and waiting for Microsoft's response to explain the reason and a permanent resolution.
Since April 29, there have been unnoticeable bounces. While we cannot be certain that Microsoft will not repeat the issue, the status is considered resolved at this time. If you receive a bounce for "banned sending IP" or "blocked using Spamhaus", please open a support request.