52 Mailprotector incidents · 2023年3月 — official updates, affected components, duration and resolution details.
我们正在调查一些使用盾牌域的连接超时或速率限制的报告。 这并不影响所有邮件的流动.
我们正在继续调查这一问题.
今早微软服务器与Shield通讯次数的减少,似乎促成了每服务器量的激增。 队伍缓解了连接超时和速率限制,队列排出. 我们没有确切的时间排出队列,但似乎应该在一小时之内,很可能更早。 我们正在继续优先调查,恢复稳定和预期的邮件流动.
队列被赶上了,我们正在观察正常的邮件流动。 由于微软服务器与Shield的通讯减少, 我们将继续在整个工作日监督我们的排队情况,以确保处理工作的稳定.
这一事件已经得到解决.
自动翻译自官方事件更新。
大约1: 00 PM ET,我们被警告在希尔德慢消息处理. 症状是消息更新缓慢和一些送出延迟. 小组确定了瓶颈,并解决了这一问题。 队列正在赶上,性能似乎正在正常化. 我们将继续监测排队情况,以便全面解决.
排队已经赶上,性能正在按预期水平运行.
自动翻译自官方事件更新。
CloudFilter 的电子邮件日志数据在 Console 不可见 。 今早9点左右, 该小组正在调查这一问题,我们将尽快提供更多信息.
电子邮件日志数据停止传输到存储层. 团队已解决了木料运输,木料正在恢复. 记录的回填将持续到下午,一旦记录赶上,我们将更新这一通知.
CloudFilter的电子邮件日志已经赶上,当前日志正在如预期的那样显示.
自动翻译自官方事件更新。
We are currently experiencing an issue with our Microsoft 365 tenant that is preventing us from receiving email replies to our support address. We are working to resolve this. If you need to reach us, please log into the support portal (support.mailprotector.com) and submit or update your request there. Portal submissions are working normally and will reach our team. We will update this notice once email replies are restored.
Support tickets and replies from Mailprotector will temporarily originate from a zendesk.com email address rather than our standard mailprotector.com domain. This is an intentional change we made to restore email reply functionality to our ticketing system. Emails arriving from zendesk.com on behalf of Mailprotector Support are legitimate. If your email filtering flags these messages as possible impersonation, they are safe to release and trust. We will update this notice when we return to our standard domain.
Support tickets and repllies are back to originating from mailprotector.com. No changes or actions need to be taken. Support emails will resume their normal behavior.
Microsoft is experiencing intermittent issues with mail flow. Please see their notice for reference. https://admin.cloud.microsoft/#/servicehealth/:/alerts/EX1331830 Mailprotector is monitoring the situation. However, mail flow with CloudFilter and Shield appears to be normal. Please check logs to identify bounces or delays from Microsoft hosts to confirm potential effects from their systems.
Microsoft Defender is restricting connectors due to false-positive reports. Please go to Microsoft Defender > Restricted Entities and unblock the listed connectors. (https://security.microsoft.com/restrictedentities) You will need to monitor and check on this restriction for the time being. We have observed that Microsoft restricts the connector after a short time, and the unblock request must be made again. We are continuing to monitor and investigate possible mitigation options.
Earlier today, Microsoft identified some of Shield's services as suspicious or compromised. This caused emails to bounce or be delayed. As of approximately 2:30 PM ET, the status began to lift, and we observed a return to expected mail flow. A support request has been opened with Microsoft to help us determine how and why this incident happened. We have not received a response yet. We are not closing this issue as resolved. Our intention is to get to the root of the incident. Our first priority was to find a way to resume normal mail flow, and it appears we have done so. We are continuing to monitor the situation, including the Microsoft notice regarding mail flow issues in Exchange Online, which appear to have been resolved.
We have begun reprocessing and redelivering emails that bounced during the affected time period. Here is what to expect: Most emails should be delivered, but not all of them. Some bounces during the incident may have been legitimate, and those messages will not be recoverable through reprocessing. Duplicates are possible. If a sender resent their email after the incident and the original message is also redelivered through our reprocessing, recipients may receive both. This is expected behavior and not a cause for concern. Reprocessing takes time. We expect this to be completed by midnight ET tonight, though it may finish sooner.
This incident has been resolved.
This morning, a third-party package repository used by our virus scanning infrastructure experienced an outage following an attack on its systems. Because our virus scanning instances rely on this repository at startup to pull the latest files and configurations, affected instances were unable to start. This reduced the capacity of our virus-scanning cluster below demand. When a virus scan cannot complete, our system quarantines the email as a precaution. As a result, emails processed during this window were quarantined rather than delivered to inboxes. No emails were lost. Affected emails can be released from quarantine by users or admins. Resolution The virus scanning cluster has been reconfigured to pull packages directly from the source repository, eliminating the dependency on the third-party intermediary. Monitoring and alerting thresholds have also been updated to surface this class of issue more quickly in the future. The issue is fully resolved.
We have been made aware that emails sent from Google Workspace accounts via CloudFilter/SafeSend are failing intermittently. There may also be delays in delivering inbound email due to an issue Google is experiencing. If you have a domain affected by this incident, we recommend temporarily disabling the outbound mail routes for Gmail. You may check the status of Google's incident at https://support.cloud.google.com/portal/system-status?product=WORKSPACE&cssp=true
This incident has been resolved.
An update to improve brand-related impersonation attacks was aggressively holding outgoing messages and sending inbound emails to the Junk E-mail folder. The update has been rolled back, and the held outgoing emails have been released. Emails delivered to the Junk E-mail folder must be moved out by users. The aggressive protection was active from approximately 2:00 PM ET to 3:00 PM ET.
Emails that were delivered to the Jail due to the aggressive behavior of the update have been released. The emails may still end up in the Junk E-mail folder if the risk level is medium or low from a new sender.
An update released at approximately 2:00 PM ET today was designed to improve protection against brand impersonation. It behaved more aggressively than intended, holding and jailing legitimate emails, particularly those with signatures containing social media links. The update was rolled back at approximately 2:50 PM ET. This issue is resolved. Actions taken to remediate after rollback: - Inbound emails delivered to Junk have been reprocessed to ensure appropriate risk and moved to the Inbox. - Released jailed emails; they may still land in Junk if the sender is new or carries a medium risk level. - Outgoing emails that were held have been released. No action needed.
Microsoft has announced a service issue affecting Exchange Online in M365. We are seeing email delivery delays to M365 domains with temporary error notifications. We will continue to monitor the Microsoft report. Mailprotector's CloudFilter and Shield services are operational.
Microsoft's status has not changed. For more information, please visit https://status.cloud.microsoft/
We will continue to monitor the status Microsoft provides at https://status.cloud.microsoft/. It appears Microsoft is making progress in some of their affected zones. Logs shows some emails are beginning to deliver, but without Microsoft's confirmation of a resolution, we expect deferrals to continue.
This incident has been resolved.
The team has identified an issue with email log ingestion and backend storage. A fix is being prepared. Logs will get backfilled once the issue is resolved. The log issue does not impact mail flow. Email delivery is performing as expected.
The log ingestion issue is resolved, and logs have been backfilled.
An issue with API notifications from Microsoft was discovered, and a fix is already being deployed. Mail flow has not been interrupted; however, the movement of emails to a folder based on sender preferences or risk level without mailbox rules was impacted. The fix is being deployed.
The fix has been deployed. New emails are being placed in the appropriate location as expected per X-ray log details. Emails that were not moved prior to the fix will remain in their current location. Moving the message will create an opportunity for Shield to act on the message, if needed.
An underlying issue at AWS is preventing an essential Radar task from completing. Radar will not complete the round-trip test and return results. We continue to monitor AWS's investigation into the failed service.
We are continuing to monitor the situation. AWS is continuing to investigate the issue.
Radar is operating as expected. However, AWS has not confirmed that the affected service has been fully resolved. We will continue to monitor the situation.
This incident has been resolved.
We are investigating errors connecting to CloudMail via webmail and email clients.
Connections to CloudMail are operating as expected. We will continue to monitor the status.
This incident has been resolved.
AWS is experiencing service issues that are impacting CloudFilter's quarantine. Emails destinated for the quarantine are queued. Emails passing filtering are being delivered to the destination recipients. Only quarantined emails are affected. You can follow the AWS updates at https://health.aws.amazon.com/health/status
AWS has reported progress but quarantine performance effects remain. We are continuing to monitor and work with AWS.
The Mailprotector Console is currently unavailable due to the AWS issues. Mail flow for good emails continues to operate as expected. To reiterate, good emails are being delivered to recipient servers. The Console and quarantine are currently affected.
The Mailprotector Console is back online. Quarantined emails continue to be queued and "good" emails are delivering to recipient servers.
Quarantine services are beginning to recover. We continue to monitor the progress reported by AWS and the recovery of queues and services affected by the outage.
Quarantine queues are continuing recovery and services appear to be operating as expected. We will monitor the AWS status through the morning.
We are currently investigating the issue. Email is being accepted in mailboxes; however, access via webmail and IMAP is interrupted.
A server service was unstable. The team has resolved the issue. Webmail and IMAP access is working as expected. We are continuing to monitor the service.
This incident has been resolved.
A permissions change caused the Mailprotector Console to lose connection to a status database. The result was a false report of mail flow errors in the Console and email notifications to admin accounts configured for the alerts. The permissions have been corrected, and the mail flow status in the Console for CloudFilter shows accurate information. There was no mail flow interruption.
We are investigating an inbound delivery delay. Delays of approximately 20 minutes have been observed.
Inbound delivery is performing as expected. We are continuing to monitor the flow.
Inbound delivery has continued to perform as expected. This incident is resolved.
A modification to handling ARC seal authentication was deployed late yesterday evening to handle an outlier situation observed over the past few months. The modification created a mail loop in specific messages that were missing some ARC seal authentication information in the headers. This specific situation caused the emails to look up the MX record for delivery, and those domains with a Shield MX record caused the email to loop. A fix for this situation was deployed shortly after the mail loop was detected. The fix now accounts for the rare circumstances involving the ARC seal and will gracefully handle mail delivery.
We are investigating an issue causing email delivery performance.
The issue has been isolated, and resources are being reallocated to empty the email delivery queue as quickly as possible.
The team has resolved email delivery performance, and inbound messages have been arriving as expected since about 1:00 PM ET. However, the issue is still monitored as API-related actions (sender training, message moves, etc.) and logs catch up. Please be aware that emails may sit in a folder a little longer than expected, and Spotlight logs may display an incorrect email location while the database catches up on activity. We expect the issue to be resolved entirely this evening.
Email delivery and API activity are operating as expected.
Mailprotector is seeing an increase in User Sync Errors affecting some Google Workspace customers due to a recent Google update. The error occurs when authenticating with Google. The error may be displayed as "Data too long for column". Resolution (required for each affected domain): 1. Delete the User Source under the User Sync tab in your console 2. Add a new User Source and reconnect to Google Workspace Please reach out to support with any additional questions or concerns.
We are continuing to monitor.
User sync errors affecting Google Workspace customers have been resolved. If an error still exists for a domain, please delete and recreate the user sync in the Mailprotector Console or open a request with the support team for guidance.
