55 Paubox incidents · май 2023 г. — official updates, affected components, duration and resolution details.
We have identified an issue blocking log in to our admin app and should have a fix in place shortly.
A fix has been implemented and we are monitoring the results - log in is confirmed functional again.
The incident has been resolved.
Мы расследуем сообщения от некоторых клиентов Inbound Security о том, что их входящие карантины пусты, несмотря на то, что почта помечена как карантин в их почтовых журналах.
Никаких дефектов не выявлено, никаких изменений отправлено не было. Есть две причины, по которым клиенты могут столкнуться с этим: 1) Паубокс почти ежедневно улучшает доставку - если что-то внезапно недоступно, вам может потребоваться выйти из системы/вернуться, чтобы обновить доступ. 2) Для отображения результатов входящего карантина необходимо выбрать "все категории" или по меньшей мере одну категорию. Если не выбраны категории, страница не покажет результатов.
Автоматический перевод официального обновления инцидента.
В настоящее время мы расследуем этот вопрос.
Этот инцидент был урегулирован.
Автоматический перевод официального обновления инцидента.
Клиенты сообщают, что входящий карантин безопасности нефункционален - исправление продолжается.
У нас есть решение, и мы отслеживаем результаты.
Этот инцидент был урегулирован.
Автоматический перевод официального обновления инцидента.
Beginning at 6AM UTC today, Microsoft began presenting an updated certificate on some of their hosted mail accounts, and our outbound servers' list of trusted authorities doesn't yet include the new one. As a result, a portion of outbound emails to recipients hosted on Microsoft 365 are currently being routed through the Paubox Secure Message Center instead of delivered directly to the recipient's inbox. Affected recipients receive a notification with a secure link to view the message in our portal. No emails have been lost and no data security has been compromised — the Secure Message Center fallback is operating as designed to protect message content when direct delivery cannot be verified. We are updating the certificate and should have a resolution shortly.
We are rolling out a new cert server by server and monitoring.
Direct delivery to Microsoft 365 recipients has been restored. Messages sent during the incident were delivered securely via the Paubox Secure Message Center; no mail was lost. Thank you for your patience.
Some customers using our hosted/webmail reported certificate issues blocking access on mobile devices. We have refreshed our certificates again - if you are still seeing an error, please clear your cookies and cache. If this does not resolve your issue, take a screenshot and email [email protected] so we can isolate your specific issue.
This incident has been resolved.
We have had a few calls from some Google Workspace customers report inbound & outbound delays on internal emails which Down Dectector corroborates - see https://downdetector.com/status/gmail/ - we will continue to monitor this.
Google has updated this site with more details: https://support.cloud.google.com/portal/system-status?product=WORKSPACE&cssp=true We will continue to monitor.
Google has published this https://www.google.com/appsstatus/dashboard/incidents/224ozRqzW4sFBDK8hLnT We continue to monitor.
Google has been releasing backed up mail - we expect resolution by 3pm PDT (their resolution estimate).
Google has marked this issue resolved.
We are currently investigating outbound delays for Hosted mail & SMTP server solution customers. We have not observed issues for any other outbound customers at present.
The issue has been identified and a fix is being implemented.
A fix has been implemented and we are monitoring the results.
Queues have recovered and delays as resolved.
We are currently investigating inbound mail delays.
We have identified a database issue and are implementing a fix.
A fix is in place and we are monitoring results.
We are continuing to monitor - the deferred mail queue has dropped ~60% from the highest spike and new mail is flowing. We anticipate full resolution within the hour.
Inbound queues are healthy again. We'll complete an internal post mortem tomorrow and have an RCA available upon request.
Customers using our hosted/webmail platform and smtp server solution have reported all their mail in logs is marked as queued. We are investigating.
We have identified the cause and are implementing a fix.
A fix has been implemented and we are monitoring results.
We observed status updates universally, but we discovered 2 customers still experiencing too many "queued" emails. Because the larger issue is resolved, we'll close this incident - but we are working on a fix and will communicate directly with the affected parties.
Outbound email customers are reporting delays in excess of 10 minutes. We are investigating.
The issue has been identified and a fix is in progress.
A fix has been implemented and we are monitoring results - current email testing is coming in under 5 minutes.
This incident has been resolved. Queues have returned to normal and delayed mail is being delivered.
We are investigating why users are unable to verify MFA codes and begin fresh sessions in our admin app. If you are already logged in, you can navigate as usual.
The issue has been isolated and a fix is in progress.
A fix has been implemented & we are monitoring results.
The issue has been resolved.
We are observing intermittent delivery delays affecting a limited subset of outbound messages to certain external domains (including hotmail.com and outlook.com). Our systems remain fully operational. We are actively monitoring and adjusting sending patterns as needed.
We are no longer observing intermittent delays - all mail that was deferred yesterday has been delivered.
Inbound email customers have reported a delay in their quarantine log propagation. Emails are marked quarantine in their mail log, but do not appear in quarantine until hours later. There is no mail flow or data loss issue, but we have identified a problem with timestamps and are working to correct it.
We have shipped a correction and are monitoring the results.
This incident has been resolved.
Microsoft365 outage reported on the Microsoft Service Health Status page
This incident has been resolved. Microsoft Service Health Status - https://status.cloud.microsoft/m365/referrer=serviceStatusRedirect
Some customers report outbound mail delays in excess of 10 min - we are investigating
We have identified the issue and are working on a fix.
We're continuing to work on the fix for this issue.
A fix is in place a new mail is flowing normally - we expect another 3-4 hours for the deferred queue to unspool. We'll continue to monitor and update you if anything changes.
Customers are reporting missing outbound mail log data - we are investigating.
Mail log data has been restored. We are continuing to monitor the deferred queue while it processes the backlog.
The deferral queue has cleared and outbound is functioning normally.
Some customers are reporting outbound deferrals from Yahoo & affiliates - we are keeping an eye on this.
We are continuing to monitor reports of Yahoo deferrals independent of today's incident.
Email to yahoo.com and affiliates (aol.com, verizon.net, att.net, sbcglobal.net, cox.net, etc) is delivering as normal.
We are investigating reports of HTTP response timeouts on Email API.
We have identified the issue and are working on a fix.
We have implemented a fix and are monitoring the results.
We have been unable to recreate timeout errors post update.
We're currently investigating delays for some inbound mail. Customers viewing their Paubox dashboard's Inbound Security Mail Log may see a mix of emails with "Delivered" status and "No Delivery Status Found".
The issue has been identified and a fix is being implemented.
A fix has been implemented and we are monitoring the results.
This incident is now resolved. Any mail that was delayed has been delivered or is delivering now. Where applicable, a new Inbound Security Mail Log entry with "Delivered" status was created. Please disregard older, duplicate "No Delivery Status Found" mail log entries for the same email due to this issue.
Some inbound customers are reporting delays and deferrals on receiving mail. We have identified the cause and are working on a resolution.
A fix has been shipped and we are monitoring results.
while we released a fix last night, some customers continue to report missing mail log statuses from Monday 9/8 to Tuesday 9/9 - we are still reviewing these reports
After thorough investigation we found: - A new log entry was added each time a piece of mail was deferred - “No Delivery Status Found” mail log rows didn’t get updated when the email was later delivered - Instead, a brand new mail log entry with the “Delivered” status was written