Expel 中断历史

1. 2026年8月30日 UTC 09:43identified

   Workbench的邮件从协调世界时(UTC)12点起被延迟. 我们已查明了根源,并正在努力进行补救。 自协调世界时上午12时起,延迟了对结果的答复、用户电子邮件通知和Org-Level电子邮件罚单通知。 SOC操作和警报摄取不受影响. 我们将在协调世界时下午12点30分前提供最新消息

2. 2026年8月30日 UTC 11:10identified

   我们已查明了这一原因,并正在继续努力进行补救。 我们还确认,由于这个问题,当有人提交可疑的打字邮件时,电子邮件将无法到达Workbench以供审查。 下一次更新时间为UTC下午1点.

3. 2026年8月30日 UTC 12:58identified

   我们正在继续努力进行补救。 妇幼保健服务不受影响。 警报摄入、侦测和调查正常进行。 Slack, Microsoft Teams, PagerDuty, Opsgenie, 和Service Now的通知仍在发送中. Workbench的电子邮件发送仍然无法提供。 目前无需采取行动;一旦恢复服务,我们将确认重新提交钓鱼申请的步骤。 下一次更新:8月31日星期一,世界协调时下午1:00(9:00 AM ET)

4. 2026年8月30日 UTC 18:14monitoring

   Workbench的电子邮件发送正在工作,排队的电子邮件正在被重播。 重播完成后将发布更新.

5. 2026年8月30日 UTC 19:53resolved

   这个问题已经解决了。 来自Workbench的电子邮件发送已经恢复,并正在正常处理网页邮件提交。 在此期间提交的可疑的打字邮件没有被Expel收到,无法处理。 请要求您的用户重新提交在此期间报告的任何信息 。 在此期间生成的通知电子邮件已经发送。 有些可能晚于最初生成的时间戳.

自动翻译自官方事件更新。

1. 2026年7月17日 UTC 02:21investigating

   9:58pm EDT,我们开始遇到Workbench数据库的问题,它阻止用户访问Workbench. 我们将在晚上11点之前提供最新情况.

2. 2026年7月17日 UTC 02:34monitoring

   我们已执行了一项固定措施,相信这一事件将得到解决。 我们正在监测基础设施的健康情况,如果没有悬而未决的问题,我们将解决这个问题。 如果你遇到问题请告诉我们.

3. 2026年7月17日 UTC 03:01resolved

   事件已经解决,所有系统再次完全可用. 没有丢失客户数据;SOC业务受到影响,但恢复正常.

自动翻译自官方事件更新。

1. 2026年7月13日 UTC 20:20investigating

   6月18日,Expel的环境发生了改变,无意中引入了默默地给供应商提示索引的逻辑. 7月1日确定后,为了尽快恢复警示,这一更改被恢复. 我们的团队已经查明了在受影响服务期间发生的几个警报,这些警报发生在我们正在重播的若干客户环境中。 如果出现任何事故,将通过Workbench通知受影响的客户。 同时,请毫不犹豫地向我们的团队提出任何问题.

2. 2026年8月12日 UTC 17:47resolved

   This incident has been resolved.

自动翻译自官方事件更新。

1. 2026年6月18日 UTC 20:37investigating

   At 2pm ET, we began experiencing a problem with our notification system that is causing notification delays to our customers. We will provide an update by 5pm ET.

2. 2026年6月18日 UTC 21:05identified

   We've identified the cause of the delayed notifications as an issue with a third-party notification vendor. A fix is currently being applied. Workbench notifications have been delayed since 19:09 UTC, and we'll provide another update at 21:30 UTC.

3. 2026年6月18日 UTC 21:35identified

   Workbench notifications continue to be delayed since 19:09 UTC due to an issue with a third-party notification vendor. They are working on a fix. All delayed notifications will be processed once the outage is resolved. Next update 22:00 UTC

4. 2026年6月18日 UTC 22:13monitoring

   The cause of the notification delays has been resolved and we are observing delayed messages being delivered. All delayed notifications will be processed and we will continue to monitor.

5. 2026年6月18日 UTC 23:28resolved

   All delayed messages have been delivered and the notifications system is fully operational.

1. 2026年6月17日 UTC 23:11investigating

   We are investigating an issue affecting the processing of phishing submissions. Beginning at 19:47 UTC, phishing submissions are not being processed, which is preventing the creation of Phishing Expel Alerts. Suspected phishing emails submitted for review during this period may not yet be analyzed. Our team is actively working to identify the cause. We will provide an update by 00:00 UTC.

2. 2026年6月18日 UTC 00:07investigating

   We are continuing to investigate this issue

3. 2026年6月18日 UTC 00:59investigating

   We are continuing to investigate this issue. Will post another update in 60 minutes.

4. 2026年6月18日 UTC 02:12investigating

   We think we have identified the problem and are working on a resolution right now. Will update again in 60 minutes.

5. 2026年6月18日 UTC 03:17investigating

   Beginning at 19:47 UTC, phishing submissions are not being processed, which is preventing the creation of Phishing Expel Alerts. Suspected phishing emails submitted for review during this period may not yet be analyzed. We've identified the cause and we are working with the vendor to address the issue. We expect to retrieve and process them once service is restored.

6. 2026年6月18日 UTC 13:29resolved

   This incident has been resolved.

1. 2026年6月17日 UTC 16:16investigating

   At 10am ET, we began experiencing a problem with alert ingestion for some AWS Cloudtrail devices. We will provide an update by 1pm ET.

2. 2026年6月17日 UTC 16:37monitoring

   A fix has been implemented and we are monitoring the results.

3. 2026年6月17日 UTC 16:59resolved

   This incident has been resolved.

1. 2026年6月9日 UTC 15:40investigating

   Workbench is currently experiencing connectivity issues and may be inaccessible for some users. We are investigating the issue and will provide an update within an hour.

2. 2026年6月9日 UTC 16:42investigating

   Workbench is still experiencing connectivity issues. We are still investigating the issue and will provide an update in 1 hour.

3. 2026年6月9日 UTC 17:22monitoring

   We have identified the issue, have rolled out a fix and are monitoring the situation.

4. 2026年6月9日 UTC 17:24monitoring

   We are continuing to monitor for any further issues.

5. 2026年6月9日 UTC 17:33resolved

   This incident has been resolved.

1. 2026年5月13日 UTC 16:23identified

   We are aware of an issue with the Expel Alerts count being inflated for customers. We have identified the root cause and are working on a remediation. SOC operations and alert ingestion are not affected. We will provide another update by 1pm EDT.

2. 2026年5月13日 UTC 16:38identified

   We have remediated the issue with Expel Alerts count being inflated for customers by shifting to a different source of statistics. We are working on fixing the underlying issue with our main source of count data. We will provide another update by 1pm EDT.

3. 2026年5月13日 UTC 16:46resolved

   The incident has been resolved, and all alerts counts are once again accurate. No customer data was lost, and SOC operations were not impacted.

1. 2026年5月1日 UTC 18:50investigating

   At 2pm EDT, we began experiencing a problem with the customer device secrets service that is preventing our infrastructure from accessing customer security devices. We will provide an update by 3:30pm EDT.

2. 2026年5月1日 UTC 19:09monitoring

   We have implemented a fix and believe the incident to be resolved. We are seeing customer security devices returning to healthy status. We are monitoring the health of the infrastructure and will post another update by 4pm EDT.

3. 2026年5月1日 UTC 20:01resolved

   The incident has been resolved, and all systems are once again fully available. No customer data was lost, and all previously-accessible security devices are once again accessible and ingesting alerts.

1. 2026年4月25日 UTC 17:42investigating

   At 1pm ET, we began experiencing a problem with our notification system that is causing notification delays to our customers.  We will provide an update by 2:30pm ET.

2. 2026年4月25日 UTC 18:09resolved

   This issue has been resolved. All notifications in the backlog have been delivered and new notifications are being delivered as normal.

1. 2026年4月20日 UTC 16:20investigating

   At 10:59AM EDT, we noticed that a set of devices have been unhealthy since last week due to invalid credentials. We believe the cause to be a change to our device credentials storage subsystem. We are working to identify the specific set of devices and restore the credentials. Once restored, all delayed alerts will be ingested. We will provide an updated by 1pm EDT.

2. 2026年4月20日 UTC 17:00investigating

   We have identified the specific subset of devices that are affected and are working on restoring the credentials for those devices. Once restored, all alerts that have not been ingested for these devices will be ingested. We will provide another update by 1:30pm EDT.

3. 2026年4月20日 UTC 17:35investigating

   We have identified the specific subset of devices that are affected and are working on restoring the credentials for those devices. Once restored, all alerts that have not been ingested for these devices will be ingested. We will provide another update by 2:00pm EDT.

4. 2026年4月20日 UTC 18:01investigating

   We have identified the specific subset of devices that are affected and are working on restoring the credentials for those devices. Once restored, all alerts that have not been ingested for these devices will be ingested. We will provide another update by 2:30pm EDT.

5. 2026年4月20日 UTC 18:34identified

   We have restored the credentials for the affected subset of devices. We are monitoring closely to ensure proper ingestion of delayed alerts. We will provide another update by 3:00pm EDT.

6. 2026年4月20日 UTC 19:02identified

   We have restored the credentials for the subset of devices. We are monitoring closely to ensure proper ingestion of delayed alerts. We will provide another update by 3:30pm EDT.

7. 2026年4月20日 UTC 19:34monitoring

   We restored the credentials for the subset of devices, and are seeing that the majority of devices have returned to a healthy state. We are investigating the remaining unhealthy devices, which do not seem related to the credentials. We will provide another update by 4:00pm EDT.

8. 2026年4月20日 UTC 20:01monitoring

   We restored the credentials for the subset of devices, and are seeing that the majority of devices have returned to a healthy state. We are investigating the remaining unhealthy devices, which do not seem related to credentials. We will provide another update by 4:30pm EDT.

9. 2026年4月20日 UTC 20:54resolved

   We restored the credentials for the subset of devices, and are seeing that the majority of devices have returned to a healthy state. We investigated the remaining unhealthy devices, which are not related to the customer secrets vault upgrade. This incident is resolved.

1. 2026年3月24日 UTC 16:58investigating

   Notifications are currently not being delivered to Microsoft Teams since 4pm UTC. We are actively investigating to determine and address the root cause. We advise you login to Workbench for timely updates and/or to engage with Expel's SOC until the issue is resolved. Updates will be shared as soon as they are available. We will provide an update by 5:30pm UTC.

2. 2026年3月24日 UTC 17:49identified

   Update: We have confirmed there are no technical issues with Workbench or with our third party systems that support notifications. Microsoft has since updated its Teams service health to indicate they are investigating a possible issue. We recommend you check your MSFT 365 service health and Microsoft's status page for updates: https://status.cloud.microsoft/m365/referrer=serviceStatusRedirect Until further notice, please continue to login to Workbench for timely updates and/or to engage with Expel's SOC. We will provide notice when Workbench notifications to Teams are operational.

3. 2026年3月24日 UTC 21:02monitoring

   Update: We are observing a broad recovery in MS Teams message delivery as of 19:10 UTC. However, Microsoft has confirmed that service remains degraded for some users in the West United States region due to ongoing infrastructure optimizations. We are continuing to monitor the situation. Any messages delayed during this incident are being automatically reprocessed for delivery.

4. 2026年3月24日 UTC 21:12resolved

   Resolved: Microsoft has confirmed that the issues affecting Teams bots and message delivery are now fully resolved. All delayed messages have been reprocessed and delivered. Service is operating normally.

1. 2026年3月2日 UTC 18:35monitoring

   We are currently monitoring an outage affecting the AWS GuardDuty service. As a result of this outage, our integration with AWS GuardDuty is currently reporting as unhealthy. We will provide an update within 60 minutes, or as soon as more information is available.

2. 2026年3月2日 UTC 19:36monitoring

   We are working on a workaround that will bypass the faulty AWS region in order resume alert ingestion from healthy regions. Once the outage in the faulty region is over, we will resume ingestion in that region. We will provide an update within 60 minutes, or as soon as more information is available.

3. 2026年3月2日 UTC 20:36resolved

   Workaround is in place, we are bypassing the faulty AWS region and all impacted devices have resumed ingestion. Once the outage in the faulty region is over, we will resume ingestion in that region. Alert Ingestion issue resolved.

1. 2026年2月2日 UTC 21:00investigating

   Expel is investigating an issue with Workbench notifications that may be preventing users from receiving outbound notifications from Workbench to User Email, Email Ticketing, Slack, MS Teams, PagerDuty, and OpsGenie integrations since 19:30 UTC. We will provide an update by 21:30 UTC.

2. 2026年2月2日 UTC 21:15identified

   We have identified the root cause as an issue with a 3rd party vendor. They are actively working on resolving the issue. We have confirmed that outbound notifications from Workbench have not been delivered to to User Email, Email Ticketing, Slack, MS Teams, PagerDuty, and OpsGenie since 19:30 UTC. These notifications are queued and will be re-sent when the issue is resolved. Alert ingestion is not affected. We will provide another update by 21:30 UTC.

3. 2026年2月2日 UTC 21:41identified

   Outbound notifications from Workbench to User Email, Email Ticketing, Slack, MS Teams, PagerDuty, and OpsGenie continue to be delayed. We will provide an update by 22:00 UTC.

4. 2026年2月2日 UTC 22:00identified

   Outbound notifications from Workbench to User Email, Email Ticketing, Slack, MS Teams, PagerDuty, and OpsGenie continue to be delayed. We will provide an update by 22:30 UTC.

5. 2026年2月2日 UTC 22:12monitoring

   We are observing that delayed messages are starting to be delivered. We will monitor the backlog of delayed notifications and the health of the system. We will provide an update by 22:30 UTC.

6. 2026年2月2日 UTC 22:33resolved

   The incident has been resolved, and all delayed notifications have been delivered. No customer data was lost, and SOC operations were not impacted.

1. 2026年1月20日 UTC 20:31investigating

   At 3:00pm ET, we began experiencing a problem with alert ingestion. We will provide an update by 4:30pm ET.

2. 2026年1月20日 UTC 20:54monitoring

   A fix has been implemented and we are monitoring the results.

3. 2026年1月20日 UTC 23:02resolved

   This incident has been resolved.

1. 2026年1月7日 UTC 19:51investigating

   Starting 1/6/25 at roughly 12:00 EST, we have identified an issue with our phishing service where outcome emails are being sent multiple times to some users and are investigating the root cause. Ingestion and email review are not affected. We will provide an update at 4pm EST.

2. 2026年1月7日 UTC 21:11identified

   We believe we have identified the root cause of duplicate phishing outcome emails and are preparing a hotfix. Next update at 5pm EST.

3. 2026年1月7日 UTC 22:00monitoring

   The hotfix for duplicate phishing outcomes has been deployed and we are monitoring. Next update at 6pm EST.

4. 2026年1月7日 UTC 22:54resolved

   The hotfix for phishing submission outcome emails was successful and behavior has returned to normal. This incident is resolved.

1. 2025年12月29日 UTC 21:26investigating

   At 3:17, we began experiencing a problem with enrichment service that is impacting detections. We will provide an update by 4:45.

2. 2025年12月29日 UTC 21:49monitoring

   We have identified and implemented a fix and are monitoring; service queues are approaching normal.

3. 2025年12月29日 UTC 22:09resolved

   Detection pipelines are caught up. Service is nominal.

1. 2025年11月18日 UTC 15:34identified

   Workbench cannot currently deliver notifications to Microsoft Teams. The issue began around 9:45am ET. Expel is actively investigating the issue to determine if it is related to the larger Cloudflare outage. We recommend you utilize email notifications in the interim, and can confirm communicating with Expel personnel via Microsoft Teams still works. The issue is specific to Workbench notifications.

2. 2025年11月18日 UTC 15:50monitoring

   The underlying issue has been resolved We are monitoring the health of the infrastructure and will resolve this if there are no outstanding issues. Please let us know if you are experiencing any problems.

3. 2025年11月19日 UTC 18:17resolved

   This incident has been resolved.

1. 2025年11月12日 UTC 15:28identified

   We’re upgrading our API connection and that may impact delivery of some alerts to Workbench. We are actively investigating and collaborating on a solution to ensure the consistent and complete delivery of all alert data. Please reach out to Orca if you’d like additional information. Thank you for your patience and understanding as we work through this.

2. 2026年4月17日 UTC 13:52resolved

   This incident has been resolved.

1. 2025年11月3日 UTC 16:56investigating

   At 10:50am ET, we began experiencing a problem with pivoting to Wiz console that resulting in some degraded analyst triage experience. We will provide an update by 12:50pm ET.

2. 2025年11月3日 UTC 17:50investigating

   We have reached out to Wiz and waiting to hear back. We will provide an update by 1:50pm ET.

3. 2025年11月3日 UTC 18:55monitoring

   Analysts are starting to successfully SSO into the Wiz console. We will continue to monitor the situation. We will provide an update by 3:00pm ET.

4. 2025年11月3日 UTC 19:43resolved

   Analysts are able to successfully SSO into the Wiz console. Incident is closed.