Expel 장애 기록

1. 2026년 9월 22일 오후 3:26 UTCinvestigating

   오전 10시 30분에 경고 발생으로 문제가 발생했습니다. SOC 운영은 영향을 받지만 한 번의 서비스가 복원됩니다. 데이터가 손실되지 않습니다. 우리는 12:30PM ET에 의해 업데이트를 제공 할 것입니다.

2. 2026년 9월 22일 오후 4:21 UTCinvestigating

   오전 10시 30분에 경고 발생으로 문제가 발생했습니다. SOC 운영은 영향을 받지만 한 번의 서비스가 복원됩니다. 데이터가 손실되지 않습니다. 우리는 1:30PM ET에 의해 업데이트를 제공 할 것입니다.

3. 2026년 9월 22일 오후 5:28 UTCinvestigating

   오전 10시 30분에 경고 발생으로 문제가 발생했습니다. SOC 운영은 영향을 받지만 한 번의 서비스가 복원됩니다. 데이터가 손실되지 않습니다. 우리는 2:30PM ET에 의해 업데이트를 제공 할 것입니다.

4. 2026년 9월 22일 오후 6:24 UTCinvestigating

   오전 10시 30분에 경고 발생으로 문제가 발생했습니다. SOC 운영은 영향을 받지만 한 번의 서비스가 복원됩니다. 데이터가 손실되지 않습니다. 우리는 3:30PM ET에 의해 업데이트를 제공 할 것입니다.

5. 2026년 9월 22일 오후 7:13 UTCmonitoring

   우리는 수정을 구현하고 해결 될 사건을 믿는다. 우리는 인프라의 건강을 모니터링하고 뛰어난 문제가 있다면 이것을 해결합니다. 어떤 문제가 발생하면 알려주십시오.

6. 2026년 9월 22일 오후 9:03 UTCresolved

   이 사건은 해결되었습니다.

공식 인시던트 업데이트를 자동 번역했습니다.

1. 2026년 8월 30일 오전 9:43 UTCidentified

   Workbench의 이메일은 12am UTC 이후 지연됩니다. 우리는 뿌리 원인을 확인하고 치료에 종사하고 있습니다. 응답, 사용자 이메일 알림 및 Org-Level Email Ticketing 알림이 12am UTC 이후 지연됩니다. SOC 운영 및 경고 발생은 영향을 미치지 않습니다. 우리는 12 : 30pm UTC의 다른 업데이트 제공

2. 2026년 8월 30일 오전 11:10 UTCidentified

   우리는 원인이 요법에 계속 작동하도록 식별했습니다. 우리는 또한이 문제로 인해, 누군가가 의심스러운 피싱 이메일을 제출할 때, 이메일은 검토를위한 Workbench에 도달하지 않습니다. 1pm UTC의 다음 업데이트.

3. 2026년 8월 30일 오후 12:58 UTCidentified

   우리는 치료에 계속 노력하고 있습니다. MDR 서비스는 공개되지 않습니다. Alert ingestion, 검출 및 조사는 일반적으로 작동됩니다. Slack, Microsoft Teams, PagerDuty, Opsgenie 및 Service Now 알림은 여전히 전달됩니다. Workbench의 이메일 배송은 사용할 수 없습니다. 이 시간에 아무 작용도 요구되지 않습니다; 우리는 피싱 제출을 위한 resubmission 단계를 한 번 서비스 복원합니다. 다음 업데이트: 월요일, 31 8월 at 13:00 UTC (9:00 AM ET)

4. 2026년 8월 30일 오후 6:14 UTCmonitoring

   Workbench의 이메일 배송은 이제 큐에 이메일이 재생됩니다. 재생이 완료되면 업데이트가 게시됩니다.

5. 2026년 8월 30일 오후 7:53 UTCresolved

   이 문제는 해결됩니다. Workbench의 이메일 배송은 복원되었으며 이메일 제출은 일반적으로 처리됩니다. 이 시간 동안 제출 한 의심의 여지없이 피싱 이메일은 Expel에 의해받지 않고 처리 할 수 없습니다. 이 기간 동안보고 된 모든 것을 재조정하기 위해 사용자를 요청하십시오. 이 시간 동안 생성 된 알림 이메일이 전달되었습니다. 그들은 원래 생성 된 타임스탬프보다 나중에 도착 할 수 있습니다.

공식 인시던트 업데이트를 자동 번역했습니다.

1. 2026년 7월 17일 오전 2:21 UTCinvestigating

   에 9:58오후 EDT, 우리는 Workbench 데이터베이스에 문제가 발생하기 시작했다. 우리는 11pm EDT에 의해 업데이트를 제공 할 것입니다.

2. 2026년 7월 17일 오전 2:34 UTCmonitoring

   우리는 수정을 구현하고 해결 될 사건을 믿는다. 우리는 인프라의 건강을 모니터링하고 뛰어난 문제가 있다면 이것을 해결합니다. 어떤 문제가 발생하면 알려주십시오.

3. 2026년 7월 17일 오전 3:01 UTCresolved

   사건은 해결되고, 모든 체계는 다시 한번 유효합니다. 고객 데이터가 손실되지 않았습니다. SOC 운영은 영향을 미쳤습니다. 그러나 정상으로 돌아갑니다.

공식 인시던트 업데이트를 자동 번역했습니다.

1. 2026년 7월 13일 오후 8:20 UTCinvestigating

   6 월 18 일, 변화는 Expel의 환경에서 만들어졌습니다. 이 변경은 7 월 1 일에 확인 된대로 경고를 복원하기 위해 회신되었습니다. 우리의 팀은 우리가 재생하는 고객 환경의 수 안에 충격을 입힌 서비스 기간 도중 일어난 몇몇 경고를 발견했습니다. 어떤 사건든지 표면, 충격을 입힌 고객은 Workbench를 통해 통보될 것입니다. 그러니 그동안 우리 팀에 어떤 질문으로 닿지 않는 것을 망설이지 말라.

2. 2026년 8월 12일 오후 5:47 UTCresolved

   이 사건은 해결되었습니다.

공식 인시던트 업데이트를 자동 번역했습니다.

1. 2026년 6월 18일 오후 8:37 UTCinvestigating

   At 2pm ET, we began experiencing a problem with our notification system that is causing notification delays to our customers. We will provide an update by 5pm ET.

2. 2026년 6월 18일 오후 9:05 UTCidentified

   We've identified the cause of the delayed notifications as an issue with a third-party notification vendor. A fix is currently being applied. Workbench notifications have been delayed since 19:09 UTC, and we'll provide another update at 21:30 UTC.

3. 2026년 6월 18일 오후 9:35 UTCidentified

   Workbench notifications continue to be delayed since 19:09 UTC due to an issue with a third-party notification vendor. They are working on a fix. All delayed notifications will be processed once the outage is resolved. Next update 22:00 UTC

4. 2026년 6월 18일 오후 10:13 UTCmonitoring

   The cause of the notification delays has been resolved and we are observing delayed messages being delivered. All delayed notifications will be processed and we will continue to monitor.

5. 2026년 6월 18일 오후 11:28 UTCresolved

   All delayed messages have been delivered and the notifications system is fully operational.

1. 2026년 6월 17일 오후 11:11 UTCinvestigating

   We are investigating an issue affecting the processing of phishing submissions. Beginning at 19:47 UTC, phishing submissions are not being processed, which is preventing the creation of Phishing Expel Alerts. Suspected phishing emails submitted for review during this period may not yet be analyzed. Our team is actively working to identify the cause. We will provide an update by 00:00 UTC.

2. 2026년 6월 18일 오전 12:07 UTCinvestigating

   We are continuing to investigate this issue

3. 2026년 6월 18일 오전 12:59 UTCinvestigating

   We are continuing to investigate this issue. Will post another update in 60 minutes.

4. 2026년 6월 18일 오전 2:12 UTCinvestigating

   We think we have identified the problem and are working on a resolution right now. Will update again in 60 minutes.

5. 2026년 6월 18일 오전 3:17 UTCinvestigating

   Beginning at 19:47 UTC, phishing submissions are not being processed, which is preventing the creation of Phishing Expel Alerts. Suspected phishing emails submitted for review during this period may not yet be analyzed. We've identified the cause and we are working with the vendor to address the issue. We expect to retrieve and process them once service is restored.

6. 2026년 6월 18일 오후 1:29 UTCresolved

   This incident has been resolved.

1. 2026년 6월 17일 오후 4:16 UTCinvestigating

   At 10am ET, we began experiencing a problem with alert ingestion for some AWS Cloudtrail devices. We will provide an update by 1pm ET.

2. 2026년 6월 17일 오후 4:37 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

3. 2026년 6월 17일 오후 4:59 UTCresolved

   This incident has been resolved.

1. 2026년 6월 9일 오후 3:40 UTCinvestigating

   Workbench is currently experiencing connectivity issues and may be inaccessible for some users. We are investigating the issue and will provide an update within an hour.

2. 2026년 6월 9일 오후 4:42 UTCinvestigating

   Workbench is still experiencing connectivity issues. We are still investigating the issue and will provide an update in 1 hour.

3. 2026년 6월 9일 오후 5:22 UTCmonitoring

   We have identified the issue, have rolled out a fix and are monitoring the situation.

4. 2026년 6월 9일 오후 5:24 UTCmonitoring

   We are continuing to monitor for any further issues.

5. 2026년 6월 9일 오후 5:33 UTCresolved

   This incident has been resolved.

1. 2026년 5월 13일 오후 4:23 UTCidentified

   We are aware of an issue with the Expel Alerts count being inflated for customers. We have identified the root cause and are working on a remediation. SOC operations and alert ingestion are not affected. We will provide another update by 1pm EDT.

2. 2026년 5월 13일 오후 4:38 UTCidentified

   We have remediated the issue with Expel Alerts count being inflated for customers by shifting to a different source of statistics. We are working on fixing the underlying issue with our main source of count data. We will provide another update by 1pm EDT.

3. 2026년 5월 13일 오후 4:46 UTCresolved

   The incident has been resolved, and all alerts counts are once again accurate. No customer data was lost, and SOC operations were not impacted.

1. 2026년 5월 1일 오후 6:50 UTCinvestigating

   At 2pm EDT, we began experiencing a problem with the customer device secrets service that is preventing our infrastructure from accessing customer security devices. We will provide an update by 3:30pm EDT.

2. 2026년 5월 1일 오후 7:09 UTCmonitoring

   We have implemented a fix and believe the incident to be resolved. We are seeing customer security devices returning to healthy status. We are monitoring the health of the infrastructure and will post another update by 4pm EDT.

3. 2026년 5월 1일 오후 8:01 UTCresolved

   The incident has been resolved, and all systems are once again fully available. No customer data was lost, and all previously-accessible security devices are once again accessible and ingesting alerts.

1. 2026년 4월 25일 오후 5:42 UTCinvestigating

   At 1pm ET, we began experiencing a problem with our notification system that is causing notification delays to our customers.  We will provide an update by 2:30pm ET.

2. 2026년 4월 25일 오후 6:09 UTCresolved

   This issue has been resolved. All notifications in the backlog have been delivered and new notifications are being delivered as normal.

1. 2026년 4월 20일 오후 4:20 UTCinvestigating

   At 10:59AM EDT, we noticed that a set of devices have been unhealthy since last week due to invalid credentials. We believe the cause to be a change to our device credentials storage subsystem. We are working to identify the specific set of devices and restore the credentials. Once restored, all delayed alerts will be ingested. We will provide an updated by 1pm EDT.

2. 2026년 4월 20일 오후 5:00 UTCinvestigating

   We have identified the specific subset of devices that are affected and are working on restoring the credentials for those devices. Once restored, all alerts that have not been ingested for these devices will be ingested. We will provide another update by 1:30pm EDT.

3. 2026년 4월 20일 오후 5:35 UTCinvestigating

   We have identified the specific subset of devices that are affected and are working on restoring the credentials for those devices. Once restored, all alerts that have not been ingested for these devices will be ingested. We will provide another update by 2:00pm EDT.

4. 2026년 4월 20일 오후 6:01 UTCinvestigating

   We have identified the specific subset of devices that are affected and are working on restoring the credentials for those devices. Once restored, all alerts that have not been ingested for these devices will be ingested. We will provide another update by 2:30pm EDT.

5. 2026년 4월 20일 오후 6:34 UTCidentified

   We have restored the credentials for the affected subset of devices. We are monitoring closely to ensure proper ingestion of delayed alerts. We will provide another update by 3:00pm EDT.

6. 2026년 4월 20일 오후 7:02 UTCidentified

   We have restored the credentials for the subset of devices. We are monitoring closely to ensure proper ingestion of delayed alerts. We will provide another update by 3:30pm EDT.

7. 2026년 4월 20일 오후 7:34 UTCmonitoring

   We restored the credentials for the subset of devices, and are seeing that the majority of devices have returned to a healthy state. We are investigating the remaining unhealthy devices, which do not seem related to the credentials. We will provide another update by 4:00pm EDT.

8. 2026년 4월 20일 오후 8:01 UTCmonitoring

   We restored the credentials for the subset of devices, and are seeing that the majority of devices have returned to a healthy state. We are investigating the remaining unhealthy devices, which do not seem related to credentials. We will provide another update by 4:30pm EDT.

9. 2026년 4월 20일 오후 8:54 UTCresolved

   We restored the credentials for the subset of devices, and are seeing that the majority of devices have returned to a healthy state. We investigated the remaining unhealthy devices, which are not related to the customer secrets vault upgrade. This incident is resolved.

1. 2026년 3월 24일 오후 4:58 UTCinvestigating

   Notifications are currently not being delivered to Microsoft Teams since 4pm UTC. We are actively investigating to determine and address the root cause. We advise you login to Workbench for timely updates and/or to engage with Expel's SOC until the issue is resolved. Updates will be shared as soon as they are available. We will provide an update by 5:30pm UTC.

2. 2026년 3월 24일 오후 5:49 UTCidentified

   Update: We have confirmed there are no technical issues with Workbench or with our third party systems that support notifications. Microsoft has since updated its Teams service health to indicate they are investigating a possible issue. We recommend you check your MSFT 365 service health and Microsoft's status page for updates: https://status.cloud.microsoft/m365/referrer=serviceStatusRedirect Until further notice, please continue to login to Workbench for timely updates and/or to engage with Expel's SOC. We will provide notice when Workbench notifications to Teams are operational.

3. 2026년 3월 24일 오후 9:02 UTCmonitoring

   Update: We are observing a broad recovery in MS Teams message delivery as of 19:10 UTC. However, Microsoft has confirmed that service remains degraded for some users in the West United States region due to ongoing infrastructure optimizations. We are continuing to monitor the situation. Any messages delayed during this incident are being automatically reprocessed for delivery.

4. 2026년 3월 24일 오후 9:12 UTCresolved

   Resolved: Microsoft has confirmed that the issues affecting Teams bots and message delivery are now fully resolved. All delayed messages have been reprocessed and delivered. Service is operating normally.

1. 2026년 3월 2일 오후 6:35 UTCmonitoring

   We are currently monitoring an outage affecting the AWS GuardDuty service. As a result of this outage, our integration with AWS GuardDuty is currently reporting as unhealthy. We will provide an update within 60 minutes, or as soon as more information is available.

2. 2026년 3월 2일 오후 7:36 UTCmonitoring

   We are working on a workaround that will bypass the faulty AWS region in order resume alert ingestion from healthy regions. Once the outage in the faulty region is over, we will resume ingestion in that region. We will provide an update within 60 minutes, or as soon as more information is available.

3. 2026년 3월 2일 오후 8:36 UTCresolved

   Workaround is in place, we are bypassing the faulty AWS region and all impacted devices have resumed ingestion. Once the outage in the faulty region is over, we will resume ingestion in that region. Alert Ingestion issue resolved.

1. 2026년 2월 2일 오후 9:00 UTCinvestigating

   Expel is investigating an issue with Workbench notifications that may be preventing users from receiving outbound notifications from Workbench to User Email, Email Ticketing, Slack, MS Teams, PagerDuty, and OpsGenie integrations since 19:30 UTC. We will provide an update by 21:30 UTC.

2. 2026년 2월 2일 오후 9:15 UTCidentified

   We have identified the root cause as an issue with a 3rd party vendor. They are actively working on resolving the issue. We have confirmed that outbound notifications from Workbench have not been delivered to to User Email, Email Ticketing, Slack, MS Teams, PagerDuty, and OpsGenie since 19:30 UTC. These notifications are queued and will be re-sent when the issue is resolved. Alert ingestion is not affected. We will provide another update by 21:30 UTC.

3. 2026년 2월 2일 오후 9:41 UTCidentified

   Outbound notifications from Workbench to User Email, Email Ticketing, Slack, MS Teams, PagerDuty, and OpsGenie continue to be delayed. We will provide an update by 22:00 UTC.

4. 2026년 2월 2일 오후 10:00 UTCidentified

   Outbound notifications from Workbench to User Email, Email Ticketing, Slack, MS Teams, PagerDuty, and OpsGenie continue to be delayed. We will provide an update by 22:30 UTC.

5. 2026년 2월 2일 오후 10:12 UTCmonitoring

   We are observing that delayed messages are starting to be delivered. We will monitor the backlog of delayed notifications and the health of the system. We will provide an update by 22:30 UTC.

6. 2026년 2월 2일 오후 10:33 UTCresolved

   The incident has been resolved, and all delayed notifications have been delivered. No customer data was lost, and SOC operations were not impacted.

1. 2026년 1월 20일 오후 8:31 UTCinvestigating

   At 3:00pm ET, we began experiencing a problem with alert ingestion. We will provide an update by 4:30pm ET.

2. 2026년 1월 20일 오후 8:54 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

3. 2026년 1월 20일 오후 11:02 UTCresolved

   This incident has been resolved.

1. 2026년 1월 7일 오후 7:51 UTCinvestigating

   Starting 1/6/25 at roughly 12:00 EST, we have identified an issue with our phishing service where outcome emails are being sent multiple times to some users and are investigating the root cause. Ingestion and email review are not affected. We will provide an update at 4pm EST.

2. 2026년 1월 7일 오후 9:11 UTCidentified

   We believe we have identified the root cause of duplicate phishing outcome emails and are preparing a hotfix. Next update at 5pm EST.

3. 2026년 1월 7일 오후 10:00 UTCmonitoring

   The hotfix for duplicate phishing outcomes has been deployed and we are monitoring. Next update at 6pm EST.

4. 2026년 1월 7일 오후 10:54 UTCresolved

   The hotfix for phishing submission outcome emails was successful and behavior has returned to normal. This incident is resolved.

1. 2025년 12월 29일 오후 9:26 UTCinvestigating

   At 3:17, we began experiencing a problem with enrichment service that is impacting detections. We will provide an update by 4:45.

2. 2025년 12월 29일 오후 9:49 UTCmonitoring

   We have identified and implemented a fix and are monitoring; service queues are approaching normal.

3. 2025년 12월 29일 오후 10:09 UTCresolved

   Detection pipelines are caught up. Service is nominal.

1. 2025년 11월 18일 오후 3:34 UTCidentified

   Workbench cannot currently deliver notifications to Microsoft Teams. The issue began around 9:45am ET. Expel is actively investigating the issue to determine if it is related to the larger Cloudflare outage. We recommend you utilize email notifications in the interim, and can confirm communicating with Expel personnel via Microsoft Teams still works. The issue is specific to Workbench notifications.

2. 2025년 11월 18일 오후 3:50 UTCmonitoring

   The underlying issue has been resolved We are monitoring the health of the infrastructure and will resolve this if there are no outstanding issues. Please let us know if you are experiencing any problems.

3. 2025년 11월 19일 오후 6:17 UTCresolved

   This incident has been resolved.

1. 2025년 11월 12일 오후 3:28 UTCidentified

   We’re upgrading our API connection and that may impact delivery of some alerts to Workbench. We are actively investigating and collaborating on a solution to ensure the consistent and complete delivery of all alert data. Please reach out to Orca if you’d like additional information. Thank you for your patience and understanding as we work through this.

2. 2026년 4월 17일 오후 1:52 UTCresolved

   This incident has been resolved.