Red Canary 장애 기록

The issue has been identified and we are currently working on a fix.

A fix has been implemented and we are monitoring results.

We are continuing to monitor for any further issues.

This incident has been resolved.

Threat detection for our Government CrowdStrike customers may be delayed due to an incident with CrowdStrike’s Falcon Data Replicator. This incident will be updated as additional information becomes available from CrowdStrike.

This incident has been resolved.

We are investigating an issue impacting AWS ingestors, which is causing a delay in our processing of Cloudtrail and GuardDuty alerts.

The issue has been identified and a fix is being implemented.

This incident has been resolved.

We are currently investigating reports of intermittent detection delays. The issue has been identified and a fix has been implemented. We are monitoring the system to confirm that it is returning to normal operations.

Monitoring has confirmed the system is operating normally and detections are no longer delayed.

We are investigating delayed telemetry surrounding Google Cloud Platform data due to underlying issues in the AWS Cloud Platform. We have notified the AWS production engineering team and are monitoring the issue to ensure they resolve the issue promptly. Telemetry, events, and detections may be delayed in our Web Portal, APIs, and systems that leverage those APIs such as Canary Exporter for GCP telemetry.

We are monitoring the system to confirm that it is returning to normal operations.

This incident has been resolved.

We are investigating ingestion issues with VMware Carbon Black Response services hosted by Red Canary. Sensor telemetry, events, and detections may be delayed in our Web Portal, APIs, and systems that leverage those APIs such as Canary Exporter. New sensor connections and installations may be impacted. The issue has been identified and a fix is being implemented. We are monitoring the system to confirm that it is returning to normal operations.

This incident has been resolved.

We are currently experiencing data processing delays, resulting in late detection notification to some customers.

This incident has been resolved.

We have identified an issue that is causing intermittent delays in processing cloud and identity data. We are actively working on a fix. Detections may be delayed.

We are continuing to work on a fix for this issue.

This incident has been resolved.

We are aware of a reported degradation with the manage.office.com domain that is impacting some Microsoft Office 365 services. At this time telemetry and detections for Microsoft Office 365 External Service Integrations may be delayed.

Microsoft Office 365 external service integrations are now functioning as expected. Detections may be delayed as we process the backlog.

This incident has been resolved.

We identified an issue impacting data processing for Google Workspace customers and have implemented a fix. Detections may be delayed.

This incident has been resolved.

We are processing a large backlog of data and detections may be delayed.

This incident has been resolved.

We are aware of a reported AWS degradation that may be impacting some services. At this time, telemetry for Jamf Threat Defense and VMware Carbon Black Response, hosted by Red Canary, may experience delays. We have observed signs of recovery in the affected services and continue to monitor the situation closely.

This incident has been resolved. Services are operating as expected.

We are aware of a reported AWS degradation that may be impacting some services. At this time, telemetry for Jamf Threat Defense and VMware Carbon Black Response, hosted by Red Canary, may experience delays. We have observed signs of recovery in the affected services and continue to monitor the situation closely.

This incident has been resolved. Services are operating as expected.

We are currently investigating an issue with some pages being unavailable in the Red Canary portal. We will update this incident as more information becomes available.

The issue has been identified and a fix is being implemented.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

We have identified an issue that is causing the Red Canary web portal and API requests to return errors intermittently for some customers. We are implementing a fix and will update this incident upon resolution of the impact.

This incident has been resolved.

We have identified an issue that may be causing a degraded portal experience for our customers. Some portal pages may be slower to respond or appear missing. A fix has been identified and is currently being implemented and tested. We will provide more information as soon as possible.

Our fix has been implemented, tested, and deployed. We've validated the portal is no longer degraded and pages that may have appeared to be missing are present.

SentinelOne declared an incident earlier today that has caused delays in receiving data from their platform: https://status.sentinelone.com/incidents/b03z8dft06hs We have engaged SentinelOne to determine how large their backlogs are and when we can expect to see data flow resume. We will update this incident as we learn more. Detections based on SentinelOne data may be delayed.

This incident has been resolved. SentinelOne has fixed the issue on their end, processed their backlog, and we are receiving data again.

We are currently observing errors with Broadcom VMware Carbon Black Cloud's alert APIs. Ingestion of CBC alerts are currently failing. We have engaged CBC to determine the cause of the errors and will post more information here as we get it. Detections based on CBC alerts may be delayed.

We have determined the errors are due to a configuration change with CBC that required customers to make changes to their environments. Therefore this affects a small percentage of our customers and we are engaging with them directly to make the required changes.

We are tracking an outage with Broadcom that is affecting authentication to Carbon Black Cloud environments. Active Remediation response actions may be delayed.

We are continuing to investigate this issue.

A link to the Broadcom status can be found here: https://status.broadcom.com/notices/gnghnazmjpbilwta-carbon-black-cloud-us-region-outage

Broadcom has updated the status to Identified. We are tracking process. https://status.broadcom.com/notices/gnghnazmjpbilwta-carbon-black-cloud-us-region-outage

Broadcom has implemented a fix and we are monitoring progress. Broadcom Status Page: https://status.broadcom.com/notices/gnghnazmjpbilwta-carbon-black-cloud-us-region-outage

This incident has been resolved.

We are currently investigating an issue processing some raw telemetry. Detections may be delayed. We will update this incident as more information becomes available.

The issue has been identified and a fix is being implemented.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.