Panther Labs 장애 기록

1. 2026년 9월 4일 AM 8:24 UTCinvestigating

   Snowflake는 현재 인프라 일부에 영향을 미치는 정전을 경험하고있다 (https://status.snowflake.com/incidents/fl9tkpj8d7hk). 결과적으로 일부 Panther 고객은 인스턴스에 쿼리 타임 아웃 또는 느린 성능을 경험할 수 있습니다. 실시간 탐지, 경고 및 ingestion은 unaffected되어야 합니다. 우리는 적극적으로 상황을 모니터링하고 있습니다. 우리는 더 많은 정보가 있기 때문에이 페이지를 업데이트합니다. 당신의 patience를 위한 감사합니다.

2. 2026년 9월 4일 AM 9:57 UTCmonitoring

   Snowflake는 수정 및 서비스가 복원되었습니다. (https://status.snowflake.com/incidents/fl9tkpj8d7hk)에 대해 모든 서비스를 보장하기 위해 환경을 모니터링하는 것이 계속됩니다. 당신의 patience를 위한 감사합니다.

3. 2026년 9월 4일 PM 12:46 UTCresolved

   Snowflake는 문제가 해결되고 서비스가 복원되었는지 확인했습니다. Panther 서비스는 일반적으로 운영되고 환경을 모니터링하는 것을 확인했습니다.

공식 인시던트 업데이트를 자동 번역했습니다.

1. 2026년 8월 3일 PM 3:20 UTCinvestigating

   Snowflake는 현재 인프라의 일부에 영향을 미치는 정전을 경험하고있다 (https://status.snowflake.com/incidents/xfv4kdj30rcb). 결과적으로 일부 Panther 고객은 인스턴스에 쿼리 타임 아웃 또는 느린 성능을 경험할 수 있습니다. 실시간 탐지 및 경고는 부정적이어야 합니다. 우리는 적극적으로 상황을 모니터링하고 있습니다. 우리는 더 많은 정보가 있기 때문에이 페이지를 업데이트합니다. 당신의 patience를 위한 감사합니다.

2. 2026년 8월 3일 PM 8:14 UTCmonitoring

   수정이 구현되어 결과를 모니터링하고 있습니다.

3. 2026년 8월 3일 PM 8:18 UTCmonitoring

   Snowflake는 underlying 사건이 해결되고 서비스는 정상으로 돌아 왔습니다. 이 시간에, 우리는 일반적으로 운영하기 위하여 Panther 서비스를 기대합니다. 예기치 않은 행동이나 문제가 계속되면 지원 팀에 연락하여 알려주십시오. 우리는이 사건을 모니터링하면서 인내심을 주셔서 감사합니다.

4. 2026년 8월 4일 AM 6:07 UTCresolved

   This incident has been resolved.

공식 인시던트 업데이트를 자동 번역했습니다.

1. 2025년 10월 20일 PM 6:55 UTCidentified

   AWS is continuing to experience service disruptions in the us-east-1 region, which may still cause errors or delays when accessing or using Panther. We’ve confirmed that data ingestion and alert processing are not impacted; events are being queued, and will continue to process downstream as services stabilize. However, it may take some time for ingestion pipelines to fully catch up. You can follow AWS’s progress directly on their Service Health Dashboard: https://health.aws.amazon.com/health/status We’ll continue to post updates here as we confirm full recovery and return to normal operation. Thank you for your patience while we monitor and validate system stability.

2. 2025년 10월 21일 AM 12:46 UTCmonitoring

   Panther services experienced temporary degradation due to a regional AWS outage in us-east-1, and have now returned to normal operation. During the incident, some users saw intermittent authentication errors and delays in data ingestion, enrichment, detection processing, and alert generation. Events are being queued, and will continue to process downstream as services stabilize. However, it may take some time for ingestion pipelines to fully catch up. Here is the current summary of how each Panther component was impacted and is actively being remediated: Data ingestion and Enrichment: - HTTP Sources: If you received 5xx errors when attempting to send data to a Panther HTTP source, please send the data again, as it was not properly ingested nor stored in a queue. - Data is currently being recovered from the Dead-Letter Queue (DLQ). We expect the data to continue flowing in over the next 24 hours as failed jobs are requeued. Detections: - Detection processing may be delayed. Panther is in the process of requeuing all events that had issues being run through associated detections. Alerting: - Alerts were generated as detections were processed, which could have been delayed. - Delivery of alerts to destinations may have been delayed. - If a destination itself was down, Panther retried a number of times to deliver the alert, then marked delivery as failed in the Console and generated a System Error. You can review AWS’s incident details on the AWS Service Health Dashboard: https://health.aws.amazon.com/health/status We’ll continue to post updates here as we confirm full recovery and return to normal operation. Thank you for your patience while we monitor and validate system stability.

3. 2025년 10월 21일 PM 5:39 UTCresolved

   Panther services have fully recovered from the regional AWS outage in us-east-1, and all systems are now operating normally. During the incident, some customers experienced intermittent authentication errors and delays in data ingestion, enrichment, detection processing, and alert generation. All impacted components have since stabilized, and ingestion pipelines have caught up. No further action is required from customers.

1. 2025년 7월 25일 PM 8:07 UTCinvestigating

   We are currently investigating an issue where some JSON log files are being written in a malformed state. This may impact log ingestion and downstream detections and searches that rely on these files. We will update this incident as our investigation and remediation process continues.

2. 2025년 7월 25일 PM 9:02 UTCidentified

   We have identified the root cause of this malformed JSON log issue and are deploying a fix across customer accounts now. We will follow up as soon as we confirm that the issue has been resolved.

3. 2025년 7월 25일 PM 10:40 UTCresolved

   We have deployed a fix for the issue that was causing JSON log files to be written in a malformed format, and have confirmed that everything is operating as expected. The issue was active from approximately 1:30 PM ET until the fix was fully deployed to all customers around 6:30 PM ET. During this time, malformed log files were not ingested, which may have temporarily impacted downstream detections or processing. We believe the affected data can be safely reingested and are currently evaluating the best approach. We will follow up with an additional update once the reingestion plan is finalized. Thank you for your patience as we work through this.

4. 2025년 7월 28일 PM 2:41 UTCpostmortem

   We’d like to share a follow-up on the data ingestion issue that occurred on July 25, 2025 between 1:30 and 6:30pm ET. The root cause was an update to [field discovery](https://docs.panther.com/data-onboarding/field-discovery) functionality that was incorrectly deployed before end-to-end testing was complete. This resulted in certain JSON files being malformed, meaning they could not be properly ingested into Panther. Not all Panther customers were affected. The impact was limited to data: * Ingested through an S3 bucket * Parsed by a schema that has field discovery enabled * This includes custom schemas, as well as Panther-managed GitHub schemas. Field discovery was enabled for managed GitHub schemas for most Panther customers. No data was lost, and we are currently working to reprocess affected files. **We will follow-up directly with affected customers when reprocessing is complete, which we expect to be tomorrow, July 29**. We apologize for the inconvenience caused by this issue and thank you for your patience throughout remediation.

1. 2025년 5월 13일 PM 8:55 UTCidentified

   Panther has identified an issue with the AWS.CloudTrail schema that is causing a small percentage of CloudTrail logs to raise classification errors and fail to ingest into Panther. We have identified the root cause of the issue and are in the process of deploying a fix. We will update this incident as our remediation process continues.

2. 2025년 5월 13일 PM 10:59 UTCresolved

   We have reverted the changes to the AWS.CloudTrail schema that were causing classification errors. At this time, you should not see any additional errors related to this issue. If you’d like to view the classification errors in your Panther Console, in Search, select the Monitor database and Classification Failures table. If you are interested in re-ingesting the logs that failed to ingest, please reach out to the Panther support team.

1. 2025년 3월 19일 PM 8:13 UTCmonitoring

   Snowflake Data Cloud is recovering from a recent incident. Customers in specific regions may experience intermittent delays or timeouts when performing queries or using Snowflake services. You may be affected if your Panther instance is in the following regions: - AWS US-East-1 (N. Virginia) - Affected Snowflake services: Snowflake Data Warehouse (Database), Snowpipe (Data Ingestion), Replication, Snowsight - Potentially affected Panther services: Data ingestion, lookup tables, scheduled queries and rules, ad-hoc queries - AWS US-West-2 (Oregon) - Affected Snowflake services: Snowflake Data Warehouse (Database), Snowsight - Potentially affected Panther services: Lookup tables, scheduled queries and rules, ad-hoc queries For real-time updates on this incident, visit: https://status.snowflake.com/. We will continue to monitor the situation and will update the status page once resolved.

2. 2025년 3월 19일 PM 10:58 UTCresolved

   Snowflake has implemented a fix and the incident has been resolved. Your Snowflake and Panther services should now function as expected. If you experience any other issues related to this incident, please contact Panther support.

1. 2025년 1월 17일 PM 10:31 UTCidentified

   Panther has identified an issue impacting alert generation (and delivery) for detections with an events threshold value greater than one. We are currently testing a potential fix, and will update the status page once we confirm the solution is effective.

2. 2025년 1월 18일 AM 1:37 UTCmonitoring

   We have identified the root cause of the issue and deployed a fix. Our team is currently monitoring the situation to ensure that the issue has been fully resolved. Once confirmed, we will officially close this incident. If you’d like to determine whether this issue impacted your account, please follow the steps outlined in the article below: https://help.panther.com/articles/9727040283-how-to-determine-impact-of-the-alert-generation-and-delivery-failures-incident

3. 2025년 1월 18일 AM 2:22 UTCresolved

   Our observations have confirmed that the fix has been successful, but we will continue to monitor to ensure there are no further issues. We sincerely apologize for the inconvenience this may have caused and appreciate your understanding as we worked to resolve the issue. If you have any questions or need further assistance, please don’t hesitate to contact our support team.

1. 2024년 12월 13일 AM 3:22 UTCidentified

   Panther has identified an issue that prevents some customers from accessing the Panther console due to an ongoing AWS incident impacting customers in the US-EAST-1 region. At this point, logging in to the Panther console is the only service that is impacted. No other services appear to be impacted. You can also follow the ongoing incident using AWS’s status page here: https://health.aws.amazon.com/health/status

2. 2024년 12월 13일 PM 12:37 UTCresolved

   The issue impacting access to the Panther console has been resolved following AWS’s remediation. All services are now operating normally. For more details about the AWS incident, you can visit their status page: https://health.aws.amazon.com/health/status Thank you for your patience. Please let us know if you experience any further issues.

1. 2024년 9월 23일 PM 1:42 UTCidentified

   We have identified an upstream issue with the IPinfo Enrichment Provider that has caused Panther to not receive new data from IPinfo since September 19, 2024. While IPinfo Lookup Tables remain functional, they currently contain data only up to September 19, 2024. If you leverage IPinfo data in your detections, this lapse could affect their accuracy. Our team is actively collaborating with IPinfo to resolve this issue as quickly as possible. We will provide updates as our investigation and remediation efforts progress. Thank you for your patience.

2. 2024년 9월 23일 PM 6:27 UTCresolved

   We are pleased to inform you that the issue with the IPinfo Enrichment Provider has been resolved. Panther is now successfully receiving new data from IPinfo, and we have confirmed that all data is up to date beyond September 19, 2024. The IPinfo Lookup Tables should be again fully operational with the latest available data. If you have any questions or encounter any issues, please contact Panther Support directly.

1. 2024년 7월 31일 AM 12:46 UTCidentified

   Panther has confirmed that customers with deployments in the AWS US-EAST-1 region may experience issues logging into the Panther Console, log processing and alert delivery delays, as well as other issues with the service due to the ongoing AWS outage that began at approximately Jul 30 3:40 PM PDT (https://health.aws.amazon.com/health/status). We will update this incident with more information as it becomes available.

2. 2024년 7월 31일 AM 1:25 UTCidentified

   Panther has confirmed that customers with deployments in the AWS US-EAST-1 region are experiencing issues logging into the Panther Console and ingesting logs via the HTTP ingest log source type. This is a result of the degradation of the AWS Kinesis service, and ~50 other AWS services that rely on Kinesis, in Panther's case in particular the API Gateway service. Other Panther services such as ingestion via other mechanisms (S3 and other data transports as well as SaaS log source pulling), detections, alerting, data lake ingestion, and search are working as intended although we're continuing to monitor in case of further AWS service degradation. The primary long term impact to Panther's end users is the loss of some data currently being sent to HTTP Ingest log sources. This data will not be recoverable by Panther. We encourage you to retain copies of that data if possible so you can resend it after service has resumed. Once AWS has confirmed service has resumed, we will notify impacted customers of the time frame during which data ingestion was impacted. You can follow AWS updates here: https://health.aws.amazon.com/health/status

3. 2024년 7월 31일 AM 4:41 UTCmonitoring

   AWS has reported that their services have fully recovered, and we have confirmed that there are no longer any issues logging into the Panther Console and all HTTP log sources should be healthy. We will continue to monitor our systems to ensure that there is no further impact to Panther customers. If you have a deployment in the AWS US-EAST-1 region and have an HTTP log source, we recommend checking your log source(s) from Jul 30 at 3:00 PM until Jul 30 at 9:32 PM PDT to ensure that there are no missing logs. Due to the nature of the HTTP data transport, Panther is not able to keep a record of which logs failed to be received. Customers can identify this by searching through the local logs generated by the system sending logs to Panther via HTTP and finding any HTTP 500 errors returned when attempting to deliver logs to Panther. You can resend these logs to Panther and Panther will correctly backdate them in the Security data lake. We will continue to monitor the situation and will provide updates if necessary.

4. 2024년 7월 31일 PM 3:00 UTCresolved

   The AWS issue has been resolved and our monitoring confirms that there is no further impact to Panther customers. As a reminder, If you have a deployment in the AWS US-EAST-1 region and have an HTTP log source, we recommend checking your log source(s) from Jul 30 at 3:00 PM until Jul 30 at 9:32 PM PDT to ensure that there are no missing logs. If you have any questions, please contact Panther Support.

1. 2024년 4월 25일 PM 8:43 UTCidentified

   Panther has identified an issue resulting in failures in the following areas: - Lookup table updates - Log source onboarding - Scheduled query runs - Cloud security monitoring You may receive alerts related to this issue. We will update this incident as our investigation and remediation process continues.

2. 2024년 4월 25일 PM 9:35 UTCidentified

   We have identified the root cause of the issue and have begun deploying a fix internally to verify that it resolves the problem. In the meantime, please refrain from onboarding new log sources until we deploy the fix and resolve this incident. We will provide an update again once we confirm the fix and deploy it globally.

3. 2024년 4월 26일 AM 2:27 UTCresolved

   After confirming the fix internally, it has been deployed globally and there should no longer be any issues. Please note that if you onboarded a new log source after 18:30 UTC on April 25, logs will be missing from the time of onboarding until now. We will continue to monitor to ensure there are no further problems.

1. 2024년 4월 19일 PM 2:48 UTCidentified

   Panther has identified an issue affecting data ingestion filters. If you have configured log filtering within Panther, you may be impacted. We are currently working on a fix for the issue. There will not be any costs associated with any additional ingest from the log filter failure. We will update this incident as our investigation and remediation process continues.

2. 2024년 4월 19일 PM 3:51 UTCidentified

   We are verifying a fix that will automatically reapply filters you previously configured, which were removed due to this incident. Once the fix has been applied, we will update the status page.

3. 2024년 4월 19일 PM 7:25 UTCresolved

   We have deployed a fix, and the previously configured raw event filters should be working again. (This issue did not affect normalized event filters.) We will continue to monitor the issue to ensure there are no further problems. Once again, you will not be charged for the additional ingest caused by this incident. If you are still experiencing issues, please submit a ticket to the support team for assistance.

1. 2023년 12월 14일 PM 5:09 UTCinvestigating

   Panther has identified an issue with detection packs, which can prevent customers from running tests properly or uploading detections. Customers may also see intermittent errors related to missing modules. This issue impacts any customer who has updated packs to the latest pack version. For any customers who have not yet updated, we recommend waiting to do so until this issue is resolved. The currently impacted versions are 3.30, 3.29, and 3.28. We will update this incident as our investigation and remediation process continues.

2. 2023년 12월 14일 PM 5:39 UTCidentified

   A fix has been identified for a new version (3.31), and customers who are comfortable waiting can hold for that availability. Otherwise, downgrading to version 3.27 can solve this issue for customers who need a resolution more immediately. Console users will need to do this on a per-pack basis, and PAT users can revert to a commit on 3.27 (or earlier) and reupload their detections. Console users should not update packs at this time, and PAT users should avoid using Panther Analysis 3.28, 3.29, or 3.30. We will update this incident when more information is available.

3. 2023년 12월 14일 PM 9:12 UTCmonitoring

   A new version of Panther Analysis(v3.31), which includes a fix for the issue with detection packs, will be available within the next 24 hours. Once released, customers can update to that new version at their convenience. CI/CD users may continue to experience problems uploading the current version of panther-analysis, in which case deleting the following rule should resolve any errors: netskope_admin_user_change.yml. This was already removed in panther-analysis, but may have remained depending on how customers merged from upstream. We will continue to monitor the problem as v3.31 is released. In the meantime, if you experience any trouble, please get in touch with Panther Support directly.

4. 2023년 12월 19일 AM 11:19 UTCresolved

   Panther Analysis v3.31 has been released, so the CI/CD users should update to this new version at their earliest convenience. For the users using the Panther Console, please navigate to Build -> Packs in your Panther Console and upgrade your Packs to v3.31. If a CI/CD user continues to experience problems uploading the current version of panther-analysis, deleting the following rule should resolve any errors: netskope_admin_user_change.yml. This rule has already been removed in panther-analysis, but it might still exist depending on how customers merged changes from upstream. If you need additional assistance or encounter any issues, please contact Panther Support directly.

1. 2023년 11월 29일 PM 8:40 UTCinvestigating

   Panther has identified an issue that may impact search, data-explorer, and scheduled queries. Customers could encounter intermittent errors when using any of these components. We are actively investigating. We will update this incident as our investigation and remediation process continues.

2. 2023년 11월 29일 PM 10:14 UTCidentified

   We have identified a potential root cause of the issue as a change affecting version 1.91 of Panther. We are going to be reverting the change and will monitor to see if the issue has been resolved. We will continue to update you as we confirm the validity of the fix.

3. 2023년 11월 30일 AM 1:30 UTCresolved

   Our team has successfully implemented a patch to address the issue impacting search, the data explorer, and scheduled queries. Our monitoring has confirmed that the patch has been successful, but we will continue to check to ensure there are no further issues.

1. 2023년 10월 11일 AM 10:17 UTCinvestigating

   Panther has identified an issue with the ingestion of "IntegrationLogs" and "AccessLogs" log types for the native Slack integrations, specifically affecting the Slack Standard and Plus plans. Our engineering team is currently investigating this behavior. We will provide updates on this incident as our investigation and remediation process progresses.

2. 2023년 10월 11일 PM 12:41 UTCidentified

   We have identified the root cause of the issue and begun deploying a fix to resolve this problem. We will update this incident again when the issues are fully resolved.

3. 2023년 10월 11일 PM 7:14 UTCmonitoring

   The fix for this issue has been deployed to all impacted accounts and our data indicates that customers are no longer encountering the errors associated with this issue. We will continue to monitor it to ensure there are no further issues. Slack logs that previously failed to ingest due to this issue are automatically being re-ingested to Panther and no action is required by customers.

4. 2023년 10월 11일 PM 10:03 UTCresolved

   This incident has been resolved.

1. 2023년 8월 29일 PM 2:01 UTCinvestigating

   Panther has identified an issue with our IPinfo Privacy data lake enrichment that is preventing the table from being updated with the latest data. Our engineering team is actively investigating the issue, and we will post an update shortly. Customers utilizing IPinfo enrichment may encounter a “lookup update failed for ipinfo_privacy_datalake” system error as a result of this issue. Apart from the failure to update, IPinfo tables are functioning as expected at this time. Other core Panther functionalities, including the detection engine and log ingestion, are unaffected.

2. 2023년 8월 29일 PM 4:12 UTCidentified

   After investigation, Panther has identified that IPinfo is sending over improperly formatted data, causing problems with our IPinfo Privacy data lake enrichment. We have engaged with IPinfo and will provide an update as soon as more information is available.

3. 2023년 8월 30일 AM 12:01 UTCresolved

   IPinfo has pushed a fix to resolve the problem, and we have confirmed that there are no more issues with our IPinfo Privacy data lake enrichment. We will continue to monitor but expect no further issues.

1. 2023년 6월 13일 PM 8:07 UTCidentified

   Panther has confirmed that customers with deployments in the AWS US-EAST-1 region may experience issues logging into the Panther Console, log processing and alert delivery delays, as well as other issues with the service due to the ongoing AWS outage that began at approximately 12:08 PM PDT today (https://health.aws.amazon.com/health/status). We will update this incident with more information as it becomes available.

2. 2023년 6월 13일 PM 10:00 UTCmonitoring

   AWS has reported a resolution to this outage. We will monitor the results to confirm that there is no further impact to Panther customers.

3. 2023년 6월 13일 PM 11:43 UTCresolved

   Our monitoring confirms that there is no further impact to Panther customers. Data that was delayed by this issue will be backfilled automatically.

1. 2023년 5월 31일 AM 9:12 UTCinvestigating

   Panther has identified an issue that is preventing the update of the IP geolocation enrichment data. Our engineering team is actively investigating the issue and we will post an update shortly. Please note that the rest of the IPinfo tables are functioning as expected at this time and the rest of the ingestion processes are not affected. The incident is still ongoing. Please disregard the status update provided in the previous announcement.

2. 2023년 5월 31일 AM 10:52 UTCidentified

   We have identified the potential root cause of the issue and are already working on a fix. We will update this incident again when more information becomes available.

3. 2023년 5월 31일 PM 11:07 UTCmonitoring

   Our team has deployed a patch containing the fix to resolve the issue with IP geolocation enrichment data. We will continue to monitor the fix to ensure that there are no further issues. No data was lost as a result of this issue, and no action is required from Panther customers.

4. 2023년 6월 2일 PM 5:14 UTCresolved

   Our monitoring indicates that the fix is working as intended and the issue is now resolved.

1. 2023년 2월 14일 PM 9:11 UTCidentified

   Panther has identified an issue with our Tor Exit Node Lookup Table that is causing the Lookup Table to fail to update. This will prevent some users from uploading new or updated detections via panther_analysis_tool or via the Bulk Uploader in the Panther Console. We have developed a fix for this issue and are currently testing it. We will update this incident as our remediation process continues. Other components of Panther, including real-time detection and alert delivery, are functioning normally at this time.

2. 2023년 2월 15일 AM 1:41 UTCidentified

   Our team has completed testing the fix for this issue. To ensure that the deployment goes smoothly and without errors, we will begin deploying the fix to customers tomorrow morning during US Eastern hours. We will provide another update when we have additional information to share.

3. 2023년 2월 15일 PM 3:46 UTCresolved

   Our team has deployed the patch containing the fix for this issue to all customer accounts. Our tests indicate that the issue has been resolved and customers are no longer affected. We will continue to monitor for any potential customer impact over the next several hours.

1. 2022년 11월 23일 PM 5:54 UTCidentified

   Panther has confirmed a Snowflake outage in the AWS us-east-1 region (https://status.snowflake.com/incidents/yh1n0ly69chm). For Panther customers hosted in us-east-1, queries in Data Explorer, scheduled queries, and viewing events that triggered alerts are potentially affected. Logs may also ingest into Snowflake with a delay. We will continue to update this incident as more information becomes available.

2. 2022년 11월 23일 PM 11:22 UTCmonitoring

   Snowflake has implemented a fix for the issue in the AWS us-east-1 region. We will continue to monitor the situation until we can confirm it is resolved, or provide more information when possible.

3. 2022년 11월 24일 AM 12:27 UTCresolved

   Panther has confirmed that the issue in the AWS us-east-1 region has been resolved. Queries in Data Explorer, scheduled queries, viewing events that triggered alerts, and log ingestion into Snowflake should now function as expected. If you experience any other issues related to this incident, please contact Panther support.