Runpanther 장애 기록

AWS is continuing to experience service disruptions in the us-east-1 region, which may still cause errors or delays when accessing or using Panther. We’ve confirmed that data ingestion and alert processing are not impacted; events are being queued, and will continue to process downstream as services stabilize. However, it may take some time for ingestion pipelines to fully catch up. You can follow AWS’s progress directly on their Service Health Dashboard: https://health.aws.amazon.com/health/status We’ll continue to post updates here as we confirm full recovery and return to normal operation. Thank you for your patience while we monitor and validate system stability.

Panther services experienced temporary degradation due to a regional AWS outage in us-east-1, and have now returned to normal operation. During the incident, some users saw intermittent authentication errors and delays in data ingestion, enrichment, detection processing, and alert generation. Events are being queued, and will continue to process downstream as services stabilize. However, it may take some time for ingestion pipelines to fully catch up. Here is the current summary of how each Panther component was impacted and is actively being remediated: Data ingestion and Enrichment: - HTTP Sources: If you received 5xx errors when attempting to send data to a Panther HTTP source, please send the data again, as it was not properly ingested nor stored in a queue. - Data is currently being recovered from the Dead-Letter Queue (DLQ). We expect the data to continue flowing in over the next 24 hours as failed jobs are requeued. Detections: - Detection processing may be delayed. Panther is in the process of requeuing all events that had issues being run through associated detections. Alerting: - Alerts were generated as detections were processed, which could have been delayed. - Delivery of alerts to destinations may have been delayed. - If a destination itself was down, Panther retried a number of times to deliver the alert, then marked delivery as failed in the Console and generated a System Error. You can review AWS’s incident details on the AWS Service Health Dashboard: https://health.aws.amazon.com/health/status We’ll continue to post updates here as we confirm full recovery and return to normal operation. Thank you for your patience while we monitor and validate system stability.

Panther services have fully recovered from the regional AWS outage in us-east-1, and all systems are now operating normally. During the incident, some customers experienced intermittent authentication errors and delays in data ingestion, enrichment, detection processing, and alert generation. All impacted components have since stabilized, and ingestion pipelines have caught up. No further action is required from customers.

We are currently investigating an issue where some JSON log files are being written in a malformed state. This may impact log ingestion and downstream detections and searches that rely on these files. We will update this incident as our investigation and remediation process continues.

We have identified the root cause of this malformed JSON log issue and are deploying a fix across customer accounts now. We will follow up as soon as we confirm that the issue has been resolved.

We have deployed a fix for the issue that was causing JSON log files to be written in a malformed format, and have confirmed that everything is operating as expected. The issue was active from approximately 1:30 PM ET until the fix was fully deployed to all customers around 6:30 PM ET. During this time, malformed log files were not ingested, which may have temporarily impacted downstream detections or processing. We believe the affected data can be safely reingested and are currently evaluating the best approach. We will follow up with an additional update once the reingestion plan is finalized. Thank you for your patience as we work through this.

We’d like to share a follow-up on the data ingestion issue that occurred on July 25, 2025 between 1:30 and 6:30pm ET. The root cause was an update to [field discovery](https://docs.panther.com/data-onboarding/field-discovery) functionality that was incorrectly deployed before end-to-end testing was complete. This resulted in certain JSON files being malformed, meaning they could not be properly ingested into Panther. Not all Panther customers were affected. The impact was limited to data: * Ingested through an S3 bucket * Parsed by a schema that has field discovery enabled * This includes custom schemas, as well as Panther-managed GitHub schemas. Field discovery was enabled for managed GitHub schemas for most Panther customers. No data was lost, and we are currently working to reprocess affected files. **We will follow-up directly with affected customers when reprocessing is complete, which we expect to be tomorrow, July 29**. We apologize for the inconvenience caused by this issue and thank you for your patience throughout remediation.

Panther has identified an issue with the AWS.CloudTrail schema that is causing a small percentage of CloudTrail logs to raise classification errors and fail to ingest into Panther. We have identified the root cause of the issue and are in the process of deploying a fix. We will update this incident as our remediation process continues.

We have reverted the changes to the AWS.CloudTrail schema that were causing classification errors. At this time, you should not see any additional errors related to this issue. If you’d like to view the classification errors in your Panther Console, in Search, select the Monitor database and Classification Failures table. If you are interested in re-ingesting the logs that failed to ingest, please reach out to the Panther support team.

Snowflake Data Cloud is recovering from a recent incident. Customers in specific regions may experience intermittent delays or timeouts when performing queries or using Snowflake services. You may be affected if your Panther instance is in the following regions: - AWS US-East-1 (N. Virginia) - Affected Snowflake services: Snowflake Data Warehouse (Database), Snowpipe (Data Ingestion), Replication, Snowsight - Potentially affected Panther services: Data ingestion, lookup tables, scheduled queries and rules, ad-hoc queries - AWS US-West-2 (Oregon) - Affected Snowflake services: Snowflake Data Warehouse (Database), Snowsight - Potentially affected Panther services: Lookup tables, scheduled queries and rules, ad-hoc queries For real-time updates on this incident, visit: https://status.snowflake.com/. We will continue to monitor the situation and will update the status page once resolved.

Snowflake has implemented a fix and the incident has been resolved. Your Snowflake and Panther services should now function as expected. If you experience any other issues related to this incident, please contact Panther support.

Panther has identified an issue impacting alert generation (and delivery) for detections with an events threshold value greater than one. We are currently testing a potential fix, and will update the status page once we confirm the solution is effective.

We have identified the root cause of the issue and deployed a fix. Our team is currently monitoring the situation to ensure that the issue has been fully resolved. Once confirmed, we will officially close this incident. If you’d like to determine whether this issue impacted your account, please follow the steps outlined in the article below: https://help.panther.com/articles/9727040283-how-to-determine-impact-of-the-alert-generation-and-delivery-failures-incident

Our observations have confirmed that the fix has been successful, but we will continue to monitor to ensure there are no further issues. We sincerely apologize for the inconvenience this may have caused and appreciate your understanding as we worked to resolve the issue. If you have any questions or need further assistance, please don’t hesitate to contact our support team.

Panther has identified an issue that prevents some customers from accessing the Panther console due to an ongoing AWS incident impacting customers in the US-EAST-1 region. At this point, logging in to the Panther console is the only service that is impacted. No other services appear to be impacted. You can also follow the ongoing incident using AWS’s status page here: https://health.aws.amazon.com/health/status

The issue impacting access to the Panther console has been resolved following AWS’s remediation. All services are now operating normally. For more details about the AWS incident, you can visit their status page: https://health.aws.amazon.com/health/status Thank you for your patience. Please let us know if you experience any further issues.

We have identified an upstream issue with the IPinfo Enrichment Provider that has caused Panther to not receive new data from IPinfo since September 19, 2024. While IPinfo Lookup Tables remain functional, they currently contain data only up to September 19, 2024. If you leverage IPinfo data in your detections, this lapse could affect their accuracy. Our team is actively collaborating with IPinfo to resolve this issue as quickly as possible. We will provide updates as our investigation and remediation efforts progress. Thank you for your patience.

We are pleased to inform you that the issue with the IPinfo Enrichment Provider has been resolved. Panther is now successfully receiving new data from IPinfo, and we have confirmed that all data is up to date beyond September 19, 2024. The IPinfo Lookup Tables should be again fully operational with the latest available data. If you have any questions or encounter any issues, please contact Panther Support directly.

Panther has confirmed that customers with deployments in the AWS US-EAST-1 region may experience issues logging into the Panther Console, log processing and alert delivery delays, as well as other issues with the service due to the ongoing AWS outage that began at approximately Jul 30 3:40 PM PDT (https://health.aws.amazon.com/health/status). We will update this incident with more information as it becomes available.

Panther has confirmed that customers with deployments in the AWS US-EAST-1 region are experiencing issues logging into the Panther Console and ingesting logs via the HTTP ingest log source type. This is a result of the degradation of the AWS Kinesis service, and ~50 other AWS services that rely on Kinesis, in Panther's case in particular the API Gateway service. Other Panther services such as ingestion via other mechanisms (S3 and other data transports as well as SaaS log source pulling), detections, alerting, data lake ingestion, and search are working as intended although we're continuing to monitor in case of further AWS service degradation. The primary long term impact to Panther's end users is the loss of some data currently being sent to HTTP Ingest log sources. This data will not be recoverable by Panther. We encourage you to retain copies of that data if possible so you can resend it after service has resumed. Once AWS has confirmed service has resumed, we will notify impacted customers of the time frame during which data ingestion was impacted. You can follow AWS updates here: https://health.aws.amazon.com/health/status

AWS has reported that their services have fully recovered, and we have confirmed that there are no longer any issues logging into the Panther Console and all HTTP log sources should be healthy. We will continue to monitor our systems to ensure that there is no further impact to Panther customers. If you have a deployment in the AWS US-EAST-1 region and have an HTTP log source, we recommend checking your log source(s) from Jul 30 at 3:00 PM until Jul 30 at 9:32 PM PDT to ensure that there are no missing logs. Due to the nature of the HTTP data transport, Panther is not able to keep a record of which logs failed to be received. Customers can identify this by searching through the local logs generated by the system sending logs to Panther via HTTP and finding any HTTP 500 errors returned when attempting to deliver logs to Panther. You can resend these logs to Panther and Panther will correctly backdate them in the Security data lake. We will continue to monitor the situation and will provide updates if necessary.

The AWS issue has been resolved and our monitoring confirms that there is no further impact to Panther customers. As a reminder, If you have a deployment in the AWS US-EAST-1 region and have an HTTP log source, we recommend checking your log source(s) from Jul 30 at 3:00 PM until Jul 30 at 9:32 PM PDT to ensure that there are no missing logs. If you have any questions, please contact Panther Support.

Panther has identified an issue resulting in failures in the following areas: - Lookup table updates - Log source onboarding - Scheduled query runs - Cloud security monitoring You may receive alerts related to this issue. We will update this incident as our investigation and remediation process continues.

We have identified the root cause of the issue and have begun deploying a fix internally to verify that it resolves the problem. In the meantime, please refrain from onboarding new log sources until we deploy the fix and resolve this incident. We will provide an update again once we confirm the fix and deploy it globally.

After confirming the fix internally, it has been deployed globally and there should no longer be any issues. Please note that if you onboarded a new log source after 18:30 UTC on April 25, logs will be missing from the time of onboarding until now. We will continue to monitor to ensure there are no further problems.

Panther has identified an issue affecting data ingestion filters. If you have configured log filtering within Panther, you may be impacted. We are currently working on a fix for the issue. There will not be any costs associated with any additional ingest from the log filter failure. We will update this incident as our investigation and remediation process continues.

We are verifying a fix that will automatically reapply filters you previously configured, which were removed due to this incident. Once the fix has been applied, we will update the status page.

We have deployed a fix, and the previously configured raw event filters should be working again. (This issue did not affect normalized event filters.) We will continue to monitor the issue to ensure there are no further problems. Once again, you will not be charged for the additional ingest caused by this incident. If you are still experiencing issues, please submit a ticket to the support team for assistance.

Panther has identified an issue with detection packs, which can prevent customers from running tests properly or uploading detections. Customers may also see intermittent errors related to missing modules. This issue impacts any customer who has updated packs to the latest pack version. For any customers who have not yet updated, we recommend waiting to do so until this issue is resolved. The currently impacted versions are 3.30, 3.29, and 3.28. We will update this incident as our investigation and remediation process continues.

A fix has been identified for a new version (3.31), and customers who are comfortable waiting can hold for that availability. Otherwise, downgrading to version 3.27 can solve this issue for customers who need a resolution more immediately. Console users will need to do this on a per-pack basis, and PAT users can revert to a commit on 3.27 (or earlier) and reupload their detections. Console users should not update packs at this time, and PAT users should avoid using Panther Analysis 3.28, 3.29, or 3.30. We will update this incident when more information is available.

A new version of Panther Analysis(v3.31), which includes a fix for the issue with detection packs, will be available within the next 24 hours. Once released, customers can update to that new version at their convenience. CI/CD users may continue to experience problems uploading the current version of panther-analysis, in which case deleting the following rule should resolve any errors: netskope_admin_user_change.yml. This was already removed in panther-analysis, but may have remained depending on how customers merged from upstream. We will continue to monitor the problem as v3.31 is released. In the meantime, if you experience any trouble, please get in touch with Panther Support directly.

Panther Analysis v3.31 has been released, so the CI/CD users should update to this new version at their earliest convenience. For the users using the Panther Console, please navigate to Build -> Packs in your Panther Console and upgrade your Packs to v3.31. If a CI/CD user continues to experience problems uploading the current version of panther-analysis, deleting the following rule should resolve any errors: netskope_admin_user_change.yml. This rule has already been removed in panther-analysis, but it might still exist depending on how customers merged changes from upstream. If you need additional assistance or encounter any issues, please contact Panther Support directly.

Panther has identified an issue that may impact search, data-explorer, and scheduled queries. Customers could encounter intermittent errors when using any of these components. We are actively investigating. We will update this incident as our investigation and remediation process continues.

We have identified a potential root cause of the issue as a change affecting version 1.91 of Panther. We are going to be reverting the change and will monitor to see if the issue has been resolved. We will continue to update you as we confirm the validity of the fix.

Our team has successfully implemented a patch to address the issue impacting search, the data explorer, and scheduled queries. Our monitoring has confirmed that the patch has been successful, but we will continue to check to ensure there are no further issues.

Panther has identified an issue with the ingestion of "IntegrationLogs" and "AccessLogs" log types for the native Slack integrations, specifically affecting the Slack Standard and Plus plans. Our engineering team is currently investigating this behavior. We will provide updates on this incident as our investigation and remediation process progresses.

We have identified the root cause of the issue and begun deploying a fix to resolve this problem. We will update this incident again when the issues are fully resolved.

The fix for this issue has been deployed to all impacted accounts and our data indicates that customers are no longer encountering the errors associated with this issue. We will continue to monitor it to ensure there are no further issues. Slack logs that previously failed to ingest due to this issue are automatically being re-ingested to Panther and no action is required by customers.

This incident has been resolved.

Panther has identified an issue with our IPinfo Privacy data lake enrichment that is preventing the table from being updated with the latest data. Our engineering team is actively investigating the issue, and we will post an update shortly. Customers utilizing IPinfo enrichment may encounter a “lookup update failed for ipinfo_privacy_datalake” system error as a result of this issue. Apart from the failure to update, IPinfo tables are functioning as expected at this time. Other core Panther functionalities, including the detection engine and log ingestion, are unaffected.

After investigation, Panther has identified that IPinfo is sending over improperly formatted data, causing problems with our IPinfo Privacy data lake enrichment. We have engaged with IPinfo and will provide an update as soon as more information is available.

IPinfo has pushed a fix to resolve the problem, and we have confirmed that there are no more issues with our IPinfo Privacy data lake enrichment. We will continue to monitor but expect no further issues.

Panther has confirmed that customers with deployments in the AWS US-EAST-1 region may experience issues logging into the Panther Console, log processing and alert delivery delays, as well as other issues with the service due to the ongoing AWS outage that began at approximately 12:08 PM PDT today (https://health.aws.amazon.com/health/status). We will update this incident with more information as it becomes available.

AWS has reported a resolution to this outage. We will monitor the results to confirm that there is no further impact to Panther customers.

Our monitoring confirms that there is no further impact to Panther customers. Data that was delayed by this issue will be backfilled automatically.

Panther has identified an issue that is preventing the update of the IP geolocation enrichment data. Our engineering team is actively investigating the issue and we will post an update shortly. Please note that the rest of the IPinfo tables are functioning as expected at this time and the rest of the ingestion processes are not affected. The incident is still ongoing. Please disregard the status update provided in the previous announcement.

We have identified the potential root cause of the issue and are already working on a fix. We will update this incident again when more information becomes available.

Our team has deployed a patch containing the fix to resolve the issue with IP geolocation enrichment data. We will continue to monitor the fix to ensure that there are no further issues. No data was lost as a result of this issue, and no action is required from Panther customers.

Our monitoring indicates that the fix is working as intended and the issue is now resolved.

Panther has identified an issue with our Tor Exit Node Lookup Table that is causing the Lookup Table to fail to update. This will prevent some users from uploading new or updated detections via panther_analysis_tool or via the Bulk Uploader in the Panther Console. We have developed a fix for this issue and are currently testing it. We will update this incident as our remediation process continues. Other components of Panther, including real-time detection and alert delivery, are functioning normally at this time.

Our team has completed testing the fix for this issue. To ensure that the deployment goes smoothly and without errors, we will begin deploying the fix to customers tomorrow morning during US Eastern hours. We will provide another update when we have additional information to share.

Our team has deployed the patch containing the fix for this issue to all customer accounts. Our tests indicate that the issue has been resolved and customers are no longer affected. We will continue to monitor for any potential customer impact over the next several hours.

Panther has confirmed a Snowflake outage in the AWS us-east-1 region (https://status.snowflake.com/incidents/yh1n0ly69chm). For Panther customers hosted in us-east-1, queries in Data Explorer, scheduled queries, and viewing events that triggered alerts are potentially affected. Logs may also ingest into Snowflake with a delay. We will continue to update this incident as more information becomes available.

Snowflake has implemented a fix for the issue in the AWS us-east-1 region. We will continue to monitor the situation until we can confirm it is resolved, or provide more information when possible.

Panther has confirmed that the issue in the AWS us-east-1 region has been resolved. Queries in Data Explorer, scheduled queries, viewing events that triggered alerts, and log ingestion into Snowflake should now function as expected. If you experience any other issues related to this incident, please contact Panther support.

We have received additional reports of alert delivery failures for customers using our Slack webhook integration after deployment of the patch yesterday (https://status.panther.com/incidents/v4bthbs6sbx4). Our engineering team is working on an additional patch to resolve these new issues. We will update this incident with more information shortly.

Our team has begun deploying the patch to customer accounts. We will update this incident again after the deployment is complete and our monitoring indicates that the issues have been resolved.

The patch containing the fix for this issue has been deployed to all impacted customer accounts. Alert delivery to Slack destinations using webhooks are now functioning as intended. To check for alerts that failed during this time and manually resend them, please follow the instructions below: 1. Open the Panther Console and navigate to Alerts. 2. Click the name of an alert that shows "Destination Delivery Failed". 3. Next to "Alert delivery failed", click "Show History". 4. Click the retry delivery button.

Panther has identified an issue that prevents alerts from being delivered to Slack destinations using webhooks. Our Engineering team has developed a fix for this issue and we plan to deploy it to impacted customer accounts soon. We will update this incident as our investigation and remediation process continues.

We have deployed the fix for this issue to affected customer accounts. New alerts should now be sent properly to Slack destinations, and the incident is resolved. To check for alerts that failed during this time and manually resend them, please follow the instructions below: 1. Open the Panther Console and navigate to Alerts. 2. Click the name of an alert that shows "Destination Delivery Failed". 3. Next to "Alert delivery failed", click "Show History". 4. Click the retry delivery button.