Detections (Rules and Policies)Alert DeliveryInvestigation, Data Retention, and Search (Data Lake)Panther Console (Web App)EnrichmentData Ingestion into Panther (Log Processing)
identified
AWS is continuing to experience service disruptions in the us-east-1 region, which may still cause errors or delays when accessing or using Panther.
We’ve confirmed that data ingestion and alert processing are not impacted; events are being queued, and will continue to process downstream as services stabilize. However, it may take some time for ingestion pipelines to fully catch up.
You can follow AWS’s progress directly on their Service Health Dashboard: https://health.aws.amazon.com/health/status
We’ll continue to post updates here as we confirm full recovery and return to normal operation.
Thank you for your patience while we monitor and validate system stability.
monitoring
Panther services experienced temporary degradation due to a regional AWS outage in us-east-1, and have now returned to normal operation. During the incident, some users saw intermittent authentication errors and delays in data ingestion, enrichment, detection processing, and alert generation. Events are being queued, and will continue to process downstream as services stabilize. However, it may take some time for ingestion pipelines to fully catch up.
Here is the current summary of how each Panther component was impacted and is actively being remediated:
Data ingestion and Enrichment:
- HTTP Sources: If you received 5xx errors when attempting to send data to a Panther HTTP source, please send the data again, as it was not properly ingested nor stored in a queue.
- Data is currently being recovered from the Dead-Letter Queue (DLQ). We expect the data to continue flowing in over the next 24 hours as failed jobs are requeued.
Detections:
- Detection processing may be delayed. Panther is in the process of requeuing all events that had issues being run through associated detections.
Alerting:
- Alerts were generated as detections were processed, which could have been delayed.
- Delivery of alerts to destinations may have been delayed.
- If a destination itself was down, Panther retried a number of times to deliver the alert, then marked delivery as failed in the Console and generated a System Error.
You can review AWS’s incident details on the AWS Service Health Dashboard: https://health.aws.amazon.com/health/status
We’ll continue to post updates here as we confirm full recovery and return to normal operation.
Thank you for your patience while we monitor and validate system stability.
resolved
Panther services have fully recovered from the regional AWS outage in us-east-1, and all systems are now operating normally.
During the incident, some customers experienced intermittent authentication errors and delays in data ingestion, enrichment, detection processing, and alert generation. All impacted components have since stabilized, and ingestion pipelines have caught up. No further action is required from customers.
Malformed JSON files
開始 2025年7月25日 20:07 UTC · 2h 33m
Outage重大なインシデント
影響を受けたコンポーネント
Data Ingestion into Panther (Log Processing)
investigating
We are currently investigating an issue where some JSON log files are being written in a malformed state. This may impact log ingestion and downstream detections and searches that rely on these files.
We will update this incident as our investigation and remediation process continues.
identified
We have identified the root cause of this malformed JSON log issue and are deploying a fix across customer accounts now.
We will follow up as soon as we confirm that the issue has been resolved.
resolved
We have deployed a fix for the issue that was causing JSON log files to be written in a malformed format, and have confirmed that everything is operating as expected.
The issue was active from approximately 1:30 PM ET until the fix was fully deployed to all customers around 6:30 PM ET. During this time, malformed log files were not ingested, which may have temporarily impacted downstream detections or processing.
We believe the affected data can be safely reingested and are currently evaluating the best approach. We will follow up with an additional update once the reingestion plan is finalized.
Thank you for your patience as we work through this.
postmortem
We’d like to share a follow-up on the data ingestion issue that occurred on July 25, 2025 between 1:30 and 6:30pm ET.
The root cause was an update to [field discovery](https://docs.panther.com/data-onboarding/field-discovery) functionality that was incorrectly deployed before end-to-end testing was complete. This resulted in certain JSON files being malformed, meaning they could not be properly ingested into Panther.
Not all Panther customers were affected. The impact was limited to data:
* Ingested through an S3 bucket
* Parsed by a schema that has field discovery enabled
* This includes custom schemas, as well as Panther-managed GitHub schemas. Field discovery was enabled for managed GitHub schemas for most Panther customers.
No data was lost, and we are currently working to reprocess affected files. **We will follow-up directly with affected customers when reprocessing is complete, which we expect to be tomorrow, July 29**.
We apologize for the inconvenience caused by this issue and thank you for your patience throughout remediation.
CloudTrail Classification Issues
開始 2025年5月13日 20:55 UTC · 2h 4m
Issues軽微なインシデント
影響を受けたコンポーネント
Data Ingestion into Panther (Log Processing)
identified
Panther has identified an issue with the AWS.CloudTrail schema that is causing a small percentage of CloudTrail logs to raise classification errors and fail to ingest into Panther. We have identified the root cause of the issue and are in the process of deploying a fix.
We will update this incident as our remediation process continues.
resolved
We have reverted the changes to the AWS.CloudTrail schema that were causing classification errors. At this time, you should not see any additional errors related to this issue.
If you’d like to view the classification errors in your Panther Console, in Search, select the Monitor database and Classification Failures table. If you are interested in re-ingesting the logs that failed to ingest, please reach out to the Panther support team.
Snowflake Data Cloud Incident
開始 2025年3月19日 20:13 UTC · 2h 46m
Issues軽微なインシデント
影響を受けたコンポーネント
Investigation, Data Retention, and Search (Data Lake)EnrichmentData Ingestion into Panther (Log Processing)
monitoring
Snowflake Data Cloud is recovering from a recent incident. Customers in specific regions may experience intermittent delays or timeouts when performing queries or using Snowflake services.
You may be affected if your Panther instance is in the following regions:
- AWS US-East-1 (N. Virginia)
- Affected Snowflake services: Snowflake Data Warehouse (Database), Snowpipe (Data Ingestion), Replication, Snowsight
- Potentially affected Panther services: Data ingestion, lookup tables, scheduled queries and rules, ad-hoc queries
- AWS US-West-2 (Oregon)
- Affected Snowflake services: Snowflake Data Warehouse (Database), Snowsight
- Potentially affected Panther services: Lookup tables, scheduled queries and rules, ad-hoc queries
For real-time updates on this incident, visit: https://status.snowflake.com/. We will continue to monitor the situation and will update the status page once resolved.
resolved
Snowflake has implemented a fix and the incident has been resolved. Your Snowflake and Panther services should now function as expected.
If you experience any other issues related to this incident, please contact Panther support.
Alert Generation and Delivery Failures
開始 2025年1月17日 22:31 UTC · 3h 51m
Issues軽微なインシデント
影響を受けたコンポーネント
Alert Delivery
identified
Panther has identified an issue impacting alert generation (and delivery) for detections with an events threshold value greater than one. We are currently testing a potential fix, and will update the status page once we confirm the solution is effective.
monitoring
We have identified the root cause of the issue and deployed a fix. Our team is currently monitoring the situation to ensure that the issue has been fully resolved. Once confirmed, we will officially close this incident.
If you’d like to determine whether this issue impacted your account, please follow the steps outlined in the article below:
https://help.panther.com/articles/9727040283-how-to-determine-impact-of-the-alert-generation-and-delivery-failures-incident
resolved
Our observations have confirmed that the fix has been successful, but we will continue to monitor to ensure there are no further issues.
We sincerely apologize for the inconvenience this may have caused and appreciate your understanding as we worked to resolve the issue. If you have any questions or need further assistance, please don’t hesitate to contact our support team.
Intermittent Login Failures
開始 2024年12月13日 3:22 UTC · 9h 15m
Outage致命的なインシデント
影響を受けたコンポーネント
Panther Console (Web App)
identified
Panther has identified an issue that prevents some customers from accessing the Panther console due to an ongoing AWS incident impacting customers in the US-EAST-1 region. At this point, logging in to the Panther console is the only service that is impacted. No other services appear to be impacted. You can also follow the ongoing incident using AWS’s status page here: https://health.aws.amazon.com/health/status
resolved
The issue impacting access to the Panther console has been resolved following AWS’s remediation. All services are now operating normally. For more details about the AWS incident, you can visit their status page: https://health.aws.amazon.com/health/status
Thank you for your patience. Please let us know if you experience any further issues.
IPinfo Data Update Issue
開始 2024年9月23日 13:42 UTC · 4h 46m
Issues軽微なインシデント
影響を受けたコンポーネント
Enrichment
identified
We have identified an upstream issue with the IPinfo Enrichment Provider that has caused Panther to not receive new data from IPinfo since September 19, 2024. While IPinfo Lookup Tables remain functional, they currently contain data only up to September 19, 2024. If you leverage IPinfo data in your detections, this lapse could affect their accuracy.
Our team is actively collaborating with IPinfo to resolve this issue as quickly as possible. We will provide updates as our investigation and remediation efforts progress. Thank you for your patience.
resolved
We are pleased to inform you that the issue with the IPinfo Enrichment Provider has been resolved. Panther is now successfully receiving new data from IPinfo, and we have confirmed that all data is up to date beyond September 19, 2024. The IPinfo Lookup Tables should be again fully operational with the latest available data.
If you have any questions or encounter any issues, please contact Panther Support directly.
AWS US-EAST-1 outage
開始 2024年7月31日 0:46 UTC · 14h 14m
Outage重大なインシデント
影響を受けたコンポーネント
Panther Console (Web App)Data Ingestion into Panther (Log Processing)
identified
Panther has confirmed that customers with deployments in the AWS US-EAST-1 region may experience issues logging into the Panther Console, log processing and alert delivery delays, as well as other issues with the service due to the ongoing AWS outage that began at approximately Jul 30 3:40 PM PDT (https://health.aws.amazon.com/health/status). We will update this incident with more information as it becomes available.
identified
Panther has confirmed that customers with deployments in the AWS US-EAST-1 region are experiencing issues logging into the Panther Console and ingesting logs via the HTTP ingest log source type. This is a result of the degradation of the AWS Kinesis service, and ~50 other AWS services that rely on Kinesis, in Panther's case in particular the API Gateway service.
Other Panther services such as ingestion via other mechanisms (S3 and other data transports as well as SaaS log source pulling), detections, alerting, data lake ingestion, and search are working as intended although we're continuing to monitor in case of further AWS service degradation.
The primary long term impact to Panther's end users is the loss of some data currently being sent to HTTP Ingest log sources. This data will not be recoverable by Panther. We encourage you to retain copies of that data if possible so you can resend it after service has resumed. Once AWS has confirmed service has resumed, we will notify impacted customers of the time frame during which data ingestion was impacted.
You can follow AWS updates here: https://health.aws.amazon.com/health/status
monitoring
AWS has reported that their services have fully recovered, and we have confirmed that there are no longer any issues logging into the Panther Console and all HTTP log sources should be healthy. We will continue to monitor our systems to ensure that there is no further impact to Panther customers.
If you have a deployment in the AWS US-EAST-1 region and have an HTTP log source, we recommend checking your log source(s) from Jul 30 at 3:00 PM until Jul 30 at 9:32 PM PDT to ensure that there are no missing logs.
Due to the nature of the HTTP data transport, Panther is not able to keep a record of which logs failed to be received. Customers can identify this by searching through the local logs generated by the system sending logs to Panther via HTTP and finding any HTTP 500 errors returned when attempting to deliver logs to Panther. You can resend these logs to Panther and Panther will correctly backdate them in the Security data lake.
We will continue to monitor the situation and will provide updates if necessary.
resolved
The AWS issue has been resolved and our monitoring confirms that there is no further impact to Panther customers.
As a reminder, If you have a deployment in the AWS US-EAST-1 region and have an HTTP log source, we recommend checking your log source(s) from Jul 30 at 3:00 PM until Jul 30 at 9:32 PM PDT to ensure that there are no missing logs.
If you have any questions, please contact Panther Support.
Panther Console Errors
開始 2024年4月25日 20:43 UTC · 5h 45m
Issues軽微なインシデント
影響を受けたコンポーネント
Panther Console (Web App)
identified
Panther has identified an issue resulting in failures in the following areas:
- Lookup table updates
- Log source onboarding
- Scheduled query runs
- Cloud security monitoring
You may receive alerts related to this issue.
We will update this incident as our investigation and remediation process continues.
identified
We have identified the root cause of the issue and have begun deploying a fix internally to verify that it resolves the problem.
In the meantime, please refrain from onboarding new log sources until we deploy the fix and resolve this incident.
We will provide an update again once we confirm the fix and deploy it globally.
resolved
After confirming the fix internally, it has been deployed globally and there should no longer be any issues.
Please note that if you onboarded a new log source after 18:30 UTC on April 25, logs will be missing from the time of onboarding until now.
We will continue to monitor to ensure there are no further problems.
Ingestion Filter Issue
開始 2024年4月19日 14:48 UTC · 4h 37m
Issues軽微なインシデント
影響を受けたコンポーネント
Data Ingestion into Panther (Log Processing)
identified
Panther has identified an issue affecting data ingestion filters. If you have configured log filtering within Panther, you may be impacted. We are currently working on a fix for the issue. There will not be any costs associated with any additional ingest from the log filter failure.
We will update this incident as our investigation and remediation process continues.
identified
We are verifying a fix that will automatically reapply filters you previously configured, which were removed due to this incident. Once the fix has been applied, we will update the status page.
resolved
We have deployed a fix, and the previously configured raw event filters should be working again. (This issue did not affect normalized event filters.) We will continue to monitor the issue to ensure there are no further problems. Once again, you will not be charged for the additional ingest caused by this incident.
If you are still experiencing issues, please submit a ticket to the support team for assistance.
Detection Upload Errors
開始 2023年12月14日 17:09 UTC · 4d 18h
Outage重大なインシデント
影響を受けたコンポーネント
Detections (Rules and Policies)
investigating
Panther has identified an issue with detection packs, which can prevent customers from running tests properly or uploading detections. Customers may also see intermittent errors related to missing modules.
This issue impacts any customer who has updated packs to the latest pack version. For any customers who have not yet updated, we recommend waiting to do so until this issue is resolved. The currently impacted versions are 3.30, 3.29, and 3.28.
We will update this incident as our investigation and remediation process continues.
identified
A fix has been identified for a new version (3.31), and customers who are comfortable waiting can hold for that availability. Otherwise, downgrading to version 3.27 can solve this issue for customers who need a resolution more immediately. Console users will need to do this on a per-pack basis, and PAT users can revert to a commit on 3.27 (or earlier) and reupload their detections.
Console users should not update packs at this time, and PAT users should avoid using Panther Analysis 3.28, 3.29, or 3.30.
We will update this incident when more information is available.
monitoring
A new version of Panther Analysis(v3.31), which includes a fix for the issue with detection packs, will be available within the next 24 hours. Once released, customers can update to that new version at their convenience.
CI/CD users may continue to experience problems uploading the current version of panther-analysis, in which case deleting the following rule should resolve any errors: netskope_admin_user_change.yml. This was already removed in panther-analysis, but may have remained depending on how customers merged from upstream.
We will continue to monitor the problem as v3.31 is released. In the meantime, if you experience any trouble, please get in touch with Panther Support directly.
resolved
Panther Analysis v3.31 has been released, so the CI/CD users should update to this new version at their earliest convenience. For the users using the Panther Console, please navigate to Build -> Packs in your Panther Console and upgrade your Packs to v3.31.
If a CI/CD user continues to experience problems uploading the current version of panther-analysis, deleting the following rule should resolve any errors: netskope_admin_user_change.yml. This rule has already been removed in panther-analysis, but it might still exist depending on how customers merged changes from upstream.
If you need additional assistance or encounter any issues, please contact Panther Support directly.
Intermittent Search Errors
開始 2023年11月29日 20:40 UTC · 4h 50m
Outage重大なインシデント
影響を受けたコンポーネント
Investigation, Data Retention, and Search (Data Lake)
investigating
Panther has identified an issue that may impact search, data-explorer, and scheduled queries. Customers could encounter intermittent errors when using any of these components.
We are actively investigating. We will update this incident as our investigation and remediation process continues.
identified
We have identified a potential root cause of the issue as a change affecting version 1.91 of Panther. We are going to be reverting the change and will monitor to see if the issue has been resolved.
We will continue to update you as we confirm the validity of the fix.
resolved
Our team has successfully implemented a patch to address the issue impacting search, the data explorer, and scheduled queries.
Our monitoring has confirmed that the patch has been successful, but we will continue to check to ensure there are no further issues.
Panther has identified an issue with the ingestion of "IntegrationLogs" and "AccessLogs" log types for the native Slack integrations, specifically affecting the Slack Standard and Plus plans. Our engineering team is currently investigating this behavior. We will provide updates on this incident as our investigation and remediation process progresses.
identified
We have identified the root cause of the issue and begun deploying a fix to resolve this problem. We will update this incident again when the issues are fully resolved.
monitoring
The fix for this issue has been deployed to all impacted accounts and our data indicates that customers are no longer encountering the errors associated with this issue. We will continue to monitor it to ensure there are no further issues.
Slack logs that previously failed to ingest due to this issue are automatically being re-ingested to Panther and no action is required by customers.
resolved
This incident has been resolved.
IPinfo enrichment update failures
開始 2023年8月29日 14:01 UTC · 10h 0m
Issues軽微なインシデント
影響を受けたコンポーネント
Investigation, Data Retention, and Search (Data Lake)
investigating
Panther has identified an issue with our IPinfo Privacy data lake enrichment that is preventing the table from being updated with the latest data. Our engineering team is actively investigating the issue, and we will post an update shortly.
Customers utilizing IPinfo enrichment may encounter a “lookup update failed for ipinfo_privacy_datalake” system error as a result of this issue.
Apart from the failure to update, IPinfo tables are functioning as expected at this time. Other core Panther functionalities, including the detection engine and log ingestion, are unaffected.
identified
After investigation, Panther has identified that IPinfo is sending over improperly formatted data, causing problems with our IPinfo Privacy data lake enrichment. We have engaged with IPinfo and will provide an update as soon as more information is available.
resolved
IPinfo has pushed a fix to resolve the problem, and we have confirmed that there are no more issues with our IPinfo Privacy data lake enrichment.
We will continue to monitor but expect no further issues.
AWS US-EAST-1 outage
開始 2023年6月13日 20:07 UTC · 3h 35m
Outage致命的なインシデント
影響を受けたコンポーネント
Detections (Rules and Policies)Alert DeliveryInvestigation, Data Retention, and Search (Data Lake)Panther Console (Web App)Data Ingestion into Panther (Log Processing)
identified
Panther has confirmed that customers with deployments in the AWS US-EAST-1 region may experience issues logging into the Panther Console, log processing and alert delivery delays, as well as other issues with the service due to the ongoing AWS outage that began at approximately 12:08 PM PDT today (https://health.aws.amazon.com/health/status).
We will update this incident with more information as it becomes available.
monitoring
AWS has reported a resolution to this outage. We will monitor the results to confirm that there is no further impact to Panther customers.
resolved
Our monitoring confirms that there is no further impact to Panther customers. Data that was delayed by this issue will be backfilled automatically.
Failure to update the latest IP geolocation enrichment data.
開始 2023年5月31日 9:12 UTC · 2d 8h
Issues軽微なインシデント
影響を受けたコンポーネント
Data Ingestion into Panther (Log Processing)
investigating
Panther has identified an issue that is preventing the update of the IP geolocation enrichment data. Our engineering team is actively investigating the issue and we will post an update shortly. Please note that the rest of the IPinfo tables are functioning as expected at this time and the rest of the ingestion processes are not affected.
The incident is still ongoing. Please disregard the status update provided in the previous announcement.
identified
We have identified the potential root cause of the issue and are already working on a fix. We will update this incident again when more information becomes available.
monitoring
Our team has deployed a patch containing the fix to resolve the issue with IP geolocation enrichment data. We will continue to monitor the fix to ensure that there are no further issues.
No data was lost as a result of this issue, and no action is required from Panther customers.
resolved
Our monitoring indicates that the fix is working as intended and the issue is now resolved.
Tor Lookup Table Errors Feb 14 2023
開始 2023年2月14日 21:11 UTC · 18h 35m
Outage重大なインシデント
影響を受けたコンポーネント
Detections (Rules and Policies)
identified
Panther has identified an issue with our Tor Exit Node Lookup Table that is causing the Lookup Table to fail to update. This will prevent some users from uploading new or updated detections via panther_analysis_tool or via the Bulk Uploader in the Panther Console.
We have developed a fix for this issue and are currently testing it. We will update this incident as our remediation process continues.
Other components of Panther, including real-time detection and alert delivery, are functioning normally at this time.
identified
Our team has completed testing the fix for this issue. To ensure that the deployment goes smoothly and without errors, we will begin deploying the fix to customers tomorrow morning during US Eastern hours. We will provide another update when we have additional information to share.
resolved
Our team has deployed the patch containing the fix for this issue to all customer accounts. Our tests indicate that the issue has been resolved and customers are no longer affected. We will continue to monitor for any potential customer impact over the next several hours.
Issues in AWS Snowflake us-east-1
開始 2022年11月23日 17:54 UTC · 6h 33m
Outage重大なインシデント
影響を受けたコンポーネント
Investigation, Data Retention, and Search (Data Lake)
identified
Panther has confirmed a Snowflake outage in the AWS us-east-1 region (https://status.snowflake.com/incidents/yh1n0ly69chm). For Panther customers hosted in us-east-1, queries in Data Explorer, scheduled queries, and viewing events that triggered alerts are potentially affected. Logs may also ingest into Snowflake with a delay.
We will continue to update this incident as more information becomes available.
monitoring
Snowflake has implemented a fix for the issue in the AWS us-east-1 region.
We will continue to monitor the situation until we can confirm it is resolved, or provide more information when possible.
resolved
Panther has confirmed that the issue in the AWS us-east-1 region has been resolved. Queries in Data Explorer, scheduled queries, viewing events that triggered alerts, and log ingestion into Snowflake should now function as expected.
If you experience any other issues related to this incident, please contact Panther support.