68 Knowbe4 incidents · 2025年9月 — official updates, affected components, duration and resolution details.
We have received reports of an issue with delivery of Email verification MFA for Workspace. We are investigating this issue and will update this page when we have more information.
We have received reports that the issue with delivery of Email verification MFA for Workspace has been mitigated by Microsoft, and we are monitoring the results.
This incident has been resolved.
We have identified an issue where users may experience delays in messages being processed, and our engineering team is looking into a fix.
Our engineering team has implemented a fix for the issue that causing message processing delays in PhishER, and we are monitoring the results.
Our engineering team has implemented a fix for the issue that causing message processing delays in PhishER.
We have received reports that the Gmail Phish Alert Button shows the error "[reportPhishAttempt()] An internal error has occurred. Please try again." when reporting phish. We are investigating this issue and will update this page when we have more information.
The issue has been identified and a fix is being implemented.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
On Friday, August 21, 2026, from approximately 05:20 to 12:55 \(UTC\), some customers experienced intermittent failures when using the Gmail Phish Alert Button add-in to report suspicious emails, receiving the "An internal error has occurred" message instead of a successful report confirmation. The issue affected customers across all regions, with the highest error volumes concentrated in the EU-West and US-East service regions. This issue was caused by a disruption in a caching service on Google's Apps Script platform, which the add-in relies on to store per-user session configuration. The service began intermittently returning empty results on read while still accepting new writes, causing affected report submissions to fail. KnowBe4 engaged Google directly and filed a formal bug report between approximately 12:19 and 12:39 \(UTC\). Error volume declined sharply shortly after as the vendor-side condition cleared, and the Gmail Phish Alert Button returned to normal performance by approximately 12:55 \(UTC\). To prevent this type of issue in the future, we are continuing to work with Google to track the resolution of the underlying caching defect. No data loss occurred as a result of this issue.
We are investigating an issue causing a Non-Delivery Report (NDR) to be returned when a message is reported via the Phish Alert Button and "Send Us a Copy" is enabled in Account Settings
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We are currently investigating an issue affecting Google PhishRIP queries, which may cause them to fail.
Based on our investigation into the recent errors affecting Google PhishRIP queries, it appears these may be related to suspended Gmail accounts. We are continuing to monitor the issue.
Upon further investigation we have determined that the errors affecting Google PhishRIP queries were related to individual issues on Google accounts.
We have identified an issue preventing account settings from loading in the SAT console.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
我们正在调查一个 有关电子邮件被 PhishRIP 意外隔离的问题
这一问题已经解决,PhishRIP的查询不再出乎意料地对信息进行断层处理。 我们还在调查可能受到影响的询问
这一事件已经得到解决.
□ ** 摘要** 2026年7月28日,PhishRIP服务的更新使得一些PhishRIP的搜索查询比他们本该有的要广泛. 不符合指定发送标准的电子邮件被隔离到一些客户账户中. 自动监测和客户支持票迅速浮出水面。 工程公司在宣布事件发生后约25分钟内恢复了这一变化,并停止了意外的采石. 然后,工程将错误的隔离邮件恢复到客户的收件箱。 □** 旋转原因** 2026年7月28日,我们向PhishRIP背后的查询服务部署更新维护. 这一更新旨在改进对无效发件人地址的处理,允许系统倒回只搜索地址的域部分. 然而,现有的验证逻辑使用一个正则表达式,需要发送字段中的"@"符号. 当一个查询使用一个空域,如[example.com](http://example.com)没有用户名或"@"符号时,regex标记域为无效. 现有验证与域搜索更新之间的这种冲突导致系统完全放弃了发送者标准. 查询与收件箱相对应,仅使用其剩余参数,在预定发送者以外的邮件进行匹配和断开。 □** 时间线** 所有时间都在协调世界时. 时间 时间 时间 时间 | -- -- -- -- -- -- -- 19:36 我们的自动监控检测出异常的查询执行模式,并通知了我们的工程师,他们立即开始调查. | QQ 19:51 QQ 问题升级为高发量出产事件,组建了反应小组. | 19:52 我们的支援团队升级了第一批举报出乎意料的检疫的顾客入场券,证实了监测警报. 19:54 原因来源于最近一个查询 API的部署,工程开始恢复变化。 事件通讯被张贴到我们的状态页面, 20:16 部署完全恢复。 QQ20:35 生产检查证实,主体和域名查询再次只匹配预定目标,不再有信息被隔离出错. 我们的情况网页也相应更新。 | 20:39 工程开始恢复错误的隔离邮件. \ 03: 49\ (下天\ )\ \ 完成所有客户查询集的恢复, 通过我们的支援团队登录 。\ \ \ \ □ ** 缓解和补救** ** 恢复守则。 ** 我们权衡了停止PhishRIP在中途的行动 与恢复代码。 停止背景任务会打乱每个租户的合法安全工作流程,因此恢复工作是更好的选择。 在升级后约40分钟内就已部署并核查了密码。 ** 电子邮件修复。 ** 为了避免真正恶意的电子邮件回到客户环境中,我们手动恢复而不是批量恢复. Engineering使用内部管理员工具分批恢复电子邮件,并和Support合作验证每个账户ID,查询ID,并首先获得客户批准. 所有报告账户一夜之间都恢复了。 □ ** 预防措施** * ** 衰退覆盖率**: 我们正在增加单域、非标准、光线发送查询的单元和集成测试,以便在部署之前抓住这些输入案例。 * 护栏**: 我们正在查询引擎中添加一个检查, 如果其核心过滤参数在评估中被丢弃, 则会阻止查询运行 。 □ ** 结束** 一个单一的验证规则导致了这种情况:一个假定每个发件人字段都包含有QQ的regex. 当查询使用空域时,该假设放弃了发送过滤器,查询匹配的邮件远远超过预期. 一旦我们找到它,修复是直截了当的,而修复工作在宣布事件发生后约25分钟内就停止了破坏. 恢复隔离邮件是一个更困难的过程:我们无法在不冒真正恶意邮件返回收件箱的风险的情况下恢复邮件的批量. 我们的小组选择在整个夜晚中谨慎行事,按支助帐户进行工作。 两项预防措施更早会发现这一点:一项测试只涵盖域域发送者查询,另一项是当发送者过滤器被剥去时拒绝运行查询的护栏. 两者现在都计划好了。 眼前的问题已经解决,所有已上报的账目都已恢复。 □** 荣耀** * ** PhishRIP:** PhishER功能让管理员搜索并隔离一个组织中每个用户的收件箱的接字邮件. * ** Regex\(正则表达式):** 软件中用于搜索文本并验证输入的图案,例如检查电子邮件地址格式是否正确. * ** 代码还原:** 后出软件更新以返回到之前的稳定版本.
自动翻译自官方事件更新。
我们正在调查用户无法在商店购买保护许可证。 我们得到更多信息后将更新这一页.
我们执行了一个定律, 我们正在监测结果.
我们收到报告说,顾客无法在商店购买保护许可证。 我们正在调查这一问题,并将在获得更多信息后更新这一页.
并监测结果以确保不再发生更多问题.
We are continuing to monitor the implemented fix for the online store, to ensure that no further issues occur.
自动翻译自官方事件更新。
我们已查明一个问题,即用户在进入防御系统时可能会得到转机,我们的工程小组正在寻找一个固定点.
已经确定了一个解决办法,我们将在结果推出时监测结果.
这一事件已经得到解决.
On Thursday, July 16, 2026, from approximately 19:57 to 21:12 \(UTC\), some customers experienced difficulty logging in to the Defend console, seeing a **Find Out More** screen instead of console access. This issue was caused by a synchronization issue between Salesforce and our internal licensing systems, which incorrectly set some customer license counts to zero. As a result, affected customers were unable to log in. To resolve this issue, our team applied a temporary license override to restore access for affected customers while correcting the underlying licensing records in Salesforce. Once corrected, the systems automatically synchronized the accurate values, and the Defend console access returned to normal performance by approximately 21:12 \(UTC\). To prevent this type of issue in the future, we are implementing improvements to the Salesforce synchronization process to ensure accurate license counts. No data loss occurred as a result of this issue.
自动翻译自官方事件更新。
我们正在调查用户无法在商店购买保护许可证。 我们得到更多信息后将更新这一页.
我们正在继续调查这一问题.
一项措施已经执行,我们正在监测结果.
这一事件已经得到解决.
自动翻译自官方事件更新。
上游地物旗服务在协作推论中出出退化的模型路由,导致PhishML评价返回0/0/0默认错误. 第三方部署一个固定装置,并部署一个代码更新器,用于生产,添加静态后退路由. 所有服务已经恢复并正常运行.
自动翻译自官方事件更新。
We have received reports that users are experiencing issues when logging into the US instance. We are investigating this issue and will update this page when we have more information.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
## **Executive Summary** On July 1, 2026, the KnowBe4 Security Awareness Training \(KSAT\) platform experienced a period of degraded performance resulting in intermittent login errors and high latency for users on our United States \(US\) instance. The issue was initiated following a routine platform deployment that introduced an unoptimized database query. This query placed an excessive operational load on our primary database reader cluster, causing database sessions to saturate and subsequent login requests to queue up. Engineering teams promptly identified the degradation, reverted the deployment, and systematically cleared the backlogged database sessions to restore optimal performance. The issue did not affect data integrity or security, and service was fully stabilized. ## **Technical Root Cause** The root cause was determined to be a newly introduced query within a standard application update. Upon deployment, this specific query pattern bypassed optimal indexing strategies, resulting in full table scans and highly extended execution times on the database reader infrastructure. As a high volume of authentication and application requests arrived concurrently, the database reader quickly exhausted its available connection pool due to these long-running, unmitigated database sessions. This resource starvation immediately manifested as severe application latency and intermittent timeouts during the user authentication process. ## **Timeline of Events** The incident began in the mid-morning hours and progressed through identification, remediation, and verification stages over a period of approximately 152 minutes. An internal high-severity incident response group was established immediately following automated monitoring alerts indicating that health checks targeting our application programming interface \(API\) routing layer were failing from regional cloud monitoring nodes. This behavior was confirmed by concurrent engineering analysis of user HTTP Archive files, which demonstrated severe latency spikes specifically isolated to the authentication endpoints. Within three minutes of establishing the response team, cross-referencing recent system changes pointed to a recent application deployment as the primary catalyst. Engineers immediately initiated the rollback process, drafting and approving a revert modification to extract the problematic code from the deployment pipeline. The deployment of the reverted codebase to the US production cluster commenced shortly thereafter. While the deployment processed over the subsequent twenty-five minutes, technical personnel prepared direct database interventions to clear the residual system strain. Once the stable code version was completely active across the fleet, engineers began systematically terminating the lingering, long-running database sessions that had been spawned by the unoptimized query. To ensure a pristine state, the database reader infrastructure was cycled twice. Following these administrative infrastructure restarts, operational telemetry showed the database reader load dropping significantly to a healthy baseline of approximately thirty-six percent. System performance normalized, and administrative logging verified that authentication requests were processing within standard latency thresholds. After monitoring the environment to confirm sustained stability, engineers officially marked the incident as mitigated, later shifting the status to fully resolved following an extended window of zero performance spikes and complete passes on all automated sanity test suites. ## **Mitigation** To alleviate the immediate infrastructure distress and restore user access, the engineering team executed a multi-phased mitigation strategy: * **Codebase Rollback:** The changes introduced in the recent deployment were immediately isolated, reverted, and redeployed to production to prevent any further generation of the unoptimized query. * **Database Session Termination:** Internal engineering tools were utilized to explicitly terminate active, long-running database queries that were blocking the connection pools. * **Infrastructure Cycling:** The database reader instances were restarted twice in succession to flush out stale memory allocations and guarantee that all orphaned database sessions were permanently cleared. ## **Preventative Measures** To prevent a recurrence of this specific issue and mitigate the impact of similar query-based database bottlenecks in the future, KnowBe4 is implementing the following actions: * **Enhanced Query Linting and Analysis:** Integrate automated query execution plan analysis into our continuous integration and continuous deployment pipelines to flag unindexed or high-cost queries before they reach production environments. * **Database Connection Pooling Guardrails:** Adjust database timeouts and implement aggressive circuit-breaker thresholds for user authentication paths to prevent single, long-running query patterns from exhausting the entire connection pool. * **Load Shedding Policies:** Implement strict application-level timeouts on read-heavy database calls to ensure they fail gracefully rather than degrading the overall availability of the core login workflows. ## **Conclusion** We sincerely apologize for the inconvenience and friction this performance degradation caused our customers and partners. KnowBe4 is dedicated to maintaining high availability and reliability across our product suites. By refining our pre-deployment automated query validation and bolstering our database connection resiliency, we are actively working to ensure the continuous, seamless operation of the KSAT platform. ## **Glossary of Technical Terms** * **API \(Application Programming Interface\):** A set of protocols that allows different software applications to communicate with one another. In this context, it routes authentication requests from the user interface to the backend servers. * **Database Reader:** A dedicated database instance or cluster responsible for handling read-only queries \(such as fetching user profiles or validating login configurations\), separating this traffic from write operations to optimize performance. * **HAR \(HTTP Archive\) File:** A JSON-formatted log file that records a web browser's interaction with a website, used by engineers to diagnose performance and network latency issues. * **Latency:** The time delay or duration it takes for a data packet or request to travel from its source to its destination and return a response. * **Sanity Suite:** A collection of automated tests executed against a deployment environment to quickly verify that the core functionality of an application is working correctly.
We have received reports that emails sent to Defend are experiencing delays being delivered to inboxes using SMTP. We are investigating this issue and will update this page when we have more information.
This incident has been resolved.
#### Summary On June 30, 2026, customers using the Defend US service experienced delays in email processing. The incident began at approximately 14:00 UTC and was fully resolved by 19:45 UTC. #### What Happened A scheduled maintenance operation began in the early morning of June 30. As this operation progressed, it placed an unexpectedly high load on our infrastructure, which caused email processing to slow down across our US service. Our team identified the issue and declared an incident at 16:00 UTC. Steps were taken to reduce the load and restore normal processing speeds, including pausing non-essential background activity and engaging our infrastructure provider for additional support. By 17:45 UTC, email delivery delays had been fully resolved. Email analysis continued to recover and was back to normal by 19:45 UTC. #### Customer Impact Email delivery \(SMTP customers\): Emails were delayed in transit by up to 30 minutes between approximately 14:00 UTC and 17:45 UTC. All emails were delivered; no messages were lost. Email analysis \(Microsoft 365 / Graph API customers\): Email analysis was delayed by an average of 30 minutes before emails were analysed, between approximately 14:00 UTC and 19:45 UTC. Email delivery to end users was not affected. #### What We Are Doing We have rescheduled the maintenance operation that triggered this incident to run during an overnight, low-traffic window, giving it sufficient time to complete without affecting the live service. We are also investing in infrastructure improvements to better isolate maintenance operations from customer-facing workloads, so that future maintenance cannot affect email processing in this way.
我们发现PhishML评价导致应用"PML:BYPASSED"标记的问题.
我们为PhishER执行了一个固定方案, 被影响的信息可以通过所有规则和行动被重播. 请注意, 如果此事件成功进行, 可能会造成重复回复 。 如有其他问题或关切,请与我们的支助小组直接联系:https://support.nowbe4.com/hc/en-us/requests/new 以下"知识库"文章包含了如何重播消息的指令: https://support.nowbe4.com/hc/en-us/articles/13169303385619-PhishER-Inbox-Guide#h 01HCNEBF8CJQ98GE9PGM7HGDZ0
这一事件已经得到解决.
2026年6月30日星期二,大约从07:40到19:15\(UTC\),顾客在PhishER的PhishML评分中经历了错误的结果. 受影响的电子邮件收到一个PML:BYPASSED标记,而不是合法的PhishML分类,受影响信件中缺少信心分数. 取决于PhishML结果的规则和行动也没有被激活. 这个问题是由大约两周前推出的代码重构引起的. 此重构引入了一个错误的更新,省略了PhishML分数运行所需的基本驱动程序. 然而,这个问题一直处于休眠状态,直到另一次更新引发了新的PhishML模式的部署,从而产生了评分问题。 为了解决这一问题,我们小组回到上次稳定部署阶段,并增加了处理由此产生的积压电子邮件评价的能力。 PhishER的PhishML得分于19:15\(UTC\)恢复了正常表现. 为了防止今后出现这类问题,我们改进了健康检查,在部署前引入了烟雾测试新模式的新终点.
自动翻译自官方事件更新。
We have identified an issue where users may receive a 500 error upon logging into KCM GRC, and our engineering team is looking into a fix.
We have identified an issue where users may receive a 500 error upon logging into KCM GRC. A fix has been implemented, and we will continue to monitor this issue.
We have identified an issue where users may receive a 500 error upon logging into KCM GRC. A fix has been implemented, and users should be able to login without error.
From Tuesday, June 23, 2026, at approximately 10:03 \(UTC\) to Thursday, June 25, 2026, at approximately 11:54 \(UTC\), some US and EU customers experienced intermittent 502 errors when logging in to KCM GRC. This issue was caused by network traffic attempting to connect to invalid multilevel subdomains, which overwhelmed the cache serving KCM GRC and resulted in login errors. Our team initially updated the configuration of our content delivery network, which temporarily resolved the errors, but they returned later that day. After further investigation, we identified the caching issue as the root cause and deployed an infrastructure-level fix to prevent multi-level subdomain traffic from affecting the cache. KCM GRC returned to normal performance by 11:54 \(UTC\) on June 25, 2026. To prevent this type of issue in the future, we are evaluating additional protections to guard against similar traffic that could affect the cache. No data loss occurred as a result of this issue.
We have received reports of the Phishing test reports tab currently being unavailable. We are investigating this issue and will update this page when we have more information.
We are continuing to investigate this issue.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
From Tuesday, June 16, 2026, at approximately 17:34 \(UTC\), to Thursday, June 18, 2026, at approximately 22:41 \(UTC\), some customers experienced intermittent unavailability of the **Phishing Security Test Reports** page in the KnowBe4 console. This issue was caused by a code change that introduced a conflict between two methods for processing phishing campaign data. As a result, phishing campaigns still using legacy phishing categories were unable to load the **Phishing Security Test Reports** page. To resolve this issue, we updated the code to process campaigns correctly under both classification systems, and the **Phishing Security Test Reports** page returned to normal performance by June 18, 2026, at 22:41 \(UTC\). No data loss occurred as a result of this issue.
We have received reports that the Human Risk Managment widget is showing no results in the KnowBe4 Security Center. We are investigating this issue and will update this page when we have more information.
We’ve implemented a fix for the Human Risk Management widget and we’re monitoring the results to make sure no further issues occur.
This incident has been resolved.
From Wednesday, June 17, 2026, at approximately 19:00 \(UTC\), to Thursday, June 18, 2026, at approximately 18:20 \(UTC\), some customers were unable to view results in the KnowBe4 Security Center’s Human Risk Management widget. This issue was caused by a recent infrastructure migration that left an internal service connection pointing to an outdated endpoint. Though the underlying data processing was unaffected, the Human Risk Management widget could not retrieve data or display results. To resolve this issue, our team updated and reapplied the connection configuration, and the KnowBe4 Security Center returned to normal performance by approximately 18:20 \(UTC\) on June 18, 2026. To prevent this type of issue in the future, we are improving our migration process to ensure that all endpoints are valid after a migration. We are also strengthening known-good rollback procedures in cases where a migration cannot be completed as planned. No data loss occurred as a result of this issue.
We've identified an issue that has resulted in inconsistent reporting, and our engineering team is looking into a fix.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
From Tuesday, June 16, 2026, at approximately 22:23 \(UTC\), to Wednesday, June 17, 2026, at approximately 20:19 \(UTC\), some customers experienced inaccurate user counts in KSAT reporting. Downstream features that rely on this data were also affected, including AIDA Orchestration, Risk Score, and ModStore recommendations. This issue was caused by an incomplete rebuild of the users' data table. A data storage policy removed historical data earlier than intended, so when a full table rebuild ran, some older data was unavailable, causing user counts to drop. To resolve this issue, our team performed a full data migration sync on the users' table and retriggered the dependent data pipelines to update downstream systems. We then tested and confirmed that user counts were returning correct results, and the KSAT console returned to normal performance by June 17, 2026, at 20:19 \(UTC\). To prevent this type of issue in the future, we are correcting the underlying data storage policy to ensure complete historical data is retained for the full rebuild process. No data loss occurred as a result of this issue.
This incident has been resolved.
On Monday, June 15, 2026, from approximately 15:30 to 16:34 \(UTC\), some customers experienced processing delays with phishing and training campaigns in the KnowBe4 console. This issue was caused by an update that introduced an incompatible software version. This software prevented our background processing service from completing queued tasks. As a result, training enrollments, notification sending, and SmartGgroup enrollments were delayed. To resolve this issue, our team deployed a fix that reverted the affected dependency and restored normal job processing. Once the fix was deployed, the affected queues cleared, and the KnowBe4 console returned to normal performance by 16:34 \(UTC\). To prevent this issue in the future, we have implemented additional testing layers for similar deployments. No data loss occurred as a result of this issue.
我们收到报告称,在浏览用户配置时发生了500个错误。 我们正在调查这一问题,并将在获得更多信息后更新这一页.
一项措施已经执行,我们正在监测结果.
这一事件已经得到解决.
2026年6月3日星期三,大约从14:49到16:09\(UTC\),客户在KSAT控制台上访问**用户详情**页面时遇到错误. 造成这一问题的原因是更新引入了缺失字段,从而阻止了**用户细节**页的加载. 为了解决这个问题,我们团队再次更新了控制台,以恢复缺失的球场,KSAT控制台在大约16:09\(UTC\)恢复了正常运行. 为防止今后出现类似问题,增加了自动化测试. 这个问题没有造成数据损失.
自动翻译自官方事件更新。