Störungsverlauf von Knowledgeowl

We are currently investigating reports of 504 errors and slow load times.

We are temporarily putting KnowledgeOwl into maintenance mode in order to let the system recover. We will be bringing everything back on line as quickly as possible.

All services are currently in maintenance mode. This includes knowledge bases, the application, files, and the API.

We are still in emergency maintenance mode as we sort out the underlying issues that were causing the 504 errors. We are so sorry and are trying to get you all back online as quickly as possible. As one of our customers said, it is the Mondayest Monday.

Our team is actively investigating and has attempted several mitigations, none of which have resolved the issue. We'll continue posting hourly updates, or sooner if the situation changes.

A fix has been implemented and we are monitoring the results. All systems should be back online. We will continue to monitor over the next hour before marking this as resolved, and we will share a postmortem once our review is complete.

The fix is holding and this incident is now resolved. We'll share a post-mortem later this week. Thanks for your patience today.

**What happened** We successfully completed a critical transition away from a legacy dependency; however, the deployment introduced a dormant performance regression. This issue remained hidden during low-traffic periods, only impacting database health once subjected to the full weight of production loads. **How we resolved it** Our monitoring systems detected the issue and alerted our team. Our engineering team placed all services into maintenance mode during investigation and troubleshooting. Once a successful fix was in place, all services were brought back online. **What we're doing to prevent this** We are evaluating improvements to our testing process to better simulate real-world traffic conditions before deploying infrastructure changes. We apologize for the downtime and appreciate everyone’s support, kind words, and patience during and after the incident.

We're investigating reports of 504 gateway timeouts and other slowness accessing knowledge bases and the KnowledgeOwl application.

We've identified the issue and are working to implement a fix.

We've rolled out a fix for the issue and we're monitoring performance.

Our fix seems to be working and we haven't seen any further issues. We'll share a post-mortem tomorrow. Thank you for your patience today!

Thank you for your patience through yesterday’s incident. We know performance impacts to our app and your live knowledge bases are never pleasant, and we appreciate your patience and grace while we worked through things. Here’s a brief postmortem of what happened and our steps to remediate: An unexpected spike in traffic led to intermittent 504 errors. To maintain core system availability, we temporarily restricted computationally heavy operations while scaling our infrastructure. Once that was complete, we returned back to normal operations. We’ve noticed no significant spikes or errors since then.

We are investigating reports of files being marked as unsafe by Microsoft Defender SmartScreen. We are working on a fix.

This issue is limited to Microsoft Edge. Someone flagged a file uploaded to a KnowledgeOwl trial account as phishing, which has since been resolved with AWS (the file and trial account were deleted). However, Microsoft Edge is now flagging all files uploaded to KnowledgeOwl as unsafe. We've submitted a report to Microsoft requesting a review. In the meantime, Edge users can help by reporting the site as safe.

This incident has been resolved. Microsoft has reviewed and unflagged our file library. Files uploaded to KnowledgeOwl should no longer be marked as unsafe in Microsoft Edge. If you are still experiencing issues, please contact support.

Microsoft Edge users experienced security warnings when accessing knowledge base files due to a SmartScreen flag on our file hosting domain. No data was compromised. We have deployed additional safeguards to prevent recurrence.

We are currently investigating the situation to find the cause of the intermittent issues we're seeing.

A fix was implemented, and we have been monitoring results. The issues have been stabilized for 30 minutes, and we will continue to watch closely for any changes.

This incident is now resolved. KnowledgeOwl is fully operational, and systems have been stable during our monitoring period. Thank you for your patience and understanding throughout today’s degraded performance. If you notice anything unexpected, please don’t hesitate to reach out. 🦉💛

A database cluster node experienced degraded performance, temporarily impacting overall app performance until automatic failover mechanisms restored normal operations.

We’re currently experiencing an outage and KnowledgeOwl is unavailable for some or all customers. Our team is actively investigating the issue and working to restore service as quickly as possible. At this time, you may see errors when trying to access your knowledge bases or the KnowledgeOwl app. We’ll share updates on this page as soon as we have more information.

We temporarily placed KnowledgeOwl into maintenance mode to support a safe and stable recovery. Services are now coming back online, and most customers should begin seeing normal access return. We’re continuing to bring systems up carefully and will monitor performance closely throughout the process. If you notice anything unexpected, please reach out. Thank you for your patience while we ensure everything returns to full health.

KnowledgeOwl is back online and functioning normally. We’re keeping the incident open for now while we continue to monitor system performance closely. We’ll move this to Resolved once we’re confident everything is fully stable. Thank you for your patience today!

This incident is now resolved. KnowledgeOwl is fully operational, and systems have been stable during our monitoring period. Thank you for your patience and understanding throughout today’s disruption. If you notice anything unexpected, please don’t hesitate to reach out. 🦉💛

**What happened** A recent security-related library upgrade introduced unforeseen CPU inefficiencies at the same time we saw a significant increase in legitimate traffic. The combination temporarily overwhelmed our resources before the system could scale as expected. **What we are doing about it** We are actively working to stabilize performance and reduce the likelihood of this happening again, including short-term mitigations while we address the underlying CPU issues. **Our apologies** Thank you for your patience and understanding, and as always, if you notice anything unexpected, please reach out. Our apologies again for the disruption!

We fixed an issue that was causing error message on knowledge bases and the KO application. It is looking to be resolved but we will continue monitoring before we mark it as resolved. Sorry for the trouble folks!

We’ve identified the root cause (an internal error) and have taken steps to ensure it won’t happen again. Have a great weekend and we’ll do our best to stay out of your inbox!

We are currently experiencing an outage affecting knowledge bases. Our team is actively investigating the issue and working on a resolution. We will provide updates as soon as we have more information. Thank you for your patience!

We are continuing to investigate this issue.

A fix has been implemented and knowledge bases are back up. We are monitoring.

This incident has been resolved. Thank you again for your patience!

We are currently investigating the situation to find the cause of the intermittent issues we're seeing.

We've narrowed down the issue to a release that went out on Friday, September 6th. The release was rolled back and we are monitoring to ensure the slowdowns and connection errors have been resolved. We will continue to investigate what went wrong with this release.

We are still investigating the underlying cause of the issue with Friday's release. However, no further issues have occurred since the rollback so we are marking this incident as resolved.

We are currently investigating errors accessing both the knowledge base and the web application.

We have blocked traffic that appears to have been malicious and are monitoring to make sure things are okay now.

We have not seen any new errors in over 20 minutes and all customers who reported the issue have confirmed it appears to be resolved. Hoot!

We are investigating reports of images not loading and errors with newly uploaded files. Many thanks to everyone who reported the issue and so sorry for the trouble. We're working on getting this fixed as soon as possible!

We think we've fixed the issue. We are continuing to monitor and confirming the fix with impacted customers. For anyone that received an error while uploaded or updating a file or image, you will need to upload it again for it to work properly.

This incident appears to have been resolved. We've been monitoring for the past hour and are no longer seeing errors with uploading files and images. We've confirmed with multiple customers that the issues appear to be resolved on their end as well. We are still determining what caused the issue and we will send a postmortem with our findings and next steps. Thanks again to everyone who reported the issue and helped us get it resolved so quickly!

We are investigating reports of slowness and errors when logging into the application and knowledge bases.

Everything seems back to normal so we are resolving. Let us know if you are still having any issues and thank you for your patience!

We just started seeing reports and warnings that KnowledgeOwl is down. Our team is focusing on getting it sorted and back online ASAP.

We are continuing to investigate this issue.

We seem to be back to normal operations, but we're continuing to monitor performance and finish our investigation of the initial root cause.

Our fix seems to have fully resolved the issue; we'll be issuing a postmortem shortly to explain root cause in more detail. Thank you all for your patience today!

One of our servers went into a failing state and began issuing a vast amount of internal connection requests. Those requests overloaded our network. Rebooting the server solved the issue, and internal traffic patterns have returned to normal. ## Next steps We are reviewing our alarms on the affected server to see if we could have detected the failure sooner and prevented the downtime.

We are currently investigating and are working to get everything back up and running asap. We will be posting updates here!

We are continuing to investigate the intermittent outages and slowness. Since just before 7 am EDT, the system seems to have gone down and recovered on its own a few times. While it appears to be back up and working, we will leave this incident active as we continue to investigate and monitor the situation. We will continue to post updates here along with a postmortem once it is resolved. Thanks for your patience this morning and our apologies for the disruption to your day!

Our CTO confirmed that all systems are back to operational and we will continue to monitor. We have a possible lead as to the cause of today's issues and will post more information as it becomes available. Sorry again for the trouble and let us know if you are still having any issues!

The system is back down. We are dealing with an abnormal amount of traffic and trying to resolve asap. We are so sorry for the trouble today.

Quick update from our end. The app and knowledge bases have come back online and seem to be stable now. We have a lead on potential root cause but we are still working through it. We are so sorry for the disruptions to your day and we will be sharing more information as we work to fully resolve the problem.

Thank you everyone for your patience today. Based on our investigation, it appears that KnowledgeOwl was attacked by malicious traffic which caused the intermittent outages and slowness. We have made some changes to mitigate the impact of the spikes, but we are not quite ready to mark this as resolved as we want to ensure that there are no more spikes of illegitimate traffic. We will be providing more information and a postmortem once we feel the issue is fully resolved. Please let us know if you are still having issues.

We are officially marking this incident as resolved. All systems have been operating normally since approximately 12 pm EDT, and we have not seen any more malicious looking traffic hitting our servers. We are continuing to investigate the source of the potential attack and we are planning to share a postmortem on Monday. We are sorry for the trouble this caused and we appreciate the grace everyone has shown as we dealt with this problem. Let us know if we can help with anything in the meantime, and we will be in touch early next week with our postmortem.

# Summary In the early morning of Friday, March 24th, our systems began to see large spikes of web traffic from various IP addresses. These traffic spikes were targeting our public website \(www.knowledgeowl.com\). This high traffic volume overwhelmed the reverse proxy system that we use to serve the public website, the KnowledgeOwl application, and customer knowledge bases. As an immediate mitigation step, we changed the configuration of our public website to no longer use our primary proxy. This seemed to stabilize the application and customer knowledge bases. The traffic spikes continued until late morning. Our investigations indicated that these connections were not initiated by customers. The associated web requests did not appear to be legitimate. While we cannot say with certainty that these connections were malicious, we are treating this incident as a distributed denial of service \(DDoS\) attack. # Next Steps The risk from high-volume traffic spikes like these is almost impossible to completely remove. However, we are reviewing our systems and processes to better handle these kinds of traffic spikes. We have already identified some concrete next steps to reduce the overall risk: ## Short-term In the short-term, we are taking three steps: 1. We are exploring ways to further separate our public website from the KnowledgeOwl application and customer knowledge bases. 1. Friday's incident provided us with data on how to better identify these types of events. We are building that knowledge into our processes moving forward. 1. We have already begun modifying our proxy and web application firewall configuration to make our traffic management infrastructure a bit more robust. We'll continue to monitor these changes and iterate on them as needed. ## Long-term This event has highlighted some potential architectural improvements to our infrastructure, mainly around our traffic-management systems. We'll review these changes' feasibility and effectiveness in the coming months. # Thank you Above all, we want to thank you for your patience during this incident. We know that we've had a higher number of downtime incidents in the last two months. We know how integral your knowledge base can be to daily operations. Our team is working hard to learn from this experience and to make KnowledgeOwl stronger in the future.

We have received multiple reports of slowness and intermittent 504 errors. We are currently investigating and will report back asap. Sorry for the trouble and thanks to everyone for letting us know!

The issues seem to have subsided around 11:45 ET. We are currently keeping an eye on it and will wait a bit before marking the issue as resolved.

This appears to be resolved. Thanks to everyone who reported the problem and sorry for the trouble today!

We are continuing to monitor for any further issues.

This incident has been resolved.

We identified two processes that contributed to this week's slowness during our investigation. Both have been on our list to update, but we're prioritizing them as a result of this week's incident. We're already testing a fix for the first process. The second is a larger change that will take longer to resolve. We'll include these fixes in our normal release notes, identified as "performance improvements". Thank you all for your patience this--and every--week.

We've had several reports of the KnowledgeOwl app and knowledge bases being slow or inaccessible this morning. We're investigating the root cause and will provide updates as we have them.

We've confirmed a full outage of the app and knowledge bases and are actively working to get it resolved. Sorry for the disruption to your day and we hope to be back online quickly!

We have identified the issue and are testing a fix.

It looks like things have stabilized due to our fix, but we are continuing to monitor performance.

The fix we implemented isn't performing as well as we'd hoped. We're taking sites down briefly to implement an additional fix.

We've rolled out a new fix and are monitoring its performance. So far we've seen some small performance spikes but nothing that should prevent access. We'll continue to monitor to be sure things are resolved.

Our monitoring has looked good and we're seeing continued normal performance across the board, so we're marking this as Resolved. We have noticed some issues with knowledge base tables of contents either missing some content or having duplicate articles. If your knowledge base is showing either of these issues, please reach out to our support team and we can get you back to a normal table of contents state. Thank you all for your patience and being so gracious with our team today through this whole outage. We'll post a full postmortem after we've fully fleshed out the root cause and next steps.

# Summary Last night we released a set of changes to the table of contents that had a bug in it. This bug caused display issues in tables of contents. First, we tried to hotfix the issue. The hotfix caused some processes to eat up more memory than normal. The memory shortage built until it caused a slowdown and an outage. At this point, we rolled the release back completely. # Next Steps We already have a new fix ready to work into the initial release. Thanks to today's issues, we've identified several opportunities for improvement: ## Short-term We're updating our testing and release processes for changes to the table of contents. We'll be using these revamped processes to test the fix and the full release before we take it live. ## Mid-term We're reviewing our enterprise- and business-level account SLAs. We'll be issuing credits to customers whose up-time SLAs weren't met this month. If you're a customer in one of these tiers, you can expect to hear from a member of our team to discuss this in more detail. ## Long-term We've also identified several possible improvements in our load-testing processes. We'll be making changes to those processes, too. # What you can do While the bug was live, it may have caused changes to your knowledge base's table of contents. Please review your table of contents. If you see duplicate articles or missing subcategories, please email [[email protected]](mailto:[email protected]) so we can get it fixed. # Thank you Thank you for your patience and grace with us through this month's issues. Outages are every software provider's worst nightmare and we are very thankful to have such amazing customers.

We are currently investigating and are working to get everything back up and running as quickly as possible. We will be posting updates here!

A fix has been implemented and we are monitoring the results.

All systems are operational and we will continue investigating the root cause. We are going to close this incident as it appears to be resolved. Please contact [email protected] if you need any help!

## Incident postmortem A traffic distribution system failed and was not restarted by automation. We are auditing these systems to ensure this failure does not happen again. Due to gaps in our overnight emergency alert tools, it took us longer than normal to resolve the issue. We are going to review our processes to make sure that more staff receive alerts when there is an outage. We will also look into training more staff to be able to handle incidents like this.

We are currently investigating and are working to get everything back up and running as quickly as possible. We will be posting updates here!

We are back up and stable right now. We are still monitoring and investigating the root cause of the issue.

All systems are operational and we will continue investigating the root cause. We are going to close this incident as it appears to be resolved. Please contact [email protected] if you need any help!

Our background job processor is currently backed up. This means delays for processes that hit that worker, including things like bulk edit jobs and PDF generation. You may see these types of tasks taking longer to complete than normal. We're working to get the worker back to full bandwidth and appreciate your patience in the meantime.

Background jobs are still experiencing some delays while the background queue recovers.

The background job processor is now fully caught up. We're continuing to monitor performance.

This incident has been resolved.

Like much of the internet, KnowledgeOwl is being impacted by issues with Amazon Web Services (AWS) ec2-us-east-1. This seems to be slowing KnowledgeOwl performance in: 1. Our Server Host 2. Our File Storage So far, we believe this slower performance could be affecting you in two major areas: File Library (images and files): You may have issues with adding new files, updating files, and/or deleting files from within articles or within the library itself. Jobs we run in the background: These jobs are run from actions like generating/updating PDFs, reindexing content, reordering articles or subcategories within a category, and some CSV exports. We expect these jobs to be slow/backed up a little even once full AWS service is restored. We'll keep you posted as we hear more from AWS, and we especially appreciate your patience today!

Background jobs and file library issues appear mostly resolved. AWS is reporting that their services have mostly recovered.

Resolving as no further incidents have occurred since the last update from AWS.

We are currently investigating reports of 504 gateway time-out errors and elevated response times.

We are getting reports that things appear to be back to normal. We will continue to monitor the situation and investigate what happened.

Performance has been stable for the last hour. Please let us know if you notice any further issues. Thanks for your patience and quick reporting today!