Störungsverlauf von Walkme

We are currently investigating this issue.

Microsoft Edge v150 has implemented a blocking policy that prevents all extensions from running on cloud.microsoft domains. This is due to changes Microsoft made to their browser, and it is impacting all extension users including WalkMe. We have escalated to Microsoft and are advocating for a fix as soon as possible.

We are continuing to investigate this issue.

Per https://github.com/microsoft/MicrosoftEdge-Extensions/issues/644#issuecomment-4933908795 "Currently, extensions are not designed to work with Copilot. At this time, the list of sites where Microsoft extensions are blocked includes Copilot, Edge:// pages, and EAS*" From our research, it seems that the affected domains are only the following 2 domains: https://m365.cloud.microsoft/ https://copilot.com/

We have become aware of a known Chromium bug that is intermittently impacting the load time of Edge/Chrome on initial boot up. This behavior is intermittently impacting certain Windows OS users with the WalkMe Extension installed. The Chromium bug is related to Chrome's Service Worker lifecycle, causing network requests or the extension itself to stall and hang indefinitely. The Chromium team is working on resolving the behavior and the progress can be followed by visiting the following link: https://issues.chromium.org/issues/484218883 If you believe you may be impacted, please open a case to the WalkMe Support team who will be able to assist with applying an interim solution while the bug is being patched.

WalkMe content is not appearing for multiple customers due to DeepUI being down. Resolved by 16:26 UTC

We are currently investigating this issue.

We are seeing that the issue is stabilizing and we are currently monitoring the results.

We are continuing to monitor for any further issues.

This incident has been resolved.

We have received reports of WalkMe intermittently loading with a slight delay for some customers

The issue has been identified and a fix is being implemented.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

Unable to publish content or settings on any US account. Error will display in Editor "Update and Publish error" The incident affected 40 systems that attempted to publish during the timeframe.

Due to a bug introduced in the newest Microsoft Edge version, many browser extensions, including the WalkMe extension, may fail to work properly on Edge versions 130 or later. Microsoft has confirmed they are working to revert the impacted Edge versions in the next release, which is scheduled for this week. An update will be provided once a more detailed release schedule is provided. WalkMe engineering has been able to determine the following workarounds to review with your IT team for an immediate solution as an alternative to waiting for the Microsoft fix. Unique Extension Deployment: WalkMe can provide a new extension that can be mass-deployed by your IT team. Please reach out to WalkMe Support for assistance. The unique extension offers additional long term benefits including: Quicker access to extension hotfixes and new features than the Edge webstore Enhanced security controls - URL allow and block lists to control where WalkMe is allowed to play Branding customizations - Customize the name and description end users see in the browser Ability to disable auto updates and control updates Reverting Edge Version: Revert to a previous version of Microsoft Edge, specifically version 128. Please find a link to the Microsoft bug here: https://github.com/microsoft/MicrosoftEdge-Extensions/issues/179

We are continuing to work on a fix for this issue.

Microsoft has released a fix for this issue in Edge stable version 130.0.2849.56. Please see their status page update under version 130 here: https://learn.microsoft.com/en-us/deployedge/microsoft-edge-known-issues#high-impact-issues With this Microsoft fix for the Edge store extension, the workaround WalkMe offered in regard to a unique extension is no longer needed. However, we'd like to highlight the benefits below if this is something users would like to convert to in the long-term. Please feel free to reach out to WalkMe Support for any assistance. Quicker access to extension hotfixes and new features than the Edge webstore Enhanced security controls - URL allow and block lists to control where WalkMe is allowed to play Branding customizations - Customize the name and description end users see in the browser Ability to disable auto updates and control updates

Microsoft has released a fix for this issue in Edge stable version 130.0.2849.56. Please see their status page update under version 130 here: https://learn.microsoft.com/en-us/deployedge/microsoft-edge-known-issues#high-impact-issues With this Microsoft fix for the Edge store extension, the workaround WalkMe offered in regard to a unique extension is no longer needed - however, we'd like to highlight the benefits of switching to a unique extension, in order for you to make an informed decision as to if this is something you would like to convert to in the future. Those benefits include: Quicker access to extension hotfixes and new features than the Edge webstore Enhanced security controls - URL allow and block lists to control where WalkMe is allowed to play Branding customizations - Customize the name and description end users see in the browser Ability to disable auto updates and control updates

We are currently investigating this issue.

We are continuing to investigate this issue.

This incident has been resolved.

We are currently investigating this issue.

The issue has been resolved and we are monitoring the results.

This incident has been resolved.

WalkMe, an SAP company, takes very seriously our obligation to comply with the law. Following Russia's invasion of Ukraine, various US and European regulatory authorities have imposed export and trade restrictions that prevent WalkMe from providing our service in the affected region.

We are currently investigating this issue.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

We are currently investigating this issue.

The issue has been identified and a fix is being implemented.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

editor/insights/console is down for US accounts. We are investigating and will update shortly.

Our Development team is investigating and working on resolving the issue as soon as possible. We will report back with an update shortly.

We are continuing to investigate this issue.

Our Development team has been able to restore most services, however, the Editor's publish functionality is still impacted. We are continuing to work hard to resolve this and will have an update soon.

WalkMe Services have been restored and our team is monitoring to ensure all items are resolved.

All WalkMe components impacted by today's outage have been restored. Thank you for your patience as our team worked to ensure WalkMe functionality was fully restored. Our initial findings indicate an issue with the WalkMe Authorization and Authentication endpoints in our US services to be the cause. Please expect this incident to be updated with a root cause analysis including all relevant details in the coming days.

### Description of Incident * On Mar 6, 2024 ,14:09 UTC, WalkMe experienced an elevated level of service errors in our Design Time API Gateways. * This issue primarily affected our WalkMe builders, who may have experienced difficulties when connecting to the WalkMe Editor, Console, or Insights. * After urgent investigation by the WalkMe engineering team, and in order to avoid further disruption, a set of recovery steps was performed on the databases and load balancers. * Once recovery steps were completed, the Gateway services returned to normal and Design Time APIs were fully functional by March 6, 2024, 20:42 UTC ### Scope of Incident * This issue primarily affected our WalkMe builders, who may have experienced difficulties when connecting to the WalkMe Editor, Console, or Insights. ### Root Cause Analysis * A new product configuration caused some of our underlying services to enter an unstable state. This triggered a ripple effect on additional internal services. ### WalkMe Corrective Action * WalkMe performed a rollback from the latest backup 17:40 UTC, to Database snapshot of 13:20 UTC ### Ongoing Commitments * To uphold WalkMe's commitment to providing reliable and uninterrupted service, we are actively monitoring our systems to ensure this issue does not recur: * WalkMe will add additional designated tests for these specific components and configurations. * WalkMe will increase the observability, monitoring and alerting on these specific components and configurations. * WalkMe will apply additional load protection layers on core services.

WalkMe Workstation is unavailable. Our Development team is currently working to restore.

Editor is not working in US data center - login, preview, publish. R&D are aware and working on resolving the issue.

This incident has been resolved.

Our Legacy SSO offering was temporarily offline due to a change made in the API mechanism that handles the requests. Our Development team has reverted the change and restored the ability to utilize the Legacy SSO offering as of September 5th at 3 PM UTC. A temporary workaround of enabling password login was provided to those customers impacted until the incident was rectified. Customers utilizing our self serve SSO offering were not impacted.

We are currently investigating an issue impacting the US Data center for the WalkMe Editor, console.walkme.com, admin.walkme.com, and insights.walkme.com. Our team is working hard to resolve the issue as soon as possible.

This incident has been resolved.

Our R&D team has identified an issue relating to incoming integrations which is impacting Segmentation or Conditions based on integrations. This may result in the Segmentation or Conditions based on Integrations incorrectly evaluating as false. The source of the issue has been identified and our team is working to resolve with the highest priority. We are constantly monitoring for updates and will report when this behavior is resolved.

We received an update that our team has been able to stabilize the behavior and are working on fully resolving the issue.

Our team is continuing to work on fully resolving this behavior with the highest priority.

Our team has identified the root cause of the issue and is working on a solution with the highest priority.

A fix has been implemented and we are monitoring the results.

After continuous monitoring our team has confirmed that the fix applied is working as expected and the Integration based Condition Service is fully operational.

Our R&D team has identified an issue relating to incoming integrations which is impacting the access of the integrations page at insights.walkme.com as well as segmentation based on integrations. The source of the issue has been identified and they are working to resolve with the highest priority. We are constantly monitoring for updates and will report when this behavior is resolved.

A fix has been implemented and we are monitoring the results. Please reach out to Support with any ongoing issues.

This incident has been resolved.

WalkMe US Editor is currently down for all users. At this time, we are unable to log into the Editor.

Connection issue which was fixed by our R&D.

## **Postmortem & Root Cause Analysis** On July 4 at 21:00 UTC, a rare technical failure in a microservice impacted the WalkMe Editor’s loading mechanism. The failure of the microservice did not activate automatic alerting that would usually have identified and escalated the issue more rapidly to WalkMe development teams for resolution. ‌ The database continually tracks issues, logs them, and escalates them as required. There are settings that determine how many of these issues the database can log. Error logs are typically removed manually after a period of time. Thresholds were set at a level that is no longer appropriate for logging issues for this microservice. When too many requests are received by the microservice, the database automatically blocks hosts/IPs. This is what resulted in the WalkMe Editor not being usable. ‌ To resolve this issue permanently, we cleared the error records for this microservice and increased the threshold for errors that can be logged in the future. We introduced a new alert to notify development teams that the error log needs to be manually cleared before thresholds are met. Finally, we also reviewed all thresholds across all microservices to ensure that alerting is appropriately activated to the development teams for any similar issue in the future.