51 Lms365 incidents · 2023年12月 — official updates, affected components, duration and resolution details.
目前,我们正面临课程完成问题,大部分涉及使用SCORM软件包的课程,将完成该软件包/软件包的课程,但课程将保持100%的完成率。 问题已经确定,将发布一个热补字.
自动翻译自官方事件更新。
We have identified an issue affecting email notifications for customers using the legacy Perform & Engage (Weekly10) notification configuration. Our investigation found that Amazon Web Services (AWS), which supports this legacy email infrastructure, flagged an issue with the associated account and temporarily paused email sending from this service. As a result, some notification emails sent through the legacy configuration may not be delivered. Customers using the default notification sender ([email protected]) are not affected. As a mitigation, we are proactively migrating affected organizations from the legacy configuration to the default notification sender and notifying impacted administrators of the sender address change. Our teams are actively working to complete these migrations and minimize further impact. We will provide further updates as needed.
All affected customers have now been successfully migrated to Azure Communication Services and have been informed of the change. Email notification delivery has been restored for the impacted organizations, and no further action is required from customers. If you have any questions or experience any further issues, you can always create a support ticket through our Help Center.
We are currently experiencing an issue affecting learner progress management actions in Learn365. Some customers may be unable to clear learner progress, edit learner progress, or force a retake. These actions may keep loading without completing or return an error. Our engineering team has identified the issue as related to a recent migration and is actively working on a fix. We expect the fix to be available soon. We will provide further updates as soon as more information is available.
The fix has been deployed, and the affected functionality should now be working as expected. If you experience any unexpected behavior, please reach out to our Support team by submitting a ticket: https://helpcenter.zensai.com/hc/en-us/articles/14197648878237-How-to-contact-Zensai-Product-Support
### **Root Cause and Analysis** During the Learn Service deployment, a database migration introduced a schema change that removed a database column still required by another platform component. As a result, the affected component was no longer fully compatible with the updated database schema, which caused learner progress management actions to fail or remain loading without completing. The issue was related to a deployment sequencing dependency between platform components. This dependency was not identified early enough during the release validation process. ### **Resolution** Once the root cause was identified, we applied a hotfix to restore the required database column. This restored compatibility between the affected platform components and resolved the issue affecting learner progress management actions. ### **Remediation Items** To reduce the risk of similar incidents in the future, we are implementing the following improvements: **Improved release dependency checks** We are strengthening release validation to ensure dependencies between platform components are identified and confirmed before deployment. **Updated database migration guidelines** We are formalizing additional safeguards for database schema changes, especially when removing database fields that may be used by multiple platform components. **Additional deployment review automation** We are introducing automated checks to help identify potentially risky database migration patterns before release. **Stronger release readiness validation** We are improving release readiness checks to ensure critical validation steps are completed before production deployment. We apologize for the disruption this caused and appreciate your patience while our teams investigated and resolved the issue.
We have identified the cause of an issue where course pages and course flyouts may fail to load in the new UI for customers whose subscription plan does not include the Skills service. Our team is currently working on a fix. In the meantime, the old UI is not affected and can be used as a temporary workaround. We will provide an update as soon as the fix is available.
We are continuing to work on a fix for this issue.
The fix has been deployed to production, and the issue should now be resolved. If you still experience any problems, please try clearing your browser cache and cookies or opening the page in an incognito/private browser window. If you have any further questions or continue to experience issues, please contact our Support team.
This incident has been resolved.
Between 05:30 and 07:30 CET, we experienced an issue in the Central US region affecting the Audit service. During this period, certain user profile synchronization events resulted in unexpected audit behavior. The issue has been identified and addressed, and the service is operating normally. We are reviewing additional safeguards to prevent similar occurrences in the future.
We have identified an issue where courses fail to load in the mobile application. This issue impacts all users attempting to access course content via the Learn365 Mobile application. Our engineering team is actively working on resolving issue. We will provide an update as soon as further progress is available.
We are continuing to work on a fix for this issue.
A fix has been successfully deployed to the production environment to address the issue causing courses not to load in the mobile application. We are currently monitoring the situation closely to ensure system stability and confirm that courses are loading as expected. Further updates will be shared if necessary.
This incident has been resolved.
Customers experienced a short‑term (12:37 - 13:46 UTC) service disruption in the US region, during which some Learn365 applications and admin pages were temporarily unavailable. The issue has been fully resolved, all services are now operating normally, and our team continues to closely monitor the system to ensure stability.
We are investigating an issue where some users see a "Your session has expired" message immediately after opening the Course Catalog webpart, even on first load or after refreshing the page. We will provide an update as soon as we have more information.
We’ve deployed a fix after identifying the root cause. We are monitoring results closely to ensure the service remains stable and the issue is fully resolved. If you continue to experience the problem, please contact our Support team by submitting a ticket: https://helpcenter.zensai.com/hc/en-us/articles/14197648878237-How-to-contact-Zensai-Product-Support
The issue has been addressed and the service is operating normally. Following the deployment of the fix, we have verified that the platform is functioning as expected. If you experience any unexpected behavior, please reach out to our Support team by submitting a ticket: https://helpcenter.zensai.com/hc/en-us/articles/14197648878237-How-to-contact-Zensai-Product-Support Thank you for your patience while we worked to resolve this matter.
We are currently investigating an issue affecting access to the Microsoft Teams apps for Learn365 and Perform & Engage. Some users may experience blank pages or apps not loading within Teams, including errors such as “There was a problem reaching this app”. Possible workaround: Users may be able to access the apps via Microsoft Teams in a web browser while the investigation is ongoing. Our findings indicate this issue is related to the Microsoft Teams platform. We have engaged Microsoft and are working with them separately to identify and resolve the root cause. We will provide further updates as soon as more information becomes available.
We have received an update from Microsoft Support. They have confirmed that this issue is part of a recent service spike impacting multiple users across the platform. Their Engineering team is actively investigating the root cause and working on implementing a fix. This spike is causing intermittent disruptions similar to those reported. Microsoft is currently reviewing telemetry and service activity to isolate the source of the issue and mitigate the impact. We will provide the next update once we receive further information from Microsoft.
Microsoft has confirmed that the issue was caused by a service degradation on their side. Their engineering team investigated the incident and implemented the necessary fixes to restore service stability. The service is now fully operational, and we are not observing any further impact. We sincerely apologize for any inconvenience this may have caused and appreciate your patience while the issue was being resolved. If you experience any related concerns, please don’t hesitate to contact our Support team: https://helpcenter.zensai.com/hc/en-us/articles/14197648878237-How-to-contact-Zensai-Product-Support
We are currently investigating an issue with users not populating within Entra ID groups that have been added to Learn365.
A fix has been implemented and we are monitoring the results
After monitoring the effectiveness of the fix that was released we have seen a full recovery of functionality to users populating within Entra ID Groups.
A customer recently reported two security-related issues affecting access controls and administrative permissions. Upon receiving the report, our team immediately investigated, validated the findings, and deployed fixes for both items. While there is no evidence of misuse, we have taken additional steps to strengthen our security posture, including initiating a new, independent penetration test to ensure comprehensive coverage. Details of Resolved Issues 1. Access control for storage links Certain file links were accessible without the expected security measures. This has now been addressed at the resource level, and an update is being deployed to ensure proper access requirements are enforced. 2. Permissions on admin functionality There was a scenario where users could elevate their permissions if they navigated to a specific page. This has been corrected to prevent unauthorised changes. If you have further questions or would like more details about the issue, please contact Zensai Support Team.
This report outlines two security-related issues that were identified, their root causes, and the actions taken to fully resolve them. We are sharing this transparently to maintain trust and ensure clarity on the steps taken to strengthen platform security. ## Issue 1: Unauthorized Access via Legacy APIs ### Summary Certain legacy API endpoints allowed unauthorized actions, including privilege escalation and access to restricted data. ### Root Cause Some older API endpoints were not aligned with current authentication and authorization standards. These endpoints were not included in recent security audits, automated scans, or penetration testing due to assumptions about their risk level and limited test scope coverage. This resulted in incomplete security validation across the full API surface. ### Actions Taken * Immediate remediation of all exposed endpoints * Full security scan of **1,700\+ API endpoints** using automated scripts and AI-based validation * Introduction of centralized endpoint security validation processes * Expansion of dynamic security scanning to include role-based scenarios * Scheduling of a new penetration test with expanded scope and guidance * Integration of security validation into development and QA workflows ## Issue 2: BLOB Storage URL Exposure ### Summary Storage URLs were publicly accessible and did not include expiration \(TTL\), creating potential unauthorized access risk. ### Root Cause During infrastructure migration from AWS to Azure, storage URL handling was reimplemented. The new implementation did not enforce TTL or access restrictions, and security validation for storage access was not formally reviewed as part of the migration process. ### Actions Taken * Immediate implementation of **TTL \(time-to-live\)** on all storage URLs * Full obfuscation of backend storage URLs * Enforcement of authenticated access for file retrieval * Updates to QA and development review processes * Formalization of security validation for infrastructure and storage changes ## Lessons Learned * Legacy systems must be continuously audited and cannot be excluded from security reviews * Security testing must cover the **entire platform surface**, not only core or new components * Infrastructure and architectural changes must trigger mandatory security validation * Penetration testing must be supported by internal processes, automation, and continuous scanning ## Commitment We have strengthened our internal security processes, expanded automated and manual validation coverage, and embedded security checks directly into development and QA workflows. These changes ensure higher resilience, stronger prevention, and earlier detection of potential risks going forward. Security and trust remain core priorities, and we are committed to continuous improvement and transparency.
We are currently investigating an issue with User being enrolled to courses. At this time, individual Users and User Groups are having difficulty being enrolled to courses.
Our development team has implemented a fix to address the enrollment issues, and we're now monitoring the system closely to ensure stability and full recovery. We appreciate your patience and will provide further updates as needed.
The hotfix that was released has resolved the issue and we have observed enrollments completing. If you are observing lingering issues with enrollments after 630am UTC please reach out to our Support Team for additional assistance.
We are currently investigating an issue with Instructor Led Training sessions having difficulty loading in the Learn365 Admin Center.
The issue has been identified, and a fix is being created.
Our team has resolved the issue that was causing difficulty loading Instructor Led Training sessions in the Learn365 Admin Center. If you continue to experience issues, please reach out to the Zensai Support Team.
We are currently aware of an issue with an Azure service outage. We are actively investigating the issue.
We are continuing to investigate this issue.
Starting at approximately 16:00 UTC, Microsoft reported DNS issues that are affecting access to the Azure Portal and potentially other services. Microsoft has initiated mitigation steps expected to restore portal access shortly and is actively investigating the root cause.
Microsoft has reported the following: "We began experiencing Azure Front Door issues resulting in a loss of availability of some services. In addition, customers may experience issues accessing the Azure Portal." We are continuing to assess the situation.
Current status from Microsoft: We have confirmed that an inadvertent configuration change as the trigger event for this issue. We have initiated the deployment of our 'last known good' configuration. This is expected to be fully deployed in about 30 minutes from which point customers will start to see initial signs of recovery. Once this is completed, the next stage is to start to recover nodes while we route traffic through these healthy nodes. Customer configuration changes will remain blocked during this time as we work towards mitigation. We will communicate to customers when this block is reverted.
Current status from Microsoft: We initiated the deployment of our ‘last known good’ configuration, which has now successfully been completed. Customers may have begun to see initial signs of recovery. We are currently recovering nodes and routing traffic through healthy nodes, and as we make progress in this workstream, customers will continue to see improvement.
Microsoft has identified the services affected by the outage: Affected Azure services include, but are not limited to: App Service, Azure Active Directory B2C, Azure Communication Services, Azure Databricks, Azure Healthcare APIs, Azure Maps, Azure Portal, Azure SQL Database, Container Registry, Media Services, Microsoft Defender External Attack Surface Management, Microsoft Entra ID, Microsoft Purview, Microsoft Sentinel, Video Indexer, and Virtual Desktop. Per Microsoft: As recovery progresses, some requests may still land on unhealthy nodes, resulting in intermittent failures or reduced availability until more nodes are fully restored. This recovery effort involves reloading configurations and rebalancing traffic across a large volume of nodes to restore full operational scale. The process is gradual by design, ensuring stability and preventing overload as dependent services recover. We expect continued improvement across affected regions. This means we expect recovery to happen by 23:20 UTC on 29 October 2025
We are pleased to report that functionality is beginning to return across all regions. Access to all aspects of Learn365 and Perform & Engage 365 should now start to be restored for users. Please be aware that Microsoft is continuing efforts to fully resolve the issue, so you may still experience intermittent access or degraded performance during this time. We appreciate your patience and will provide further updates as the situation evolves.
From Microsoft updates: An unintended configuration change in Azure Front Door (AFD) caused a global service disruption by introducing an invalid state that prevented numerous nodes from loading properly. This resulted in latency, timeouts, and uneven traffic distribution. The root cause was a deployment process flaw combined with a software defect that allowed the invalid configuration to bypass validation. Microsoft restored stability by rolling back to the last known good configuration and has since implemented additional safeguards to prevent recurrence. For more details, please refer to the Microsoft status page: https://azure.status.microsoft/en-us/status/history/ If you experience any other issues, please don’t hesitate to contact our technical support by submitting a ticket: https://helpcenter.zensai.com/hc/en-us/articles/360019112178-Get-help-from-the-Zensai-Product-Support-team
On October 29, 2025, our platform experienced intermittent latencies, timeouts, and errors due to a widespread disruption in Azure Front Door \(AFD\), a critical component of Azure-based infrastructure. This affected our services reliant on AFD for global content delivery, impacting customer access and performance. As a consumer of Azure services, we were directly affected by this Microsoft-managed incident. Below is a summary of the root cause, resolution efforts, and planned remediation based on Microsoft's preliminary incident report \(PIR\). Root Cause Identification and Analysis The outage was triggered by an inadvertent tenant configuration change by Microsoft within Azure Front Door, which introduced an invalid or inconsistent configuration state. This caused a significant number of AFD nodes to fail to load properly, resulting in increased latencies, timeouts, and connection errors across dependent services, including our platform. As unhealthy nodes were removed from the global pool, traffic became imbalanced, exacerbating the issue and leading to intermittent availability even in partially healthy regions. The root trigger was a faulty tenant configuration deployment process, where Microsoft's protection mechanisms—designed to validate and block erroneous deployments—failed due to a software defect. This allowed the invalid configuration to bypass safety checks and propagate globally. Our internal monitoring confirmed the impact aligned with Microsoft's timeline, starting around 15:45 UTC on October 29, with services like Azure App Service and Azure SQL Database \(which underpin our SaaS offerings\) experiencing degraded performance. Resolution Microsoft's response began shortly after the incident onset. Key steps included: * At 16:04 UTC on October 29, investigation commenced following monitoring alerts. * By 16:15 UTC, teams examined recent configuration changes in AFD. * Initial communications were posted to the public status page at 16:18 UTC, with targeted notifications to impacted customers \(including us\) via Azure Service Health at 16:20 UTC. * To contain the issue, all new customer configuration changes to AFD were blocked at 17:30 UTC. * A "last known good" configuration was deployed starting at 17:40 UTC, with global pushes beginning at 18:30 UTC. * Manual node recovery and gradual traffic rebalancing to healthy nodes started at 18:45 UTC. * By 00:05 UTC on October 30, the impact was fully mitigated for most customers, though a small subset \(not affecting our platform\) experienced lingering issues. From our side, we monitored the situation via Azure Service Health alerts. Our platform returned to normal operation in line with Microsoft's recovery, with no residual issues reported by our users. Remediation Items As this incident originated from Microsoft's internal processes, there are limited direct actions we can take to prevent similar Azure-managed outages in the future. However, Microsoft has already implemented immediate safeguards, including a review of their protection mechanisms and the addition of enhanced validation and rollback controls to block faulty deployments.
We are currently investigating an issue impacting access to Perform365 and Engage365. When attempting to load the applications, users are encountering a "504 The service behind this page isn't responding to Azure Front Door" gateway timeout error. Our engineering team is actively working to identify the root cause. We apologize for any inconvenience and will provide updates as more information becomes available.
We would let you know that Perform & Engage 365 is now running smoothly. We’ve applied a fix to prevent the server from crashing, and the application is fully functional. The root cause was an inefficient query that failed to complete. To prevent similar issues, we’ve implemented a monitoring script that automatically terminates long-running queries, ensuring the database server remains stable. If you experience any issues, please don’t hesitate to contact our technical support by submitting a ticket: https://weekly10.zendesk.com/hc/en-us/articles/14012220303133-Submit-a-request-to-Zensai-Product-Su…
We are currently investigating reports of a disruption affecting the loading of the Learn365 Webparts. Our engineering team is actively working to identify the root cause and restore full functionality as quickly as possible. We will provide updates as soon as more information becomes available.
We are continuing to investigate this issue.
A fix has been successfully deployed to address the issue affecting the loading of the Learn365 Webparts. We are continuing to monitor system performance closely to ensure stability. We’ll provide a final update once we confirm the issue is fully resolved. Thank you for your patience during this time.
The issue has been resolved. The incident was linked to a recent SharePoint Online update, changes in MSAL (Microsoft Authentication Library), and a few unexpected edge cases that impacted service stability. Our engineering team has deployed a fix and confirmed that all affected services are now operating normally. We will continue to monitor for any residual impact, but no further disruptions are expected at this time. Thank you for your patience and understanding throughout the investigation.
We are aware of the issue affecting the creation of sessions ('Oops, something went wrong' error) in certain regions and are actively investigating the matter to resolve it as quickly as possible. In the meantime, if you are experiencing this problem, we recommend using the following workaround: you can create a Teams meeting and paste the meeting link into the session settings to ensure your sessions are created successfully. Thank you for your patience and understanding as we work to address this issue.
We are continuing to investigate this issue.
This incident has been resolved.
# **Session Creation Disruption in Select Regions** **Summary** Customers in select regions were unable to create sessions due to an unexpected issue related to Microsoft 365 licensing changes. When attempting to create a session, the system produced an error “Oops, something went wrong”, preventing the generation of a Teams meeting link. **Root Cause** Microsoft recently modified the licensing structure for certain Microsoft 365 accounts: [https://www.microsoft.com/en-us/licensing/news/Microsoft365-Teams-2025](https://www.microsoft.com/en-us/licensing/news/Microsoft365-Teams-2025) Previously, **Exchange Online Plan 1** often included **Teams** by default as part of a bundled suite. For many customers, session creation continued to work because their accounts already had an active Teams license. However, for affected customers, the Teams component was no longer included automatically. As a result, our platform was **unable to generate a Teams meeting link** during session creation. **Additional Note on Email App Settings** Microsoft confirmed that the following email app permissions are sufficient for proper access to users’ mailboxes and calendars: * Outlook on the Web * Exchange Web Services \(EWS\) * MAPI \(Outlook Desktop\) **Resolution** To restore full functionality and successfully create a session with a Teams meeting link, сustomers affected by the change need to ensure that both of the following licenses are assigned: 1. **Exchange Online Plan 1** 2. **Microsoft Teams license** In Learn365, reconnecting the account through **Notification Settings** is recommended to refresh permissions and ensure stability. Once the appropriate actions are applied, session creation and automatic Teams meeting generation resume working as expected. If you have any questions or are still experiencing issues, please reach out to us by opening a support ticket: [https://helpcenter.zensai.com/hc/en-us/articles/14197648878237-How-to-contact-Zensai-Product-Support#h\_01HBZEW0Z0FGSEJ6Z8VNKTFGK0](https://helpcenter.zensai.com/hc/en-us/articles/14197648878237-How-to-contact-Zensai-Product-Support#h_01HBZEW0Z0FGSEJ6Z8VNKTFGK0) We will be happy to assist you.
We are currently experiencing issues with Perform&Engage 365 access as a result of the ongoing Azure Front Door outage reported by Microsoft (https://azure.status.microsoft/en-us/status). Some users may encounter degraded performance or difficulties loading pages. We are closely monitoring the situation and will provide updates as soon as more information becomes available. Thank you for your patience and understanding.
Compliance Service is also affected by the Azure Front Door issue. Document conversion and deployment may be delayed while the Microsoft services are degraded.
As per Microsoft, https://azure.status.microsoft/en-gb/status Current Status: Azure customers using Azure Front Door (AFD) should see consistent availability with slightly higher latencies decreasing as we continue to recover additional resources.
We are pleased to inform you that the issue impacting access to Perform&Engage 365 and the Compliance Service due to the Azure Front Door outage has been fully resolved. This aligns with Microsoft's confirmation that the underlying outage has been mitigated, with all affected services recovering completely. Our monitoring confirms that Perform&Engage 365 and the Compliance Service are now operating normally, with no further disruptions observed. We sincerely apologize for any inconvenience this may have caused to your daily operations and appreciate your patience during this time. If you encounter any lingering issues, please reach out to our support team.
We would like to inform you that we are currently addressing an issue affecting user access to Learn365 within MS Teams. This disruption is due to a global MSAL update implemented by Microsoft. To ensure seamless functionality, we are working diligently to update our MSAL version. Please note that the app will be temporarily unavailable today as we complete this process. Thank you for your understanding and patience.
We would like to provide you with an important update regarding the temporary disruption to Learn365 within MS Teams. Initially, this issue was believed to be related to the MSAL update implemented by Microsoft. However, upon further investigation, we have determined that the root cause is linked to the ongoing Azure Front Door outage reported by Microsoft (https://azure.status.microsoft/en-us/status). As a result of this outage, some users may experience degraded performance or difficulties loading pages. Please rest assured that we are closely monitoring the situation and are in constant communication with Microsoft to ensure a timely resolution. We understand how critical uninterrupted access to Learn365 is for your day-to-day activities, and we sincerely apologize for any inconvenience this may have caused. We will continue to keep you informed with updates as soon as more information becomes available.
Based on internal monitoring and system logs, we can confirm that Learn365 in Microsoft Teams is operating with no major disruptions observed. It corresponds to Microsoft’s latest update on the Azure Status Page (https://azure.status.microsoft/en-gb/status): Current Status: Azure customers using Azure Front Door (AFD) are experiencing consistent availability, with slightly elevated latencies that continue to decrease as additional resources recover.
We are pleased to inform you that the issue affecting access to Learn365 within MS Teams has been fully resolved. This aligns with Microsoft's confirmation that the underlying Azure Front Door outage has been mitigated, with all affected services recovering completely. Our monitoring confirms that Learn365 is now operating normally in Microsoft Teams, with no further disruptions observed. We sincerely apologize for any inconvenience this may have caused to your daily operations and appreciate your patience during this time. If you encounter any lingering issues, please reach out to our support team.
Due to urgent technical maintenance, Perform & Engage DE was temporarily unavailable for approximately 40 minutes. The issue has now been resolved, and the platform is fully operational.