Störungsverlauf von Codetwo

We're observing delays in email delivery in the Germany West Central region. We're investigating this right now. It looks like degraded performance of Exchange Online Protection (EOP). Signatures are added normally.

We’ve confirmed that the delays are due to degraded availability of Exchange Online Protection in the Germany West Central region and are not related to CodeTwo. We’ve contacted Microsoft and we’re actively monitoring the situation.

We’re continuing to monitor the situation. Email signatures are being appended correctly. However, delivery of messages with signatures is still delayed due to an issue with Microsoft Exchange Online Protection servers, which are experiencing degraded performance and are processing messages from our service slower than usual. We can see that the issue is widespread in Germany. Other services that rely on the Exchange Online mail flow pipeline are experiencing similar delays in this region. Microsoft is currently working to resolve the problem, and we are actively monitoring their progress. Thank you for your patience. We’ll provide another update in 90 minutes or sooner if anything changes.

We are seeing an improvement in message processing times. This is likely due to lower traffic levels during off-peak hours in Germany. Microsoft continues to investigate the issue. At this time, the problem remains isolated to EOP servers located in the Frankfurt region. Thank you for your patience. We’ll provide another update in 3 hours or sooner if anything changes.

Microsoft reports in EX1331830 that Exchange Online users may experience significant delays or failures when sending and receiving email messages in Exchange Online, with some messages remaining undelivered for over an hour. The impact is currently limited to users in North America and Germany (although we do not see these delays in North America). Microsoft is continuing to analyze the issue and determine next steps. This confirms this is not a CodeTwo infrastructure issue. At the same time, we are observing a significant improvement in server-side mail processing performance in Germany for all CodeTwo users in this region. Queues are now nearly cleared. We continue to monitor the situation. All signatures are added correctly. Thank you for your patience. We’ll provide another update in 3 hours or sooner if anything changes.

Microsoft reports in EX1331830 that they are now evaluating targeted service infrastructure resets as a potential mitigation and they're collecting additional diagnostic data to determine the next steps. We continue to observe stable mail flow performance in the Germany West Central region, with no delays in other regions. All signatures are added correctly. We’ll continue to monitor the situation and provide another update in 3 hours or sooner if anything changes.

Microsoft reports in EX1331830 that their configuration changes have reached approximately 50% of the affected infrastructure and message queues are steadily improving. This is also confirmed by our telemetry. More information can be found in the Microsoft 365 Admin Center under EX1331830 (https://admin.cloud.microsoft/#/servicehealth/:/alerts/EX1331830), if accessible. CodeTwo is and has been operating normally throughout this incident. All emails were delivered with signatures attached. As the issue is no longer affecting our mail flow and is not related to CodeTwo infrastructure, we are resolving it. Our teams will continue to monitor the situation 24/7/365.

We’re investigating an issue affecting email delivery for messages processed by CodeTwo services to apply email signatures. We’re also checking whether this may be related to an Azure DNS problem. Please stay tuned for further updates.

We’re seeing indications that Azure DNS issues could have affected email flow for some customers using CodeTwo services. DNS appears to be ok again, and several previously affected customers have already confirmed successful email delivery. We’re continuing to investigate to rule out any other causes. No configuration changes or updates were made to our services today.

Microsoft has acknowledged the issue after receiving reports from multiple organizations. They are investigating on their side as well, but the root cause has not yet been identified. On our side, we are no longer receiving additional reports and all systems appear to be operating normally, but we’ll continue to monitor the situation and carry on with our investigation.

It looks like the issue is no longer occurring, but we are continuing to monitor the situation and are still investigating the root cause. Here’s a quick summary of events: 1. Some of our mail processing servers were temporarily unreachable, most likely due to an Azure DNS outage. As a result, some emails sent by our customers (using the cloud-mode for signatures) were rejected with a bounce‑back (NDR) by Exchange Online because our mail processing domain could not be resolved at that moment. 2. Some emails still went through. This happened either because they did not hit the Azure DNS issue (not all messages were affected), or because they were queued by Exchange Online for delivery and Azure DNS came back online before the system generated an NDR. 3. Once DNS became available again, our domain was visible and queued messages could be delivered normally. 4. Nothing in this incident occurred within CodeTwo’s systems. The issue happened entirely before any message reached our service pipeline, which is why our service logs did not show any abnormalities. 5. Emails that did not generate an NDR (did not bounce back) can be assumed to have been delivered successfully. Emails that generated an NDR (bounced back) will need to be resent by the original senders. We are continuing to investigate this issue with Microsoft (who has now confirmed the Exchange Online infrastructure experienced an issue) and are trying to determine why Azure DNS became unavailable for a short period. We'll let you know about the results of that investigation. All CodeTwo services are operating normally.

We have confirmed that the situation has returned to normal, and Microsoft has acknowledged that the issue originated on the Microsoft 365 side. For now, we are closing this incident, but our 24/7/365 Monitoring Team will continue to keep a close eye on the service. We will share more details in the post‑mortem once Microsoft provides us with their RCA.

# Summary On March 18, 2026, a small subset of CodeTwo customers experienced temporary issues with email delivery. The incident was caused by a Microsoft networking and DNS‑related issue within Microsoft 365 infrastructure and was not the result of a failure in the CodeTwo service. # Impact * Only a limited number of customers were affected. * The issue occurred intermittently. * The impact was limited to customers whose mail flow depended on Microsoft 365 services affected by the incident. # Root cause Microsoft confirmed that the incident \(ID MO1255228\) was caused by a temporary issue with their networking infrastructure, where part of the infrastructure intended to handle production traffic was only partially reinstated. This led to DNS and load‑balancing issues, affecting services such as Exchange Online. The incident originated entirely within Microsoft’s infrastructure and was outside of CodeTwo’s control. # Resolution Microsoft manually reinstated the full networking infrastructure and confirmed service health recovery. Following this action, telemetry indicated stable and healthy service operation. # Why CodeTwo was affected CodeTwo services integrate directly with Microsoft 365 and Azure and rely on Microsoft‑managed DNS and networking components. DNS failures at the Microsoft platform level affect all services integrated with Microsoft 365, including third‑party solutions. There is no technically viable way for any Microsoft 365-integrated service to be fully immune to platform‑level DNS or networking failures on Microsoft’s side. # Preventive measures * CodeTwo services are monitored 24/7/365. * We operate redundant and failover‑capable infrastructure. * In case of Microsoft‑side incidents, we take all actions within our control to minimize customer impact, including traffic rerouting where technically possible. Microsoft stated that this type of incident has been extremely rare and committed to improving safeguards to prevent similar events in the future. # Current status Microsoft has confirmed that the affected services were fully restored and operating normally. No further issues related to this incident have been observed.

Microsoft Azure is currently experiencing availability issues in the West US region, which may affect email delivery speed for customers using server-side signature rules. We are currently investigating the problem.

At this time, we have not received any customer reports related to this issue, but as a precaution, we are failing over to redundant infrastructure within the same region.

After switching over to tertiary infrastructure, Azure is now processing our requests correctly and email delivery speed has returned to normal. Emails that were delayed on the previous two environments should be delivered soon with signatures applied. We are continuing to investigate why the primary Azure resources are unhealthy; however, all new emails are currently being processed through our regional failover services and are being delivered in a timely manner with signatures.

The failover infrastructure is operating correctly, and our telemetry indicates that previously delayed emails have now been delivered with signatures applied. The issue appears to be mitigated, however, our 24/7/365 Monitoring Team will continue to closely observe the service. We will switch back to the primary Azure infrastructure once it becomes available. This transition will be transparent to customers and will not cause service interruptions. No customer action is required. We will also work with Microsoft to determine the root cause of the regional service issue.

This incident has been resolved.

We are currently investigating an issue causing email delivery delays for customers whose services are located in the United States data centers. The problem appears to stem from degraded performance of Microsoft Exchange Online Protection (EOP). End users sending emails that receive CodeTwo signatures on the server side (Cloud mode) may experience delivery delays, as Microsoft Exchange Online Protection is not processing these messages in a timely manner. This issue is very likely also affecting other Microsoft 365 users who do not use CodeTwo. We are actively investigating the problem and checking whether we can force a failover to other instances that are not impacted by the EOP outage.

We are seeing that Canadian customers are also affected by the Microsoft Exchange Online Protection outage.

Microsoft has just published information about the outage on X: “We’re investigating a potential issue impacting multiple Microsoft 365 services, including Outlook, Microsoft Defender, and Microsoft Purview. Further information can be found in the admin center under MO1221364.” https://x.com/msft365status/status/2014422298506285161

Another update from Microsoft on X: "We've identified a portion of service infrastructure in North America that is not processing traffic as expected. We're working to restore the infrastructure to a healthy state to achieve recovery. More information can be found at https://status.cloud.microsoft or under MO1221364 if accessible."

The fix implemented by the MS Exchange Online team worked and our service is fully operational again. The most recent update from Microsoft: "We’re continuing to see steady improvements in service availability for Exchange Online and Microsoft Teams and our broader recovery efforts remain actively underway. Engineers are further refining load balancing measures to accelerate recovery and are methodically addressing the remaining impacted services."

We are currently experiencing a localized backend communication issue between Azure services in the UK region, which affects the ability of some customers to sign in using Microsoft accounts to the Signature Management page (app.codetwo.com) and manage their rules and signatures. All indicators suggest that this is an issue on Microsoft’s side: we have not made any changes today, and the exact same configuration works without interruption across our other nine regions and eighteen datacenters. We are actively investigating the issue. All other services remain fully operational, and email signatures continue to be added as usual. No action is required on your side at this time. Please stay tuned for further updates here.

We switched the affected nodes to alternative, healthy resources within the region, which appears to have resolved the issue. We are no longer seeing any sign-in timeouts or errors. We will continue to monitor the situation.

This incident has been resolved.

A subset of customers using OWA and the New Outlook are experiencing issues when adding/loading pictures in email signatures via email signature add-ins (such as CodeTwo). Some images load slowly, while others fail to load entirely. We have reproduced the issue in our own CodeTwo production environment for certain users. We are currently in direct contact with the Microsoft product group, and they have confirmed that they are seeing the same problem. The issue is under active, high-priority investigation by Microsoft. Further details are pending, but early findings indicate that the problem is not limited to Add-ins - it also affects OWA and New Outlook clients in general. Initial analysis suggests latency in OAuth as the potential root cause. Please wait for more information. In the meantime, you can: * Choose signatures without pictures for your end-users, or * Use server-side mode, where signatures are added by CodeTwo in the cloud (not via Microsoft add-ins).

Microsoft is still investigating this issue. So far, we know the problem is related to loading images in OWA and New Outlook and is not specific to add-ins - it affects any type of images in Outlook. Fortunately, the issue is not widespread. If you have any questions, please feel free to reach out to our Support Team (which is available around the clock) while we await Microsoft’s resolution. https://www.codetwo.com/company/contact

We have an update from Microsoft Product Group about this issue: "We suspect the latency issue is caused by high CPU usage regression. That could cause latency issues in uploading image. We already have the reverts done and the fix will deploy via a fast track soon. We need to monitor saturation over the next 24-48 hours." As a reminder: This issue is not related to add-ins. It concerns inline images in general within the New Outlook and OWA and affects a subset of customers globally, regardless of whether they use any signature add-in solution.

Microsoft has published official information about this incident on the Service Health Dashboard (SHD) under ID: EX1183800. Please refer to the SHD for more details. Since this officially confirms that the issue was not related to CodeTwo, we are resolving this incident on our end. If you have any questions, please raise a service request with Microsoft. You are also welcome to contact our Support Team at: https://www.codetwo.com/company/contact. All CodeTwo services have been operational during the incident. Thank you for your understanding and cooperation.

We are currently observing degraded performance across Azure resources globally. This appears similar to the issues experienced with Azure Front Door a couple of weeks ago caused by Microsoft engineers. As a result, some users may encounter difficulties accessing Microsoft 365 services and certain CodeTwo applications that rely on the same infrastructure. Affected services may include the CodeTwo Admin Panel and Signature Management Apps, which are used for managing rules, signatures, and additional configuration settings. Our telemetry indicates that mail flow remains uninterrupted at this time, and email signatures are being applied as expected. Check back here for more info and follow Microsoft status pages and notifications in the Microsoft 365 Admin Center.

Downdetector is currently reporting widespread performance issues affecting Azure and Microsoft 365 applications globally. We are also observing partial impact on Amazon Web Services at the same time. We are actively monitoring CodeTwo mail flow and can confirm that email signatures are still being added as expected. We are currently awaiting an official statement from Microsoft regarding the ongoing service disruptions.

Here is the current status from Microsoft: User impact: Users may be unable to access the Microsoft 365 admin center and see delays when accessing other Microsoft 365 services. More info: Additionally, admins are reporting issues when attempting to access some Microsoft Purview and Microsoft Intune functions. Users are also seeing issues with add-ins and network connectivity in Outlook. Current status: We're reviewing reports of an issue impacting Microsoft Azure and Microsoft 365 services, including impact to the Microsoft 365 admin center and other services, including but not limited to those described in the more info section of this communication. We're reviewing corresponding service telemetry to isolate the cause of the issue and determine our next troubleshooting steps. Scope of impact: Any users attempting to access Microsoft 365 services may be impacted. Link: https://status.cloud.microsoft/ Microsoft status on X/Twitter: https://x.com/MSFT365Status/status/1983571232122671591 More details can be found in the Service Health Dashboard under MO1181369.

Here's another update from MICROSOFT suggesting it is a DNS issue (again): Starting at approximately 16:00 UTC, we began experiencing DNS issues resulting in availability degradation of some services. We have taken action that is expected to address the portal access issues here shortly. We are actively investigating the underlying issue and additional mitigation actions.

Microsoft is currently working to revert changes caused by a faulty DNS configuration, which has led to widespread service disruptions. In response, we have proactively rerouted some of our services away from the affected nodes, successfully bypassing the impacted Azure Front Door infrastructure. At this time, users should be able to access the Signature Management App, although for some, it may take up to 45 minutes for changes to propagate. However, the Admin Panel may still be temporarily unavailable. Our telemetry confirms that email signature imprinting services have remained operational throughout the Microsoft Azure outage and have not been affected.

We’re seeing signs of recovery as Microsoft continues working to resolve the DNS configuration issue on their end. Some users are no longer experiencing problems accessing CodeTwo resources, while others may still encounter delays or temporary unavailability when accessing CodeTwo apps. According to Microsoft, the full recovery process on their side may take up to four hours - they are planning to finish by 23:20 UTC on 29 October 2025.

The issue appears to be resolved now. A bunch of guys were about to turn blue (when we realized Microsoft has messed up the DNS configuration) - we're breathing again. Thank you for your patience and understanding! This was a widespread incident affecting multiple Microsoft resources worldwide. Some of the performance issues we experienced as a result were unavoidable. Nevertheless, we’ll continue to enhance our high availability capabilities to make our are services more resilient to similar problems in the future. Please stay tuned, and thank you for your continued trust and business. If you have any questions or need assistance, please reach out to our Customer Success Team - we’re online 24 hours a day. And yes, our SOC Team also works around the clock, continuously monitoring our services both reactively and proactively. These teams, along with our Product Team, will remain on “DEFCON 1” until we’re certain the issue has been fully mitigated.

Some of our apps - such as the Admin Panel and the Manage Signatures App - may experience degraded performance, including slow loading times or failure to load entirely. This issue is due to widespread disruptions affecting Microsoft 365 and Azure services. You can track updates via https://x.com/MSFT365Status/status/1976221477859983782 or https://status.cloud.microsoft, although the latter may also be intermittently unavailable as it's hosted by Microsoft. Microsoft has identified its Front Door services as the likely cause, which is also impacting standard Microsoft 365 users. We are actively monitoring the situation on our end. Despite the app loading issues, email signatures continue to be appended correctly, and mail delivery remains unaffected.

We’re seeing that the Azure Front Door Services issues are now also affecting users in the UK and UAE regions. As a result, Microsoft 365 users in these areas may experience intermittent login issues when accessing their M365 resources. Some CodeTwo apps - also hosted on Azure - may be impacted as well, due to the widespread nature of the disruption. Important Reminder: All email signatures are being appended as expected, and we are not observing any issues with mail delivery at this time. You can follow updates on this Microsoft incident via: https://x.com/MSFT365Status/status/1976221477859983782 https://status.cloud.microsoft (Note: this page may also be intermittently unavailable)

Microsoft has recovered most of affected resources. We are monitoring the results.

This incident has been resolved.

We are currently investigating an issue with delays in email delivery for messages with server-side signatures sent by users in Australia.

The problem with delays in email delivery appears to be limited to a small subset of customers in Australia who may have experienced degraded performance of Microsoft 365 today. Our services are operating normally and the signatures are imprinted. We're monitoring the situation.

It looks like the incident affected only a handful of tenants in Australia East, and the overall impact was very limited and not related to CodeTwo. We’re not seeing any issues with our services, so we’re closing this incident. Our 24/7/365 Proactive Monitoring Team will continue to actively monitor the service. If you are experiencing any problems, please feel free to reach out to our Support Team - we’re available around the clock.

Microsoft is investigating an issue where some users are experiencing delays when sending emails from Microsoft 365. As a result, messages sent through our system are also affected, since we are part of the same mail pipeline. Please refer to EX1144276 in the admin center for more details. Here’s Microsoft’s status update on X: https://x.com/msft365status/status/1961409977400132058 Microsoft currently states that the issue is limited to Europe.

Microsoft continues to investigate this issue. Please refer to EX1144276 in the admin center for more details. Here’s Microsoft’s status update on X: https://x.com/msft365status/status/1961409977400132058 All emails processed through CodeTwo are currently being delivered with signatures, although some may experience delays. CodeTwo is operating normally.

The problem has been resolved by Microsoft. For further information, please check Service health in the Microsoft 365 Admin Center (Incident ID: EX1144276). CodeTwo is operating and has been operating normally. All emails were delivered with signatures attached.

Microsoft is currently experiencing Azure availability issues across several nodes in Australia. We have switched our infrastructure to healthy instances within the same region. However, some emails that were being processed during the failover are still stuck in the affected node. As a result, messages sent in the last 25 minutes may be delivered with a delay to final recipients. All current messages are being delivered without delay, and all emails include signatures as expected. At present, all services are operating normally, and no action is required on your end. We expect the issue to be resolved within approximately 30 minutes.

This incident has been resolved.

Microsoft is currently experiencing problems with Exchange Online Protection (EOP). EOP servers appear to have degraded performance and are slow to process emails entering and leaving Microsoft 365 tenants. As a result, some messages processed by CodeTwo may also be delayed while they wait for EOP to scan them before being delivered to final recipients. We have reported the issue to Microsoft and are waiting for a resolution. In the meantime, we have scaled up our services to increase the chances of routing emails through EOP servers that are not affected by this problem. What should you do if you're impacted? Inform your users that email delivery might be delayed (usually by just a few minutes or less) and wait for Microsoft to resolve the issue. Please note that this is a Microsoft-side problem, and it also affects users who are not using CodeTwo or any other smart hosts in the affected region. All signatures are added normally.

Microsoft has confirmed that the issue is on their end. The issue ID is EX1079935. While email delays in Australia are decreasing, we are now seeing the UK region slowly starting to be affected. We anticipated this and have already warmed up failover services to help mitigate the impact as the issue emerges in this region. Our services are continuously retrying to send messages that were previously stuck in queues before reaching the EOP servers so that we can send them through healthy EOP instances. All messages will be delivered with signatures, although some may be delayed.

It looks like Microsoft has resolved the EOP issue. We are no longer observing any email queues before EOP servers, and all messages are being delivered without delays. We will continue to monitor the situation.

This incident has been resolved.

There’s a problem with our certificate provider (Sectigo), which is currently unable to create new certificates for some customers trying to start a trial subscription. As a result, new users attempting to sign up for our server-side or combo-mode service won’t be able to complete the setup process until the issue is resolved. All existing subscriptions for existing Customers are unaffected. Signatures are being added as normal. We're monitoring the situation. Our Tech Support team is online - feel free to reach out to them if you have any questions.

The issue seems to be resolved, even though our certificate provider hasn't officially confirmed it yet. Our telemetry shows that all certificates are now being issued correctly. All affected customers should no longer experience any problems. Our Support Team will contact each impacted customer to ensure everything is running smoothly. As always, we will continue to monitor the service 24/7/365.

This incident has been resolved.

We are getting reports from Customers that Outlook signatures are not working for some users, and the CodeTwo Outlook Addin is missing. The issue affects only a subset of users within impacted tenants. We're investigating the problem together with Microsoft.

We are continuing to investigate this issue. At this time, we see that the majority of users remain unaffected. Please also note that cloud (server-side) signatures are functioning normally. Therefore, if you are using combo mode and the Addin is not working, signatures will still be added to your emails, but only after they have been sent.

We have received information from some of the users who experienced this problem that the add-in is visible in Outlook and working again. We are still investigating this issue with Microsoft.

We are continuing to investigate this issue. Microsoft has now engaged their Outlook and Office extensibility teams in Redmond to assist with the investigation. We can still confirm that the issue is random, affecting only a limited number of tenants and a small number of users within each affected tenant. As a reminder: server-side signatures are functioning normally. Therefore, if you are using combo mode and the add-in is not working, signatures will still be added to your emails, but only after they have been sent. If you have any questions, our Customer Support team can be reached here: https://www.codetwo.com/company/contact

This issue continues to be under investigation. We have noticed that other vendors are reporting similar issues with their Outlook add-ins on their status pages, indicating this might be a Microsoft-related problem. Fortunately, the number of impacted end-users remains very low. Server-side signature rules are functioning normally. Other components are also fully operational.

We are continuing to investigate this issue.

Microsoft has identified the problem on their end. The issue was caused by incorrect information being written into Microsoft's entitlement caches due to a configuration error on Microsoft's end. Fortunately, the "bleeding" was minor, affecting only a small percentage of users. Microsoft engineers have implemented and deployed a fix to prevent further impact on users. The caches are now being repopulated for end-users who were affected by this issue. Microsoft is confident that the problem will be fully resolved before the business day begins in EMEA on Monday.

We are monitoring the results of the fix implemented by Microsoft. Some impacted customers have already confirmed that the fix worked. If affected users still don't see the Add-in, they can try restarting Outlook or OWA, then trigger the Add-in by composing a few new emails.

Most impacted customers have confirmed that Microsoft's fix has successfully propagated to their Microsoft 365 tenants, and the Add-in is now available again in Outlook/OWA for affected end-users. If any impacted end-users still report that their Add-in is missing, please instruct them to restart or reload Outlook/OWA multiple times and to try composing several new emails to trigger the Add-in. Should the problem persist, contact our Customer Support Team at: https://www.codetwo.com/company/contact. Customer Support will prioritize this issue and respond promptly. Our Monitoring and Development Teams will continue actively monitoring this situation to ensure the issue does not recur. The problem was beyond our control and resulted from incorrect data being written to Microsoft's entitlement caches due to a configuration error on Microsoft's side. According to our telemetry data, approximately 2% of end-users may have experienced this issue. We sincerely apologize for any inconvenience caused. We will be meeting with Microsoft's engineering and product groups to discuss this issue and recommend preventive measures for the future.

We are currently investigating an issue affecting the availability of the CodeTwo Admin Panel (emailsignatures365.codetwo.com) for some users across all regions. While some users may experience sign in difficulties, repeated attempts to sign in often succeed. All email signatures are being imprinted as usual. Signature editing remains accessible via app.codetwo.com. We appreciate your patience as we work to identify and resolve the issue.

The issue has been identified and a fix is being implemented.

The incident has been resolved. The root cause appeared to be a problem with Azure AD B2C. Restarting the affected services resolved the issue. We will be working with Microsoft to investigate the cause on their end, and to improve our preparedness for similar incidents in the future.

Microsoft has just reported on their status page (https://azure.status.microsoft/status) that they are experiencing network infrastructure issues in ALL Azure regions. If you are experiencing problems signing in to CodeTwo services at the moment, this is due to the above mentioned problem. As of now, our email signature imprinting services are not affected. We are actively monitoring this case and will keep you updated. Here's the official status from Microsoft: Network Infrastructure - Issues accessing a subset of Microsoft services We are investigating reports of issues connecting to Microsoft services globally. Customers may experience timeouts connecting to Azure services. We have multiple engineering teams engaged to diagnose and resolve the issue. More details will be provided as soon as possible.

Microsoft has just reported on their status page (https://azure.status.microsoft/status) that they have implemented networking configuration changes and have performed failovers to alternate networking paths to provide relief. Microsoft's monitoring telemetry shows improvement in service availability, and they are continuing to monitor to ensure full recovery. As of now, our email signature imprinting services are not affected. We are actively monitoring this case and will keep you updated.

Microsoft is now actively looking into this issue because the problem is on their end and affects connectivity to Azure services. Since this incident is not caused by CodeTwo services or infrastructure, we will mark it as resolved. Please refer to Microsoft’s Azure status page (https://azure.status.microsoft/status) or to issue MO842351 in Microsoft 365 admin center for further info and to monitor the status of their incident.

Microsoft is currently experiencing a problem with Microsoft 365 add-ins. Some add-ins are not working or working intermittently for a subset of users. All add-ins may also completely disappear from the Microsoft store occasionally. This is a Microsoft issue, and it's not limited to CodeTwo since we observe that other add-ins (even Microsoft native add-ins) are also affected. As a result of this problem, CodeTwo signatures might not be added in Outlook (desktop, OWA and mobile). Server-side (cloud) signatures are added normally. Only signatures added by the Outlook add-in might be affected. We're currently investigating this with Microsoft.

Microsoft engineering team has reported: We have discovered an issue with the backend service that provides add-ins, and we have engaged the backend team to address this ASAP. We are making progress and will let you know as we get more information from them. Just as a reminder - Outlook add-ins are currently affected due to a Microsoft issue, but server-side (cloud) signatures are added normally. Only signatures added by the Outlook add-in might not work.

The Outlook product group is currently deploying a fix to production. It should be available worldwide within 45 minutes.

The fix has been deployed. Some customers already notified us the problem is solved (so did our Q&A team and SOC). We are currently helping Microsoft monitor for any further issues.

We can confirm most of users are no longer affected by the issue. Some users might need to restart Outlook or refresh OWA to fully fix the problem.

This incident has been resolved.

Microsoft has officially confirmed the problem was on their end, and published communication about this incident to the Service Health section of the Microsoft 365 Admin Center for all affected tenants \(Communication ID: EX829938\). You can find it in the “Issue history” section. We also pasted it below: # _Users may be unable able to use already acquired add-ins within the Outlook desktop app and Outlook on the web_ _**User impact**_ _Users may have been unable to use already acquired add-ins within the Outlook desktop app and Outlook on the web._ _**More info**_ _The issue could have been seen in either the new or classic Outlook desktop app, as well as within the Outlook mobile app._ _**Scope of impact**_ _This issue may have affected any user of already acquired add-ins within the new or classic Outlook desktop app, Outlook on the web, and the Outlook mobile app._ _**Root cause**_ _A recent code change resulted in a bug affecting the service that gets add-in manifests into the host, which led to this impact._ _**Next steps**_ _-We’re \(Microsoft\) reviewing our code change testing and validation procedures to assess areas for improvement to avoid similar issues from being reintroduced in the future._ _- We’re \(Microsoft\) refining our understanding of the code bug by reviewing what the code change intended to do and analyzing how it led to this behavior, so that we can better understand how it caused the impact._

We are currently investigating email processing issues in Australia. A subset of users may experience delayed email delivery. Email signatures are added normally. The next update will be provided in 30 minutes or as events warrant.

It looks like Microsoft Exchange Online Protections’ performance is degraded at the moment which means it is unable to process messages sent from CodeTwo and other vendors in a timely manner. We have notified Microsoft Support about the problem. We are, however, seeing sings of recovery which might suggest the problem should be mitigated soon.

Microsoft has just published on X a status about a major outage of Microsoft 365: https://x.com/msft365status/status/1806149130663649355?s=46 as well as in the Microsoft 365 Admin Center (Issue ID: MO805755). This is a Microsoft issue. Please keep monitoring the communication from Microsoft to stay up to date with the mitigation steps during this outage.

We can see the situation has improved significantly. The queues are almost gone. Microsoft has just reported: “We determined that a recent change within Azure networking infrastructure led to impact. We reverted this change and we're monitoring our telemetry to ensure that affected services recover as expected.” We will continue to actively monitor the situation.

This incident has been resolved. Delayed emails have been delivered with signatures. For more information about the incident with Microsoft 365, please refer to the Microsoft 365 Admin Center (Incident ID: MO805755). All CodeTwo services are operational, emails are delivered without any delays and signatures are added as normal.

Below you can find the root cause analysis \(RCA\) from the Post Incident Report provided by Microsoft regarding this incident \(full report available in Microsoft 365 admin center, ID: MO805755\). In short, a change in a routing policy led to a configuration issue within Microsoft’s network routing infrastructure, causing impact to multiple Microsoft 365 services in the Asia-Pacific and Australia region. The incorrect routing policy that caused the outage was rolled back on June 27, 2024, at 2:02 AM UTC. All CodeTwo services remained healthy during the incident and all delayed emails have been delivered with signatures. **RCA FROM MICROSOFT’S POST INCIDENT REPORT \(Microsoft 365 admin center Issue ID: MO805755\):** _**Scope of Impact**_ _This issue could have impacted users globally, however, was mostly experienced by users hosted within Australia and Asia-Pacific due to the timeframe of impact overlapping with core business hours in those regions._ _**Incident Start Date and Time**_ _Thursday, June 27, 2024, at 1:18 AM UTC_ _**Incident End Date and Time**_ _Thursday, June 27, 2024, at 12:30 PM UTC_ _**Root Cause**_ _We’ve determined that a recent change caused a configuration issue within our network routing infrastructure, causing impact to multiple Microsoft 365 services._ _Specifically, in preparation for a planned network upgrade project, a change was made to our automation procedures supporting the upgrade. This change caused the automation to generate an incorrect routing policy that was not captured by our safety test systems in pre-checks for the project. Our WAN consists of two different planes for redundancy. When this incorrect routing policy was applied in the production network, a very large volume of traffic that is usually routed over plane one was routed over to plane two, and then sent back over plane one to reach its destination. This not only induced a very large latency increase, but it also caused congestion on both planes. The incorrect routing policy that caused the outage was rolled back on June 27, 2024, at 2:02 AM UTC._ _Due to the severity of the incident, several Microsoft 365 services experienced sustained impact and required further intervention to reach full recovery, which was attained on June 27, 2024, at 12:30 PM UTC._ _Extended impact for Exchange Online_ _A component used by the Exchange Online frontend proxy service became stuck due to the network conditions, preventing it from recovering once the network conditions were restored. Subsequently, manual recovery interventions were required due to code architecture patterns and the temporary exhaustion of automated recovery actions during the initial impact._

We are investigating a problem where Outlook add-ins in Microsoft 365 are not working or working intermittently. They also disappear from the Microsoft store occasionally. This is a Microsoft issue and it's not limited to CodeTwo since we observe that other add-ins (even those from Microsoft) are also affected. As a result, CodeTwo signatures might not be added in Outlook (desktop, OWA and mobile). Server-side (cloud) signatures are added normally. Only signatures added by the Outlook add-in are affected. We're currently investigating this with Microsoft.

Microsoft is now actively looking into this issue because the problem is on their end and affects all add-ins, including their own. Since this incident is not related to CodeTwo services or infrastructure, we will mark this incident as resolved. If you have any questions, feel free to contact Microsoft Support or our Technical Support Team: https://www.codetwo.com/company/contact

Microsoft Azure Active Directory B2C services are down for the second time today preventing users from signing in to CodeTwo Admin Panel (https://login.codetwo.com). We have received a notification from Azure Portal: [Microsoft] Engineers are investigating an alert for Azure Active Directory B2C in Europe. If you want to manage signature or autoresponder rules, just log in to CodeTwo from this website: https://app.codetwo.com. Apart from Admin Panel (https://login.codetwo.com), all other services are operational. Signatures are added as normal.

Microsoft provided the following update about the problem: We have identified the root cause of this issue and we have a potential mitigation plan in-progress.

This incident has been resolved.