Umbrellar 장애 기록

We want to inform you that we are currently aware of an issue affecting Azure Front Door, which may be impacting your service availability or performance. Microsoft has acknowledged the incident and is actively working on a resolution. You can follow their updates directly via https://azure.status.microsoft/en-us/status. We are closely monitoring the situation and will keep you informed as we learn more. Your experience is important to us, and we appreciate your patience and understanding.

At this stage, Microsoft anticipate full mitigation within the next four hours as they continue to recover nodes. This means we expect recovery to happen by 12:20PM on 30 October 2025. We will provide another update on our progress within two hours, or sooner if warranted

Microsoft has deployed its last known good configuration and recovery is progressing. They are actively restoring nodes and rerouting traffic through healthy systems. While improvements are being seen, some requests may still hit unhealthy nodes, causing intermittent failures or reduced availability. The recovery process involves reloading configurations and rebalancing traffic across a large fleet. This is being done gradually to maintain stability and avoid overload as dependent services come back online. Microsoft expects full mitigation by 12:40PM NZDT on 30 October 2025, and will communicate if recovery completes sooner. Customer configuration changes remain temporarily blocked by Microsoft to prevent interference with recovery efforts. They will notify when this restriction is lifted.

We are continuing to work on a fix for this issue.

We’re pleased to confirm that the recent Microsoft Azure service disruption has now been fully resolved. All affected Azure services are now operating normally, and recovery activities have been completed across all impacted regions. Our own monitoring also confirms that your hosted environments are stable, accessible, and performing as expected. The outage was linked to a configuration change within Azure Front Door (AFD) that caused DNS and connectivity issues across several regions. Microsoft promptly rolled back the change, restored normal operation, and is now conducting a full review of the event.

We are currently experiencing a service disruption affecting multiple customers on VMware in Airedale DC. Our platform and network engineering teams are actively investigating the issue in parallel to identify the root cause and implement a resolution as quickly as possible. We will provide updates as more information becomes available.

We are continuing to investigate this issue.

Network engineers have isolated the issue and have completed a change to resolve this issue. An RCA will be conducted and shared with the next 48 hours

We identified an issue as a result of an unexpected host failure that occurred in our data centre this morning, which resulted in virtual machine downtime, slow performance, or loss of network connectivity. Initial analysis indicates that the root cause may be related to power supplies on the physical hardware. Our engineering team has: All Virtual Machines have been automatically migrated to a Healthy Host. Additional Hosts have been made available into the cluster to ensure hosting capacity. Further investigation is underway to determine root cause. Continued monitoring to prevent recurrence. We are continuing to monitor system performance and will follow up with a detailed post-incident report within 24-72 hours including root cause and corrective actions.

Since the initial incident, another host has failed. In response, we have added an additional host to the pool to ensure continued stability and capacity. Our engineering team is actively investigating this second failure to determine whether it is related to the original issue or an isolated event.

We are continuing to monitor for any further issues.

This incident has been resolved.

Umbrellar Engineers are aware of a network outage affecting the Auckland region and are investigating the issue. Further updates to come.

The network issue has now been resolved. There was an outage affecting many customers in the Auckland region. Any customers still impacted, please contact 0800 344 493.

We are aware of services being down in our Christchurch datacentre and are in contact with the providers to restore service as quickly as possible. Further updates to come.

Services are beginning to recover. Still under investigation.

We are working with our Datacentre provider to understand the cause of the outage. All services should be operational again now and are being monitored. Please contact help.apac@pax8.com if you have any unresolved issues.

All services in Christchurch are operating normally again.

Dear Partners, As you know there is a State of Emergency in Auckland/Northland NZ, due to Cyclone Gabrielle, there may be a delay in responding to phone support or email. This is to ensure the safety of our staff and are required to follow the guidance provided by the local authorities Rest assured we will endeavor to your request as soon as possible. Stay safe. Thank you. help@mycsp.io 0800 344 493 ivr 3

This incident has been resolved.

We have identified a login/authentication issue with the Umbrellar MyCSP Portal impacting users that cannot self-service. We are fixing it as soon as we can and are working closely with Microsoft. If you want to place an order, please let us know the order details by emailing us at help@mycsp.io and we will assist you in provisioning them as soon as possible.

The login/authentication issue to the Umbrellar MyCSP Portal is now fixed. If you are still facing issues accessing MyCSP, please request your Global Admin to log in and accept the request for application permissions. We will continue to monitor this issue to ensure it was fully resolved. If you have any questions, please let us know by email to help@mycsp.io

We are continuing to monitor for any further issues.

We have verified that the login/authentication issue to Umbrellar MyCSP Portal is now resolved. We apologize for the service disruption. If you have any questions, please let us know by email to help@mycsp.io

What is it? The OpenSSL Project will release a security fix (OpenSSL version 3.0.7) for a new-and-disclosed CVE on Tuesday, November 1, 2022. This CVE is categorized as “CRITICAL” and affects all OpenSSL versions after 3.0. Who is impacted? Users running newer operating systems (OpenSSL versions after 3.0) and application stacks like NodeJS. Linux Distro OpenSSL Version CentOS Linux release 7.9 1.0.2 CentOS 8 (1.1.1) CentOS Stream 9 (3.0.1) Debian 11 (bullseye) (1.1.1) Eneavour 2022.09.10 (1.1.1) Fedora 34 (1.1.1) Fedora 35 (1.1.1) Fedora 36 (3.0.5) Fedora Rawhide (3.0.5) Kali 2022.3 (3.0.5) Linux Mint 21 Vanessa (3.0.2) Mageia 7 (1.1.1) Mageia 8 (1.1.1) Mageia Cauldron (3.0.5) OpenMandriva 4.3 (3.0.3) OpenMandriva Cooker (3.0.6) OPNsense 22 1.1.1 OpenSuSE Leap 15.2 (1.1.1) OpenSuSE Leap 15.3 (1.1.1) OpenSuSE Leap 15.4 (1.1.1) Proxmox 6 1.1.1 Redhat ES 9 3.0 Rocky Linux release 9.0 (Blue Onyx) 3.0.1 Slackware 14 1.0.1 Ubuntu 20.04 (1.1.1) Ubuntu 22.04 (3.0.2) Node.js v18.x and v19.x use OpenSSL v3. Therefore these release lines are impacted by this update. References https://isc.sans.edu/forums/diary/Upcoming+Critical+OpenSSL+Vulnerability+What+will+be+Affected/29192 https://www.globalsign.com/en/blog/urgent-patch-openssl-november-1-avoid-critical-security-vulnerability https://nodejs.org/en/blog/vulnerability/openssl-november-2022/

We are continuing to work on a fix for this issue.

This incident has been resolved.

There is a known issue with Microsoft Azure and Ubuntu 18.04 LTS servers that have been updated recently where DNS resolution is broken. Updates can be seen here - https://azure.status.microsoft/en-us/status via Microsoft. Users of Ubuntu 18.04 LTS are recommended to review the Ubuntu bug page and apply the recommended fix.

This incident will be closed as it is isolated to a particular OS and version. If you are impacted by this bug, please visit the Ubuntu Launchpad Bug Tracker for workarounds - https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1988119

We are currently experiencing issues with our phone system provider and inbound phone calls via the 0800 number may not go through. Our team are working with the provider on rectifying this issue. Please use help@umbrellar.com for queries if you are unable to reach our 0800 number. More updates to follow.

The issue has been identified as a fault with Microsoft Teams and its supported services. Microsoft are tracking this with the ID TM402718 and it appears to be global in nature. Once more updates are provided under TM402718 we will provide further updates.

The Microsoft Teams global outage under service health notice TM402718 is gradually coming back online. The 0800 number is now operational and we are continuing to monitor its health.

Phone services are working normally again.

Core networking services have been disrupted in our Auckland datacentre and this is being investigated and will be resolved as soon as possible. Further updates to follow.

The issue has been identified as the core router in Auckland flapping causing grey failure response. Routing engines have been restarted, and the time issue is related to convergence. Further updates to follow.

Networking services have recovered and are under observation by the network engineering team. If you are still experiencing problems, please notify help@umbrellar.com or phone 0800 344 493.

A restart of the line card in the degraded core router helped restore networking services. Network engineers are still monitoring the situation. If you are still experiencing problems, please notify help@umbrellar.com or phone 0800 344 493.

This incident has been resolved at approximately 12:45pm and subsequent monitoring checks are healthy.

We are investigating an issue with the Umbrellar Cloud Portal that is impacting some users. We will provide an update as soon as more information becomes available. If you have any questions, please let us know by email to help@umbrellar.com.

The portal is now up and running. Customers should be able to log in to the Umbrellar Cloud Portal and Salesforce Community. We will continue to monitor the portal to ensure it is stable before resolving this issue. If you have any questions, please let us know by email at help@umbrellar.com.

We are continuing to monitor for any further issues.

This incident has been resolved.

We have determined an issue with Umbrellar MyCSP Cloud Portal resulting in the portal being offline. We are working to resolve this issue and will provide an update when we have further information. If you have any questions, please let us know by email at help@mycsp.io.

This incident has been resolved.

We are investigating an issue with signing in to the MyCSP Portal that is impacting some users. We will provide an update as soon as more information becomes available. If you have any questions, please let us know by email to help@mycsp.io Thank you for your patience.

We are continuing to investigate this issue.

This incident has been resolved.

We are getting multiple reports of clients and end users being unable to view websites hosted by us. All indications point to a DNS fault occurring within Spark. We will continue to monitor the situation and provide updates as things change.

Further investigations show the DNS outage experienced by some end-users is isolated to Spark as an ISP and DNS zones hosted with the DNS provider Discount Domains. Umbrellar recommends clients of Discount Domains contact their support department https://help.discountdomains.co.nz/s/ for further information on the outage. The Umbrellar network is operating normally and the DNS zones are accessible via other ISPs.

We are continuing to work on a fix for this issue.

We've determined that an upstream API is returning errors which is stopping some customers from logging in to MyCSP portal. This is due to the global outage with Salesforce which they are currently investigating. We will provide an update when we have further information. If you have any questions, please let us know by email to help@umbrellar.com.

The upstream API is now functioning correctly and customers should be able to log in to the MyCSP Portal. We will continue to monitor the API to ensure it is stable before resolving this issue. If you have any questions, please let us know by email to help@umbrellar.com.

We have verified that the issue with Salesforce that affected our services has been fixed, and we have also verified stability of the upstream API and MyCSP portal availability. The disruption is now resolved. We apologise for the service disruption. If you have any questions, please let us know by email to help@umbrellar.com.

Microsoft is investigating an issue impacting Azure Active Directory. More information and updates can be found on the Azure Status page at https://status.azure.com.

This incident has been resolved.

We are investigating an issue with the MyCSP Billing System that is impacting invoices generation. We will provide an update when we have further information. If you have any questions, please let us know by email to help@umbrellar.com.

We've identified an issue that impacted invoices generation on the MyCSP Billing Platform. We are working to resolve this issue and will provide an update when we have further information.

All invoices on the MyCSP Billing Platform were successfully generated so disruption of service is now resolved. We apologise for the inconvenience. If you have any questions, please let us know by email to help@umbrellar.com.

The Umbrellar Support Team is currently working with 3rd Party vendors to resolve an issue with resolution of the Umbrellar.io domain. This is affecting access to the Azure Stack portal as well as external access to Storage Accounts

This incident has been resolved. Domain resolution for Umbrellar.io is working as expected. No further issues are expected.

Please take note that due to maintenance being performed by Vector the Umbrellar Auckland Datacenter will be affected by a power outage starting at 10pm Today (19/10/2020). The outage will last until 2am Tomorrow morning (20/10/2020). No impact is expected and the Datacenter Staff will be on site to monitor UPS and Generator power during this time.

The maintenance completed at 1:38AM. Power has been restored and this Incident is now closed.