Störungsverlauf von Macstadium

The team has identified an issue affecting private cloud functionalities in Las Vegas, and is executing emergency maintenance to address the issue. Until this is resolved, customers may experience temporary outages in the following features: - Orka clusters in Las Vegas - ASAv Firewalls in Las Vegas - Virtual Private Cloud in Las Vegas

The cluster has been updated to a stable state. We are monitoring the system to ensure it remains stable.

This incident has been resolved.

The team has identified an issue with private cloud functionalities in Las Vegas. This affects a portion of the following customers: - Orka customers with a cluster in Las Vegas - Cisco ASAv Firewall customers running in Las Vegas - All Virtual Private Cloud customers with a tenant in Las Vegas During this time, customers may not be able to access these resources. The team is working directly with the vendor on a remediation.

This incident has been resolved.

An incident has occurred with our hosting provider, which is impacting our ability to communicate through tickets. While everything within the MacStadium portal itself is operational, our responses to ticket messages may be delayed as a result of the issue with our provider. We are monitoring the issue with our hosting provider and will provide an update as soon as one becomes available. We appreciate your patience while our provider is actively working to resolve the issue as quickly as possible.

The issue with our hosting provider has been resolved and our communication capabilities have been restored. There is still a backlog of messages that will be sent out, but everything is operating as expected. Thank you for your patience as our provider worked to resolve issues impacting our ticket communication.

We are aware of an issue preventing users from accessing the Support Center and Bare Metal pages within the MacStadium customer portal. Although some customers might not be able to view the Bare Metal page, customers should still be able to connect to their bare metal services, without issue. Our team is actively investigating and working to resolve the issue as quickly as possible. In the meantime, customers can create support tickets by emailing [email protected]. We will provide updates as we have more information. Thank you for your patience. -MacStadium Support Team

We are continuing to investigate this issue.

We have identified the cause of the issue preventing users from accessing the Support Center and Bare Metal pages within the MacStadium customer portal and are actively working on implementing a fix. Although some customers might not be able to view the Bare Metal page, customers should still be able to connect to their bare metal services, without issue. In the meantime, customers can create support tickets by emailing [email protected]. We will provide updates as we have more information. Thank you for your patience. -MacStadium Support Team

A temporary solution has been implemented which will allow users to view the Support Center and Bare Metal pages within the MacStadium portal. Customers can once again create new support tickets and view bare metal details. Please note that the temporary solution will still prevent some users from viewing old tickets as well as tickets created through bare metal purchase requests. We expect to have this part of the problem fully resolved by the end of the week. We will continue to monitor the temporary solution and will provide further updates regarding a full resolution as they become available. Thank you for your patience as we work through this issue. -MacStadium Support Team

The issue preventing users from accessing the Support Center and Bare Metal pages within the MacStadium customer portal has been resolved. All customers should be able to create tickets and view bare metal within the customer portal, without issue. Thank you for your patience while we worked through this issue.

Dear Customers, We are aware of an issue affecting the Private Cloud service in our Atlanta, GA USA data center. At this time, the service is hard down, and customers cannot access the system. Our team is actively investigating and working with our vendor to resolve the issue quickly. We will provide updates as we have more information. Thank you for your patience. MacStadium Support Team

Dear Customers, We have identified an issue with the VSAN that has caused the Private Cloud service in our Atlanta, GA USA data center to be down. Our team is actively working with our vendor to take down the environment for repair. We will continue to provide updates as we have more information. Thank you for your patience. MacStadium Support Team

Dear Customers, We continue to work actively with our vendor to repair and restore our Private Cloud service in our Atlanta, GA USA data center as quickly as possible. We will continue to provide updates as we have more information. Thank you for your patience. MacStadium Support Team

Dear Customers, During a standard VSAN expansion process, the Verge system experienced unexpected behavior, resulting in the environment going down. Verge has outlined a five-step plan to recover the environment, including a full disk scan, journal walk process, repair cycle, diagnostic review, and VSAN connection. At this time, the first step (full disk scan) is approximately 70% complete. We are actively engaged with Verge leadership to work towards a resolution as quickly as possible, but unfortunately, an ETA is not known at this time, and we believe there is likely quite a way to go to bring Private Cloud back online. We will provide additional updates as we have more information to share. Thank you for your patience. MacStadium Support Team.

Dear Customers, Our efforts to recover the Atlanta-based Private Cloud environment continue as we work hand in hand with Verge through their restoration process. The system is still working through the first step (full disk scan)--we anticipate another 10 hours for this first step to complete. An overall restoration ETA is unknown; we believe it will likely take some time to bring Private Cloud back online. We will provide additional updates as we have more information to share. Thank you for your patience. MacStadium Support Team.

Dear Customers, Our efforts to restore the Atlanta-based Private Cloud environment are progressing, and we have now reached Step 4: Diagnostic Review in VergeIO’s five-step recovery plan. At this stage, the system is being analyzed to identify any gaps or unrecoverable data blocks. We remain actively engaged with VergeIO leadership to work toward a resolution as quickly as possible. Unfortunately, an exact ETA for full restoration is still unknown, but we will continue providing updates as we have more information. Thank you for your patience. MacStadium Support Team

Dear Customers, The diagnostic review is ongoing, and the VergeIO team is actively working to recover as much missing data as possible. Our teams remain fully engaged to support the recovery efforts and bring the system back into production as quickly as possible. We do not have a definitive ETA for full restoration, but we will continue to provide updates as more information becomes available. Thank you for your patience and understanding. MacStadium Support Team

Dear Customers, The diagnostic review is still ongoing, and the VergeIO team has successfully recovered a portion of the remaining missing data. Efforts continue to restore as much data as possible before bringing the system back into production. We remain actively engaged in the recovery process and will provide further updates as more information becomes available. At this time, we do not have a definitive ETA for full restoration. Thank you for your patience and understanding. MacStadium Support Team

Dear Customers, Private Cloud/VergeIO services in ATL1 have been restored. We have reached out individually to affected customers regarding the restoration of their environments. MacStadium Support Team

Dear Customers, Private Cloud/VergeIO services in ATL1 have been restored. We have reached out individually to affected customers regarding the restoration of their environments. MacStadium Support Team

Dear Customers, We are aware of an issue affecting the Customer Portal ticketing system. Some users may be unable to create new tickets or view and respond to existing ones. Our team is actively investigating and working to resolve the issue as quickly as possible. We will provide updates as we have more information. Thank you for your patience. MacStadium Support Team

Dear Customers, The issue affecting the Customer Portal ticketing system has been resolved. You should now be able to create, view, and respond to tickets as expected. We are actively monitoring the system to ensure continued stability. Thank you for your patience, and we apologize for any inconvenience this may have caused. MacStadium Support Team

Dear Customers, Since implementing the solution, we have observed no further issues with the Customer Portal ticketing system. As a result, we are closing out this incident. Thank you for your patience and trust. MacStadium Support Team

As Storm Éowyn approaches the Dublin data center this Friday, we want to assure you that we are fully prepared to maintain seamless operations at our data center. Our co-location provider and our team have robust measures in place, including: Backup power systems, including generators and fuel reserves, to ensure uninterrupted service. Redundant network connections to maintain connectivity. The Irish government has however advised that potential danger to life is possible whilst travelling. After careful consideration, MacStadium has made the decision not to have onsite support at our Dublin data center from Friday 6am GMT until Friday 1pm GMT in order to protect our employee's safety. We continuously monitor weather conditions and have contingency plans in place to ensure your services remain operational and secure. Should you have any concerns or need assistance during this time, please don’t hesitate to reach out to our support team. Thank you for trusting us with your business.

This incident has been resolved.

As winter weather approaches the Atlanta, Georgia area this Tuesday and Wednesday, we want to assure you that we are fully prepared to maintain seamless operations at our data center. Our co-location provider and our team have robust measures in place, including: • Backup power systems, including generators and fuel reserves, to ensure uninterrupted service. • Redundant network connections to maintain connectivity. • On-site and on-call personnel ready to respond swiftly to any weather-related challenges. We continuously monitor weather conditions and have contingency plans in place to ensure your services remain operational and secure. Should you have any concerns or need assistance during this time, please don’t hesitate to reach out to our support team. Thank you for trusting us with your business.

This incident has been resolved.

As winter weather approaches the Atlanta, Georgia area this Friday and Saturday, we want to assure you that we are fully prepared to maintain seamless operations at our data center. Our co-location provider and our team have robust measures in place, including: Backup power systems, including generators and fuel reserves, to ensure uninterrupted service. Redundant network connections to maintain connectivity. On-site and on-call personnel ready to respond swiftly to any weather-related challenges. We continuously monitor weather conditions and have contingency plans in place to ensure your services remain operational and secure. Should you have any concerns or need assistance during this time, please don’t hesitate to reach out to our support team. Thank you for trusting us with your business.

We are continuing to monitor for any further issues.

This incident has been resolved.

Valued MacStadium Customers, On Dec 19th, 2024 from 02:30 UTC to 04:30 UTC, the MacStadium Engineering Team will be conducting network infrastructure upgrades. All hosts in the Las Vegas Data Center will experience a loss of network connection during the upgrade process. The maintenance is rolling, so no server should experience more than 15 minutes of downtime. If you have questions, please contact MacStadium Customer Support by opening a ticket via the Customer Portal or email us at [email protected]. Additionally, visit MacStadium’s Status Page (status.macstadium.com) for updates before, during, and after this change.

Apologies this was sent in error

This incident has been resolved.

We are expecting inclement weather related to Hurricane Helene at our Atlanta Data Center. Our data center partner is monitoring all critical infrastructure at the site. All generators have been exercised and fuel levels are at their maximum levels. MacStadium and our DC partner’s staff will be on site to provide support during this weather event. Please open a ticket should you experience an issue.

This incident has been resolved.

Our Las Vegas Data Center is currently experiencing Network issues. Our Engineers are looking into this and we will provide updates as we work through the issue. Thank you for your patience while we work this issue.

Our Las Vegas Data Center Network issues have been resolved. Our Engineers will continue to monitor. Please contact us if you experience any issues. Thank you.

There is a power interruption affecting some of our customers in our Atlanta facility. We are currently working with our data center partner, DataBank, to restore power.

Our data center colo partner is actively engaged and troubleshooting the power interruption at the site. We will post the ETA when it is provided to us.

Our data center colo partner has isolated the failure to a specific UPS. They are currently working to remediate the failure. There is still currently no ETA.

Our data center colo partner has restored power to the facility. We are working to confirm power has been restored to all affected customers.

We have confirmed power has been restored to all affected customers at this time. Please open a ticket if you’re experiencing any issue to your Atlanta-based servers.

We are experiencing a broad outage of customers' ability to access their cloud environments in the Atlanta data center. We are actively investigating and will provide additional updates.

All cloud environments are still operational. Only the ability to communicate with them has been affected. We have narrowed down the issue to a network configuration. We expect to correct the issue near term.

We have determined root cause and are working to resolve the issue.

We are implementing the fix now and customers are coming back online. We should have all systems up shortly.

A fix has been implemented. All services are online, and we are monitoring the results. If you are still experiencing issues, please open a ticket via the Customer Portal.

We wish to advise our Las Vegas customers that we are experiencing carrier failures to and from our Las Vegas data center at this time. We expect no customer impact apart from operating at reduced capacity.

Our data carriers have provided us with an ETA of 13:00 UTC/ 05:00 PT when they expect the restoration of data capacity. We apologise for any inconvenience related to this reduced capacity.

We wish to advise our Las Vegas customers that we have been informed by our data carriers that full connectivity capacity has been restored. We will continue to closely monitor the situation and once again we apologise for any inconvenience related to this issue.

We wish to advise our Las Vegas customers that the traffic connectivity to our Las Vegas data center remains at full connectivity capacity, we will step down these notifications but we will continue to monitor. We would like to thank our customers for their patience and once again apologise for any inconvenience related to this issue.

One of the Las Vegas ISPs has experienced a hardware failure, causing the site to experience reduced Internet capacity. Currently, traffic has converged to alternate ISPs to minimize impact. The Estimated Time of Repair from the ISP is 2030 UTC.

The ISP identified the piece of hardware that failed and is in the process of replacing it now. No updated ETA has been provided.

Services have been routed off the failed hardware, and no further impact is expected. Normal capacity has been restored.

This incident has been resolved.

Different aspects of our Customer Portal, including creating Support Tickets and viewing individual host information, are currently experiencing a service interruption. Our Developers are actively investigating this issue at present.

We are continuing to investigate the issue. At this time as Support Tickets created through the Customer Portal are blocked, please either email your support request to [email protected] or reach out to us through LiveChat.

Our Developers have resolved the service interruption in our Customer Portal, and we have confirmed that both server details and Support Ticket creation are currently functioning as expected. All systems are now Operational.

There is a network issue in Dublin that is affecting multiple customers currently and may cause loss of connectivity. We have multiple Network Engineers investigating the issue currently and we will update as soon as we have more information.

Our Network Engineers are continuing to Investigate the issue and look for a solution. We will update you once we know more information.

Our Network Engineers were able to identify the issue that caused the Network impact and resolve it. All traffic has been restored. Our Engineers are continuing to monitor the network.

This incident has been resolved.

The Atlanta Data Center has had a power disruption which will affect a small number of customers. MacStadium has engaged the Data Center provider and power will be restored momentarily. We apologise for any inconvenience. Kind Regards Luke Kavanagh MacStadium Incident Manager

MacStadium have worked with our data center provider to restore power to all affected customers, Macstadium are continuing to restore service to any remaining affected customers. Once again our apologies for any inconvenience.Kind Regards Luke Kavanagh MacStadium Incident Manage

The Atlanta Data Center power and services are now fully restored.We apologise for any inconvenience & we will continue to work with our data center provider to mitigate similar events from reoccurring.

This incident has been resolved.

MacStadium Engineers have discovered a Network issue that may cause 10% or more packet loss for multiple customers at the Dublin data center. The engineers are currently investigating the issue and working on remediation steps. We will update the message once we have more information.

The issue has been resolved. We do not expect any further issues, but we will continue to monitor the situation.