Empist outage history

Due to the Cloudflare outage, we are experiencing sporadic issues with Deskware. Our team is actively monitoring the situation and reviewing the Cloudflare status page for updates. You can also follow Cloudflare’s status page for additional information: https://www.cloudflarestatus.com/

We are continuing to monitor for any further issues.

All Cloudflare services have been fully restored and platforms are fully operational.

We are currently experiencing delays with incoming and outgoing email due to an ongoing AWS outage. Internal email should continue to function normally. All delayed messages will be automatically delivered once the issue is resolved. For real-time updates on the AWS outage, please visit https://health.aws.amazon.com/health/status PLEASE NOTE: You may also be experiencing issues with other AWS dependent services.

AWS is in the process of resolving the service disruption. You can stay informed by visiting https://health.aws.amazon.com/health/status

It appears that email delivery is slowly being restored, and any queued messages are being sent out. It may take time to fully catch up.

AWS has reported continued progress in restoring affected services, and we are seeing steady improvement with emails being delivered.

We are seeing significant progress as services in the US-EAST-1 (N. Virginia) region continue to recover. Our teams are actively ensuring that all queued emails are being released and that normal mail flow is fully restored.

All queued email has been delivered and services are operational.

We are currently investigating connectivity issues to the Chicago datacenter. We will update as soon as we have more information.

While we are still investigating the root cause, we have been able to restore stable connectivity and all servers are back online. We will continue monitoring for the next few hours before setting this Incident to Resolved.

There have been no further issues and so we are setting this Incident to Resolved.

We are aware of some intermittent cloud performance issues that may be impactful for EMPIST cloud clients. We are currently investigating the source of the latency. We will update this incident within 2 hours or sooner when we have identified the source of the issue.

We are continuing to investigate this issue.

We have identified some potential drive issues that we are working on clearing. This is in progress and we should have an update within the hour. Most users are reporting that they can access the cloud and work as needed, though performance will not be optimized until we address any lingering issues with the drives.

The issues with the drives have cleared and performance appears to be restored. We will continue monitoring for the next couple hours to ensure that the performance is stabile and optimum within the infrastructure.

There have been no further reported issues and performance has remained stable.

We have identified an issue with Barracuda's Cloud Platform. The following is a link to the most current status updates from Barracuda's status page - https://status.barracuda.com/. Services are being restored but this is still affecting some of our customers, specifically in that they cannot access quarantine pages. We will continue monitoring this incident until all services have been confirmed as restored. Next update at 17:00.

Barracuda services continue to return to normal. It appears as though their Cloud S&V services are amongst the services returning to normal. We will continue monitoring this and provide another update at 18:30.

Services continue returning to full performance. We are removing the degraded performance status for Barracuda's Cloud S&V Services. We will check in again at 20:30.

We continued monitoring this through to resolution. All systems are performing optimally as expected.

We are currently investigating this issue.

The queues are returning to normal. We are continuing to monitor.

Our team is continuing to monitor the queues. All email is being delivered but with a delay.

All services have restored to normal performance. Our team continues to monitor the mail flow.

Our Spam & Virus Service has remained stable and so we closing this incident.

We are currently investigating an email delivery delay with our Spam firewalls.

All email has been flowing since a little before 5 PM and the queues appear normal again. We will continue monitoring for another hour to ensure there are no further issues before setting this to resolved.

Setting the current impact to None and the service to operational.

There have been no further issues detected so we are setting this incident to resolved.

We are currently investigating an email delivery delay with our Spam firewalls.

We are continuing to investigate this issue.

The issue has been identified and a fix is being implemented.

Incident has been resolved and all email has been delivered.

We are currently investigating a Hosted Exchange issue.

We have identified the issue and applying a fix. Customers have been able to login to web access.

The fix has been applied. All services are restored and we will continue to monitor closely.

Microsoft is investigating an issue where email is being sent to the Junk Folder. You can track updates via their Twitter Account at https://twitter.com/MSFT365Status.

The issue has been identified and the emails are being reprocessed. This is currently being monitored. The following are the two recent updates on the M365 account in Twitter: "We’re continuing to monitor as the remaining messages are being reprocessed. We’ll provide an estimated timeline as soon as one is available. Additional info is available via the admin center under EX258373." "The majority of emails should now be reprocessed however, the remaining emails may take some time to complete. We’ll provide the next update within the next hour. Additional info is available via the admin center under EX258373."

Microsoft did not provide a further update confirming final resolution, but per our monitoring, the issue appears to be fully resolved. We will continue to check Microsofts incident feed and advise if anything changes. Certainly, contact our support immediately if you suspect the issue is persisting. After changes administered by Microsoft, it's always suggested to close applications and restart them if you detect any lingering issues.

Microsoft is investigating an issue with email message visibility in Outlook. Outlook on the web appears to be unaffected. More details can be found under EX255650 in the MS admin app - https://aka.ms/M365AdminMobile or on their official Twitter Feed at https://twitter.com/MSFT365Status. EMPIST will continue to monitor and provide updates as they are available.

This update from Microsoft: "We’ve identified the root cause and are applying a fix, which will reach affected users over the next 3 to 4 hours. Users will need to restart the Outlook client to apply the fix after it's received. We expect to restore service to affected users by May 12, 2021, at 3:00 AM UTC." We'll continue watching for updates from Microsoft as they work to resolve this issue.

Here is the most recent update from Microsoft: "The deployment of our fix is progressing as expected and we continue to anticipate that it'll complete on Wednesday, May 12, 2021 by 3:00 AM UTC."

Microsoft confirmed that the issue was resolve and we continued monitoring to ensure there were no further issues. A restart of Outlook is recommended. Here is the official update from Microsoft. "We've finished the deployment of our fix and we've confirmed via monitoring and user reports that the issue has been mitigated. Users will be required to restart their applications for the changes to take effect. Please reference EX255650 for further details." We are closing this Incident at this time.

We are experiencing issues delivering email to Office 365. We are working with Microsoft to resolve the problem. All of the email being sent will be delivered when the issue is resolved.

Mail flow has been restored and the queues are flushing. All email should be delivered within the next hour.

We are continuing to monitor for any further issues.

There have been no further issues. We are closing this Incident at this time.

We are experiencing issues delivering email to Office 365 due to an upstream routing issue from our data centers. Our engineers are working on rerouting the messages. All external email is being queued and will be delivered once issue is resolved.

The upstream routing issue was resolved and all queued email has been delivered. Our engineers will monitoring to make sure everything is working properly.

No further issues. Incident has been resolved.

Microsoft identified an issue earlier today at 1:10 PM CDT intermittently impacting some users ability to access some Microsoft 365 Services. They have identified root cause and will be providing another update at 5:30 PM CDT. You can monitor progress of this outage on their site by visiting https://status.office.com/. We will advise when they have provided another update confirming that all services are fully up and running. Here is their most current update: Microsoft 365 Service health status Title: Can't access Microsoft 365 services User impact: Users may be unable to access Microsoft 365 services. More info: Users may see impact to the following Microsoft 365 services: SharePoint Online/OneDrive for Business - Services are healthy Microsoft Teams - Services are healthy Microsoft Forms - Services are healthy Microsoft Threat Protection - Services are healthy Microsoft Intune - Services are healthy Exchange Online - Services are healthy Outlook.com - Services are recovering Admin center - Services are recovering Current status: We're conducting final validation measures on the remaining impacted services to ensure that we have reached full recovery. Scope of impact: Any user who is attempting to access the impacted services could be impacted by this event. Start time: Wednesday, October 7, 2020, at 6:10 PM UTC Root cause: A change to network infrastructure caused impact within Microsoft 365 services. Next update by: Wednesday, October 7, 2020, at 10:30 PM UTC

Microsoft confirmed that the outage is now resolved. Here is the most recent update from their status page. We will continue to monitor for another 2 hours to ensure that there are no further reported issues from users. Microsoft 365 Service health status Title: Admins may experience delays or be unable to access the admin center User Impact: Admins may have experienced delays or been unable to access the admin center. More Info: Additional details can be found in the admin center under MO223787. Current status: We've completed our mitigation actions and confirmed via extensive monitoring that the issue has been resolved. Scope of impact: Admins in the North America region may have experienced delays or been unable to access the admin center. Start time: Wednesday, October 7, 2020, at 6:10 PM UTC End time: Wednesday, October 7, 2020, at 10:30 PM UTC

There have been no other reported cases and all services seem to be fully restored. We are closing this outage at this time.

We've identified an issue that is affecting some of our hosted exchange clients. We are currently working to resolve the issue, however email may be intermittent while we work towards a resolution.

The degradation of service appears to have been caused by a storage issue on a Lun in our datacenter. The impact is minimal isolated to a small percentage of users and limited to OWA and public folders. We are continuing to troubleshoot the Lun in question and will advise when full service is restored.

The issue with the Lun has been addressed. The Exchange Server is online. All Exchange functionality has been confirmed working. We will continue monitoring for the next 2 hours to ensure that there are no further issues.

We are continuing to monitor for any further issues.

There have been no further issues and we are setting this to resolved.

We are experiencing delays with outbound delivery of email through our Spam & Virus servers. We are currently investigating the issue.

We have identified the issue and this is also impacting inbound email. Our engineers are actively troubleshooting.

We have applied a fix to the devices and mail flow has resumed. We are continuing to monitor.

The services have been stable for over 3 hours now. Our team will continue to monitor but please report any issues to support@empist.com.

We are currently experiencing an issue with email delivery on our Hosted Exchange. This issue is due to a major fiber cut between our data centers. Crews are already dispatched and working on repair. All emails will be delivered as soon as connections can be restored.

Mail flow is back to normal. All emails that were queued have been delivered and there is no need to resend emails. We are still experiencing an issue with the fiber, but our engineers delivered the email using another connection method. We will continue to monitor until the fiber has been repaired.

Mail flow continues to work properly and crews are still working on resolving the issue with the Fiber. Our team continues to monitor closely.

The fiber was repaired and all connectivity was restored at approximately 5pm Central time on 11/19.

Microsoft experienced an outage this morning that impacted some of our clients who utilize O365. While the outage did not directly impact any services that EMPIST provides to you, indirectly it impacted clients access to Sharepoint, OneDrive and email. For this reason, we wanted to add it here to our status page and will continue to monitor it for the next couple hours. The first reports of issues online started about 9:15AM and things came back up around 9:45AM. There may still be lingering issues, but at least now Microsoft is aware and investigating. Below is the link to the official Microsoft 365 Status Page as well as 3rd party sites that track reported incidents: https://status.office365.com/ https://downdetector.com/status/office-365/ https://outage.report/ Please feel free to contact our support team if you have any additional questions or concerns.

There have been no further incidents and the issue appears to be fully resolved. We are setting this outage to Resolved, but will reopen if we detect anything else.

Barracuda identified intermittent issues with their email encryption services earlier today. They have a fix and are currently implementing / testing it. We will continue to monitor on our end and provide an update as soon as there is one available. Here is the link to Barracuda's official incident update from their status page: https://status.barracuda.com/

Barracuda resolved the issue and all systems have returned to full operational status. We continued to monitor as well and are now setting this outage to resolved.

Webroot DNS protect service is experiencing an outage at their east coast data center.  This is affecting DNS lookups for the majority of our clients who are using our DNS protect service, and SecureAnywhere DNS Protect service.  We are currently working on implementing a temporary workaround for those clients who are affected and will continue to monitor the situation and will update when the vendor has resolved the underlying issue.

Webroot officially set this issue to resolved at 12:53 pm CT. We have been monitoring on our end since their last update and are confident we can now officially set this Outage to resolved. Here is the link to Webroot's Incident Page - https://status.webroot.com/incidents/t2l22tkrq2vd.