Luxsci outage history

An issue is affecting some servers at our Phoenix datacenter. Our data center provider Oracle is investigating the issue. Users may experience degraded services during this time.

Oracle has updated us stating that their engineers are actively working to identify the root cause and restore normal service.. We do not yet have an ETA for when service will be fully restored.

Status: Resolved Message: Oracle has confirmed the issue affecting our Phoenix data center has been resolved. All services have returned to normal operation. We are sorry for any inconvenience this issue may have caused.

Proofpoint, our Premium Email Filtering provider, is experiencing issues that are affecting mail delivery. We have contacted Proofpoint regarding the issue. They are aware of the issue and are currently investigating it. We will update this post once we have additional information.

This incident has been resolved.

Due to a side effect of the Amazon Web Servers (AWS) outage from yesterday, some of LuxSci's customer domains may have had archival disabled for a short time frame. This occurred between 9:19pm ET and 2am ET.During this time email may not have been archived. The problem has been resolved and archival has been re-enabled on all domains that had it previous to this issue.

Please be aware you may see bounces on your LuxSci account that are with respect to emails going to @us1.journaladdress.com. These bounces are emails being sent to the Premium Archival Service. This service uses Amazon Web Service for some of it’s operations. Currently, Amazon is experiencing a world wide outage which is in turn causing these bounces. Once Amazon comes back online, the bounces should stop coming in.

We are aware of the cause of the issue and are monitoring AWS.

Emails to @us1.journaladdress.com are now being delivered as normal. Old emails that are queued will start to be delivered to our customer's archives as our system works through the queued emails.

A subsection of users may be experiencing intermittent delays receiving inbound email messages. The delay is currently under investigation.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

You may be experiencing intermittent delays in receiving email being filtered by Proofpoint. We are aware of the delays and have a priority, escalated support ticket open with Proofpoint on this issue. We will update this notice as more information becomes available from Proofpoint.

The delay in delivery emails flowing through Proofpoint has been resolved. It may take a little while for Proofpoint to deliver all the email that was queued during this issue. No emails were lost during this issue.

MobileSync users may experience intermittent sync delays with email or calendars. These delays are only impacting MobileSync and do not impact other integration options.

The MobileSync issue has been resolved.

We are currently investigating an issue with ProofPoint where they are doing scheduled maintenance, which should not be impacting mail flow, however, message delivery is being delayed.

Mail flow through ProofPoint resumed around 12:40 PM EST, however, the ProofPoint console won't be available until ~12:00 AM UTC.

ProofPoint Maintenace is now complete. Mail flow did resume yesterday around 12:40 PM EST

We are currently investigating reports that some emails are not appearing in search results in the Sonian Email Archiving service. All emails are still currently being received and backlogged for journaling, there has been no data loss. Once Sonian deploys a patch to the service, all historical emails that are not displaying will be properly journaled after a delay. We will continue to post updates as they are made available.

The Sonian Email Archiving issue has been resolved and emails are now being journaled normally.

We are currently investigating an issue with our Shared WebMail service. We will update this notice once it has been resolved.

An issue with one of our WebMail servers caused email to be unavailable to users who were logged into the affected WebMail server. The issue was investigated by our operations team and diagnosed as a database issue on the server. The database issue was resolved, and the server is now responding properly. A fix was put in place to prevent the database issue from recurring.

LuxSci is currently experiencing an issue where Two Factor Authentication SMS messages are not being delivered to the recipient's phones. This affects some of our customers' ability to log into their accounts if they have SMS chosen as the Two Factor Authentication method. We are currently investigating the issue with our SMS provider.

Our SMS provider has implemented a solution to the SMS Two Factor Authentication issue some of our customers have been experiencing. The solution will need to propagate down to the phone carriers, which can take up to 3 days. Some phone carriers are already working again with our SMS notifications.

The SMS solution from our SMS provider has propagated to the major phone carriers. All customers should now be able to receive SMS Two Factor Authentication codes on their phones again.

This incident had been identified and technicians are currently working to mitigate the problem. We will let you know when this problem is resolved.

With the successful installation of additional cooling systems, we are continuing to monitor the issue for servers that have been affected by this problem.

Our datacenter reports that services have been restored. We will continue to monitor to ensure continued stability of our services.

Oracle has reported this data center incident has been fully resolved earlier this morning. To our knowledge, only three customer servers were actually impacted.

We are experiencing network latency issues in our Ashburn, Virginia, data center. The network latency could cause webmail and website pages to load slowly. We are investigating the issue and hope to have it resolved soon.

Our network is returning to normal operations. We will continue to monitor the situation.

This issue is resolved.

There is currently an issue with Avast Anit-virus's Web Shield blocking access to LuxSci.com domains. When you try to go to https://luxsci.com or https://webmail.luxsci.com Avast will prevent the connection. Avast will then show a popup stating the connection was blocked due to LuxSci.com being a phishing site. This is a false positive error. We have confirmed our website does not contain any phishing attacks. We have started the process with Avast to get this false positive reading removed from Avast Anti-virus software. Until the issue is resolved, users will need to add an exception within Avast in order to get to LuxSci's websites. Here are instructions on how to add the exception in Windows. 1: Open Avast. You can click the Avast icon in your system tray next to your clock on the Taskbar 2: Click the 'General' button 3: Click 'Exceptions' 4: Click the 'Add exception' button 5: In the 'Type a file path, folder path, or URL' section, type luxsci.com 6: Click the 'Add Exception' button You can then open a new browser tab, and you should be able to access LuxSci.com from the new tab. We found via testing that attempting to open LuxSci.com in a browser tab that gave you the Avast error will not work, so a new tab will need to be opened.

Users of Avast Anti-virus should no longer see a phishing warning when going to LuxSci.com domains. If users still see warnings about LuxSci.com domains, they may need to update their Avast Virus Definitions.

We are experiencing an ongoing issue with a subset of our webmail servers. This issue is causing connectivity issues for some users on the affected webmail server. We will update this status once we have more information.

Our operations team has diagnosed the issue with shared webmail. A solution was put in place and the shared webmail servers that were affected should be fully operational now.

LuxSci Is currently seeing some issues with webmail that are causing users to have problems logging In and staying connected to the site. Our Operations team is investigating the issue and we expect the problem will be resolved soon.

The webmail issue has been resolved. All users should be able to access webmail at this time.

Customers sending email though server rs6161 (i.e., email1.luxsci.com, email4.luxsci.com, and smtp.luxsci.com) were intermittently experiencing slower connections, timeouts, and connection rejections. Investigation showed that this was due to 100s of concurrent connections from many different and differing IP addresses to SMTP SSL port 465. A majority of these were not even trying to "speak SMTP" ... but were just connecting and being rejected after a few seconds. However, the net effect was to "clog up" SMTP sending on this server. We first tried to simply increase the SMTP processing capacity on the server and wait a bit to see if the influx either (a) abated as these sometimes do after a short time, or (b) the sustained impact was small enough it could be absorbed by simple limit increases. This turned out not be be the case. As of now we have mitigated this situation by blocking inbound SMTP connections on port 465. This is blocking all of these invalid connections. However, as 465 is a valid SSL SMTP port, it may also block some customers. For those customers who are now blocked, we recommend changing SMTP port to: 1. The standard SMTP TLS port 25 or 587, or 2. Port 6465, which is our alternate SMTP over SSL port. For those customers who are affected but can not change their port and who are on a fixed IP address, please contact LuxSci support and we can white list your IP address so that you can connect to port 465 on this server. LuxSci will periodically check and remove this block on port 465 if/when the connection influx abates.

We are currently investigating this issue.

We are continuing to investigate this issue. It appears to be a routing issue and the data center is investigating.

The issue has been identified as a networking issue at the data center. They are working on a resolution, we do not have an ETA yet.

The team at the data center are continuing work on the issue. We currently have between one and two hours ETA until the issue is resolved.

Unfortunately it is taking longer than expected to resolve the networking issue at the data center. We do not have a new ETA for this to be fixed, however we will update when we have more information.

A fix has been implemented, showing positive results, and we will continue to monitor the status.

This incident has been resolved.

We are currently investigating an issue that has taken a number of virtual servers in Rackspace, Dallas Texas offline.

These servers are currently being brought back online.

All affected servers are now back online. The root cause of the issue was an administrative error at Rackspace.

Issue: This morning approximately between 9am EDT to 10:27am EDT there was an issue with inbound emails that go through ProofPoint bouncing with the error "User Unknown". Cause: The issue was caused by a misconfiguration between LuxSci's email servers and ProofPoint's email servers. Status: The issue was resolved at 10:27am EDT.