Historia awarii Inky

I database issue is causing a partial outage on the US Inky dashboard. Email processing is unaffected but settings will not be able to be changed while we correct the issue.

The issue has been identified and a fix is being implemented.

A fix has been implemented and we are monitoring the results.

Normal operations have been restored to Inky dashboard services

We are aware of a problem where Google Workspace Inky partners' emails are not being sent or received, intermittently. The Kaseya R&D Team is currently investigating this issue.

We are continuing to investigate and will provide another update as soon as we have more information to provide.

The issue has been identified by our dev team and is due to an issue with Google Workspace. You can view their status page here: https://www.google.com/appsstatus/dashboard/incidents/224ozRqzW4sFBDK8hLnT Inky's service is dependent on Google Workspace's ability to send emails and will be dependent on Google Workspace restoring service. We will provide another update as we have more information or Google Workspace restores email functionality.

Google has reported that email functionality has been restored and this issue has been resolved. Internal monitoring is showing INKY traffic has returned to normal.

On April 8ths, we noticed an issue with INKY partners where emails were failing to be sent or delivered, with a timeout error. After some initial investigation, this turned out to only be affecting partners who were using INKY with Google Workspace. There were no failures within the INKY product itself and the timeouts were specific to Google, who was experiencing an issue at that time. ‌ The RCA for this Google outage can be found here: [https://www.google.com/appsstatus/dashboard/incidents/224ozRqzW4sFBDK8hLnT](https://www.google.com/appsstatus/dashboard/incidents/224ozRqzW4sFBDK8hLnT)

We have detected a delay in outbound message processing in the Inky EU regions.

Inky Operations have identified what we believe is the issue and have taken corrective actions outbound mail delays should begin to return to normal levels.

Inky Engineers have restored normal mail processing times in the EU region and are monitoring to ensure stability.

This incident has been resolved.

On March 31s, our dev team observed that an email filter host was having a disproportionate amount of traffic sent to it, that was slowing down the response time.  This in turn caused additional pressure on some of our backend resources that were slowing down the email processing time.  This traffic did not cause the service to crash and auto scaling did not occur due to the CPU processing of the items not breaking a threshold.

Microsoft has identified a portion of service infrastructure in North America that is not processing traffic as expected. They are working to restore the infrastructure to a healthy state to achieve recovery. The issue impacting multiple Microsoft 365 services, including Outlook, Microsoft Defender, Microsoft Exchange, and Microsoft Purview. Further information can be found in the admin center at status.cloud.microsoft or under MO1221364. Be aware that Microsoft's Status Page has been impacted by this as well.

Microsoft reports that they have resolved the issue

Around 12:50 AM EST, INKY's outbound servers in one region experienced errors that made it impossible for them to accept outbound mail. During this time any attempts to hit this set of servers would result in a deferral where the sending server would be asked to try again. The retry would hit servers that were not experiencing issues, and the email would be processed normally but with a delay of a few minutes - dependent on how often the sending server was retrying. At ~7:40 AM EST, more servers hit these same errors reducing the number of servers able to accept the outbound mail to a much smaller number resulting in the possibility of outbound delays at this time going up until the issue was resolved across all servers at 8:20 AM EST. During this time, we also uncovered an issue with the alerting tied to this particular error with the outbound servers that was remedied as well. We are still actively monitoring this issue to ensure everything is running as expected.

The initial fix we put in place did not remedy the issue and it is possible that some mail delays could still occur. Our engineers are working diligently to address the issue. We will continue to update as necessary.

Everything running smoothly as expected and has been have we monitored throughout the day. Fixes now in place to correct issues tied to alerting and to prevent similar failures from occurring again.

Inky services had a mail processing delay from 11:15 to 12:00 on 12/5. An internal update failed and resulted in a slow down in mail processing. The result being that messages sent from 11:15 to approximately 12:00 ET on 12/5 may have experienced delivery delays during that period of time.

Processing has returned to normal speeds and deferred messages are being processed normally

This incident has been resolved.

Due to the AWS outage yesterday, users may experience the INKY Dashboards running a bit slower and showing some dated information on the Summary page. AWS is working through the backlog of data and as that is completed, the Dashboard speed and accuracy should return.

The backlog seems to have caught up, and Dashboard operations are back to normal.

We are currently investigating the cause of deferrals we are seeing with some email from O365 causing a delay in the delivery of email.

Inky continues to investigate the issue with email delays

Inky is continuing to monitor for any further signs of Microsoft connection difficulties. We are seeking a response from Microsoft on the cause of the issues that appear to have been isolated to either a single Microsoft region or server.

We have not seen a reoccurrence of this issue it appears that the external issue to Inky has been resolved.

We have identified an issue sending email to Microsoft Groups/Distribution Lists with Empty From Headers. Typically, this would only be older groups in a tenant with this configuration and a patch is being put together to handle these correctly.

Update has been made to correctly handle and deliver the emails to groups with an empty From header.

If a recipient attempts to view an encrypted message via the "Sign in with email" option, they will receive an error and will not be able to view the message at this time. We are investigating the issue and hope to have a fix in place soon.

The fix is in place and authorization through the Sign in with email option should be working as expected again.

We have received reports that some customers have seen a delay in mails getting from Microsoft to Inky. Some customers have received a deferral notification from Microsoft when attempting to hand mail to Inky for processing. At this time Inky systems are operating normally and we are investigating any possible connectivity issues between Inky and Microsoft.

Inky engineers are monitoring to see if there is any further indication of deferrals being received from by customers from Microsoft. Inky engineers did not identify any issues with connectivity between Inky and Microsoft but have noted that traffic from Microsoft dropped between 23:30 UT and 00:15 UT. Normal traffic resumed from Microsoft after that time. Throughout that timeframe Inky was able to establish connections to and send traffic to Microsoft as normal. Inky engineers are continuing to monitor.

Inky Engineers monitored traffic from Microsoft overnight and no further disruptions or drops in traffic from Microsoft servers was detected.

Between 15:00 ET and 18:00 ET on March 24, 2025 block list entries made by teams may not have properly saved. Inky has corrected the databasing issue that initiated the incident and no further impact is expected. We apologize for the inconvenience.

We have recognized the accuracy of our Graymail detection has taken a downturn due to an issue with a recent update. Our engineers are working on a fix now to get it back to expected levels.

Fixed the issue with the update and we are seeing expected levels of detection for the Graymail category once again. We will continue to monitor and ensure it stays within expected norms.

Graymail detection has been running as expected for four hours now since the fix was put in place. Closing the issue as resolved.

On March 29, 2024, Andreas Freund posted a backdoor warning to the public oss-security@openwall mailing list. The backdoor targets the liblzma library, part of the xz compression package. Many open source (and likely commercial) software systems incorporate this library, critical among them the Secure Shell Daemon (sshd). Virtually every Linux-based host worldwide ships with sshd, as it is the standard mechanism to allow secure remote logins. Many non-Linux systems support sshd as well. Further analysis of the exploit and an ongoing effort to maintain a timeline of events surrounding it suggest that the attacker’s goal was to allow arbitrary remote code execution on affected hosts with superuser (root) level permissions. This is known as an RCE vulnerability and essentially gives the attacker full access to all resources on the affected system. Immediately upon learning of this vulnerability, the INKY operations and development teams began assessing potential impact to INKY and INKY customers. Fortunately, the findings show that INKY could not have been affected; here’s why: - INKY uses the Ubuntu Linux distribution for all hosted services. The backdoor patches were never released into the Ubuntu “stable channel” meaning INKY servers would never have incorporated the affected liblzma version. (The affected versions are version 5.6 and 5.6.1; our servers all remain 5.2.4.) - All access to INKY servers that process sensitive customer data (e.g., emails and attachments) is further gated by bastion hosts. Only the bastion host has network connectivity to production servers, and all access to the latter must happen via an extra hop through the former. - Logging in to any INKY bastion host requires the use of a provisioned hardware token. This means that any ssh attempt to the bastion receives a challenge requiring the user to press a button on a physical device (a Yubikey). This in turn prevents anyone without physical possession of one of the handful of authorized tokens from connecting to the bastion. To further reduce the likelihood of impact from an unknown exploit, INKY upgrades system components and open-source dependencies conservatively; we wait until new versions have been released for some time before upgrading to them. (This helps ensure overall system stability as well, in that bugs discovered soon after a new release won’t affect INKY systems either.) We will continue to remain vigilant against supply chain attacks like these and will proactively notify customers of any potential impact to INKY systems or customer data. Fortunately, in this case, we can assure our customers there was no impact from CVE-2024-3094 to INKY or customer data handled by INKY services.

On March 29, 2024, Andreas Freund [posted a backdoor warning](https://www.openwall.com/lists/oss-security/2024/03/29/4) to the public oss-security@openwall mailing list. The backdoor targets the liblzma library, part of the xz compression package. Many open source \(and likely commercial\) software systems incorporate this library, critical among them the Secure Shell Daemon \(sshd\). Virtually every Linux-based host worldwide ships with sshd, as it is the standard mechanism to allow secure remote logins. Many non-Linux systems support sshd as well.  Further analysis of the exploit and an ongoing effort to maintain a [timeline](https://research.swtch.com/xz-timeline) of events surrounding it suggest that the attacker’s goal was to allow arbitrary remote code execution on affected hosts with superuser \(root\) level permissions. This is known as an [RCE vulnerability](https://en.wikipedia.org/wiki/Arbitrary_code_execution) and essentially gives the attacker full access to all resources on the affected system.  Immediately upon learning of this vulnerability, the INKY operations and development teams began assessing potential impact to INKY and INKY customers. Fortunately, the findings show that INKY could not have been affected; here’s why:  * INKY uses the Ubuntu Linux distribution for all hosted services. The backdoor patches were never released into the Ubuntu “stable channel” meaning INKY servers would never have incorporated the affected liblzma version. \(The affected versions are version 5.6 and 5.6.1; our servers all remain 5.2.4.\) * All access to INKY servers that process sensitive customer data \(e.g., emails and attachments\) is further gated by [bastion hosts](https://en.wikipedia.org/wiki/Bastion_host). Only the bastion host has network connectivity to production servers, and all access to the latter must happen via an extra hop through the former. * Logging in to any INKY bastion host requires the use of a provisioned hardware token. This means that any ssh attempt to the bastion receives a challenge requiring the user to press a button on a physical device \(a Yubikey\). This in turn prevents anyone without physical possession of one of the handful of authorized tokens from connecting to the bastion.  To further reduce the likelihood of impact from an unknown exploit, INKY upgrades system components and open-source dependencies conservatively; we wait until new versions have been released for some time before upgrading to them. \(This helps ensure overall system stability as well, in that bugs discovered soon after a new release won’t affect INKY systems either.\)  We will continue to remain vigilant against supply chain attacks like these and will proactively notify customers of any potential impact to INKY systems or customer data.  Fortunately, in this case, we can assure our customers _there was no impact from **CVE-2024-3094** to INKY or customer data handled by INKY services._

Relaying information from Microsoft about an issue they are currently experiencing with Exchange Online EX736432 - O365 Admins should be able to see all updates and details in O365 Service Health Dashboard Title: Some users may be unable to send, or see delays up to two hours when sending email messages through Exchange Online User impact: Users may be unable to send, or see delays up to two hours when sending, email messages through Exchange Online. More info: These failures and delays could occur when sending email through any Exchange Online connection method. Users may receive a Non-Delivery Report (NDR) when a message fails to send.

EX736432 - O365 Admins should be able to see all updates and details in O365 Service Health Dashboard Title: Some users may be unable to send, or see delays up to two hours when sending email messages through Exchange Online User impact: Users may be unable to send, or see delays up to two hours when sending, email messages through Exchange Online. More info: These failures and delays could occur when sending email through any Exchange Online connection method. Users may receive a Non-Delivery Report (NDR) when a message fails to send. Current status: We're continuing to analyze some sample message IDs, as well as requesting extended message traces for those sample message IDs, to determine if additional mitigation actions are required to fully resolve the issue. Scope of impact: Some users who are serviced through the affected infrastructure and are sending email messages through any Exchange Online connection methods may be impacted. Start time: Sunday, March 10, 2024 at 2:00 PM CDT Root cause: A recent change intended to resolve another ongoing issue was enabled and introduced unbalanced network traffic to the affected infrastructure, leading to impact. Next update by: Friday, March 15, 2024 at 12:00 PM CDT

Microsoft has closed the issue saying it has been resolved. Mar 15, 2024, 12:01 PM CDT Title: Some users may be unable to send, or see delays up to two hours when sending email messages through Exchange Online User impact: Users may have been unable to send, or see delays up to two hours when sending, email messages through Exchange Online. Final Status: We’ve received confirmation from impacted users that email messages are delivering as expected. The aforementioned persisting impact as reported by a small number of users isn’t related to this event. Scope of impact: Some users who were serviced through the affected infrastructure and were sending email messages through any Exchange Online connection methods may have been impacted. Start time: Sunday, March 10, 2024 at 2:00 PM CDT End time: Thursday, March 14, 2024 at 1:41 PM CDT . Root cause: A recent change intended to resolve another ongoing issue was enabled and introduced unbalanced network traffic to the affected infrastructure, leading to impact. Next steps: - We're assessing how impact related to the change wasn't identified prior to deployment so that we can improve our change management processes. This is the final update for the event.

Currently investigating processing delays impacting Outbound Mail protection and Internal Mail Protection

We are continuing to investigate the issue. If you are impacted by the delay, you can disable the IPW Outbound Rule (the Priority 0 Rule) in your transport rules to bypass Inky for the outbound emails.

We have reached out to Microsoft to attempt to determine what is causing what appears to be an issue affecting mail sending from Office 365. Multiple services outside Inky appear to be affected by connection problems from Office 365 that are resulting in delays of message sending

Error rates have returned to normal. We continue to seek an update from Microsoft on what led to the increase in connection failures from the Exchange Online system.

We will continue to closely monitor the errors from Microsoft servers throughout the day while we seek an explanation from them for the connection issues they were experiencing.

Stability has continued from Microsoft since the errors stopped yesterday. Out of an abundance of caution, we are continuing to monitor throughout the day. We do feel that any team that had disabled the IPW Outbound transport rule should re-enable the rule to resume the Banner Stripping and Outbound Protection features of INKY.

Not seeing any reoccurrence of the errors we were seeing with Microsoft. Delivery has been steady throughout the day without issue. Closing out this incident.

We are currently investigating issues affecting some users being able to log into the Dashboard or Reporting Emails that require authentication.

Authentication and log in are fully functional again. We are monitoring everything to ensure it is working as expected.

This incident has been resolved.

Please be advised that Microsoft has issued an incident EX682041 indicating that they are working on an issue affecting their outbound spam filter policies. Tenants that have an admin contact defined in their outbound spam policy may be BCC'd on messages sent by their users that Microsoft is incorrectly flagging as spam. Microsoft is working on the issue and Inky will be continuing to monitor the event.

Microsoft has confirmed that they believe that they have corrected the impact from this incident.

We have identified the issue and implementing the fix now. Will update as soon as it is in place and traffic returns to normal.

Fix is in place for mail flow and we are currently expanding capacity to handle the backlog of email that built up during the outage.

Fix has been applied on the Dashboard. The backlog of emails have cleared with the additional capacity. We will continue to monitor.

All systems are back to normal and steady.

# Post incident report: Start: 16-September-2023 1400 UTC End: 16-September-2023 1630 UTC Duration: 2 hours 30 min ## Summary: Mail was delayed and the sections of the Inky Dashboard were unavailable for approximately two and a half hours on Saturday, September 16th due an expired server certificate. ## Root Cause: Due to human error, a server certificate expired before we were able to replace it with a renewed one. ## Mitigation Action: We renewed the server certificate and increased the capacity to process the backlog as quickly as possible to get traffic back to normal. ## Customer Impact: For approximately two and a half hours, users' email was delayed during the period between the server expiring and us installing the new server certificate. During this period, there were also sections of the Inky Dashboard that were unavailable. ## Follow-up Items and Preventative Measures: * We have implemented multiple alerting and monitoring processes to mitigate the danger of missing any future server certificate renewals.

Inky is investigating an issue affecting the dashboard and preventing data from loading. Administration functions and settings changes remain available but you may not currently see graphs load while we resolve the issue.

Inky has isolated the issue affecting dashboard performance and we are restoring normal operations

We have corrected the issue which effected data display on the dashboard.