51 recorded Send With Us incidents since August 2018, with official updates, affected components, duration and resolution information.
Some customers may be experiencing elevated error rates on the Sendwithus API. We are investigating.
The issue was identified and a fix has been implemented. We are monitoring the results.
This incident has been resolved.
Some customers are experiencing intermittent connectivity issues with Sendwithus. The team is investigating.
The issue has been identified and a fix is being implemented.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
Our upstream image upload management provider is currently experiencing an outage and our engineering team is looking into it.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
Some customers are experiencing higher than normal error rates in the dashboard and API. The Sendwithus team is investigating.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
A subset of emails sent between 2 am and 11 am PDT July 14th had incorrect inline attachments added to them. Our team has identified and resolved the issue. Sendwithus support will be reaching out to impacted customers.
# Summary Between 18:33 July 13th and 11:20 July 14th 2023 \(PDT\), inline attachments from a small number of emails were incorrectly attached to an unknown number of emails sent by other accounts. # Impact & Metrics | **Question** | **Answer** | | --- | --- | | What was the impact | Incorrect inline attachments were incorrectly attached to a maximum of 17% of overall emails sent by Sendwithus | | Who was impacted | A subset of Sendwithus users sending emails between July 13th 6:33 pm PDT and July 14th 11:20 am PDT may have been impacted | # Detection & Response | **Question** | **Answer** | | --- | --- | | When was the incident detected | July 14th 2023, 10:21 am PDT | | How was the incident detected | An impacted user reached out to the Sendwithus Customer Support team and the incident was quickly escalated | # Timeline _Times are in Pacific Time_ **Jul 13, 2023** **09:44** A new feature related to sending inline attachments was released **18:33** The first detection of a send using the feature since the change **Jul 14, 2023** **10:21** A user reported some of their recipients were receiving unexpected inline attachments on their emails **10:45** Customer Support notifies on-call engineers and the incident is escalated **11:00** Status page updated **11:10** Resolved the issue by reverting to the previous version of Sendwithus **12:40** Underlying issue identified **13:00** Engineering team began an investigation to determine total user impact **17:00** First emails to personally reach out to impacted users sent # Root Cause Analysis **Problem:** A user has reported that some of their emails contain an unexpected attachment. Why? **Reason:** Recent changes were made to our systems to support sending an arbitrary number of inline attachments. This change introduced a bug which resulted in a variable being reused for subsequent emails sent by the same process. Why? **Reason:** An optional parameter in a function definition had a default reference-type value. The issue was not identified before deploying to production. Why? **Reason:** Our current testing, linting, and code review processes were not sufficient to identify the problem. # What can be improved * Changes are being made to our testing and linting processes to catch this category of bug in our CI pipeline * The transient nature of inline attachments and a lack of thread/process level identification in our archived transaction logs has made it difficult to identify exactly which customers and emails were affected. Due to this only the maximum number of _potential_ impacted emails can be identified * Increased controls to improve cross-tenant isolation of information # Corrective actions * The problematic change was quickly identified and reverted after we became aware of the issue * Changes have since been made to the codebase to reduce the probability of similar issues manifesting elsewhere
Our engineering team is currently investigating an issue with the Email logs page. Email sending may be delayed.
The issue has been identified and a fix is being implemented.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
Some customers are running into issues connecting to the Sendwithus API and dashboard. Our engineers are currently looking into the issue.
The issue has been identified and a fix is being implemented.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
Some customers may be experiencing elevated error rates on the Sendwithus API. We are investigating.
The issue has been identified and a fix is being implemented.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We are investigating the issue where users are seeing incorrect data in the customer's panel
We have temporarily disabled the feature while we investigate.
The issue has been identified and a fix is being implemented.
A fix has been implemented and we are monitoring the results.
The root problem has been resolved. Sendwithus support will be reaching out to impacted customers.
We're experiencing issues with forwarding Mailgun webhooks to accounts. The team has identified the problem and are working on a resolution.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
Some customers emails are failing to send correctly due to a rendering change. We've identified the problem and are working on a fix. We will resend any failures once the solution is live.
A fix has been implemented and we are monitoring the results.
The underlying problem has been resolved and all failed emails have been resent.
Some customers may be experiencing slower dashboard loading times. The Sendwithus team is investigating. Email is continuing to send normally.
We are continuing to investigate this issue.
The issue has been identified and a fix is being implemented.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
Some customers may be experiencing a slower than normal dashboard. The Sendwithus team is investigating. Email is continuing to send normally.
The issue has been identified and a fix is being implemented.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
Some customers have reported connection issues with the Sendwithus dashboard. Our engineering team is currently investigating.
This incident has been resolved.
Some customers may be seeing Sendwithus API and application errors. We are investigating the issue and will have an update soon.
The issue has been identified and we are implementing a fix.
A fix has been implemented and we are monitoring the results.
Sendwithus performance has returned to normal. We are continuing to monitor the results.
This incident has been resolved.
We've identified an issue with email sending and some messages may be delayed. A fix has been implemented.
A fix has been implemented and we are monitoring the results. Emails are sending normally again.
This incident has been resolved.
Our team has identified an issue where emails may be delayed for customers using SendGrid. We have released some changes to help mitigate this issue and we are actively working with the SendGrid team to get this resolved.
We have seen a significant decrease in delayed emails from SendGrid accounts and our team will continue to monitor.
We have not seen any errors in the last several hours. Our team is continuing to monitor.
This incident has been resolved.
We have identified an issue that is causing intermittent errors on the Sendwithus API and are working to resolve the issue. HTTP 5xx errors are safe to retry.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We are currently investigating this issue.
The issue has been identified and a fix is being implemented.
A fix has been implemented, dashboard access has been restored, and we are monitoring the results.
The dashboard has been running smoothly for the last several hours.
We have identified a problem with the reliability of the Sendwithus API and were working on a resolution. HTTP 5xx errors are safe to retry.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
