Send With Us 장애 기록

1. 2026년 9월 16일 오후 1:38 UTCinvestigating

   일부 고객은 Sendwithus API에서 높은 오류율을 경험할 수 있습니다. 우리는 투자입니다.

2. 2026년 9월 16일 오후 2:15 UTCmonitoring

   문제가 식별되고 수정이 구현되었습니다. 우리는 결과를 감시하고 있습니다.

3. 2026년 9월 16일 오후 2:22 UTCresolved

   이 사건은 해결되었습니다.

공식 인시던트 업데이트를 자동 번역했습니다.

1. 2023년 9월 7일 오후 7:39 UTCinvestigating

   Some customers are experiencing intermittent connectivity issues with Sendwithus. The team is investigating.

2. 2023년 9월 7일 오후 7:43 UTCidentified

   The issue has been identified and a fix is being implemented.

3. 2023년 9월 7일 오후 7:47 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

4. 2023년 9월 7일 오후 7:59 UTCresolved

   This incident has been resolved.

1. 2023년 8월 30일 오전 2:09 UTCinvestigating

   Our upstream image upload management provider is currently experiencing an outage and our engineering team is looking into it.

2. 2023년 8월 30일 오전 2:17 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

3. 2023년 8월 30일 오전 2:31 UTCresolved

   This incident has been resolved.

1. 2023년 8월 2일 오후 5:29 UTCinvestigating

   Some customers are experiencing higher than normal error rates in the dashboard and API. The Sendwithus team is investigating.

2. 2023년 8월 2일 오후 5:35 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

3. 2023년 8월 2일 오후 5:55 UTCresolved

   This incident has been resolved.

1. 2023년 7월 14일 오후 11:07 UTCresolved

   A subset of emails sent between 2 am and 11 am PDT July 14th had incorrect inline attachments added to them. Our team has identified and resolved the issue. Sendwithus support will be reaching out to impacted customers.

2. 2023년 7월 19일 오후 6:20 UTCpostmortem

   # Summary Between 18:33 July 13th and 11:20 July 14th 2023 \(PDT\), inline attachments from a small number of emails were incorrectly attached to an unknown number of emails sent by other accounts. # Impact & Metrics | **Question** | **Answer** | | --- | --- | | What was the impact | Incorrect inline attachments were incorrectly attached to a maximum of 17% of overall emails sent by Sendwithus | | Who was impacted | A subset of Sendwithus users sending emails between July 13th 6:33 pm PDT and July 14th 11:20 am PDT may have been impacted | # Detection & Response‌ | **Question** | **Answer** | | --- | --- | | When was the incident detected | July 14th 2023, 10:21 am PDT | | How was the incident detected | An impacted user reached out to the Sendwithus Customer Support team and the incident was quickly escalated | # Timeline _Times are in Pacific Time_ **Jul 13, 2023** **09:44** A new feature related to sending inline attachments was released **18:33** The first detection of a send using the feature since the change **Jul 14, 2023** **10:21** A user reported some of their recipients were receiving unexpected inline attachments on their emails **10:45** Customer Support notifies on-call engineers and the incident is escalated **11:00**   Status page updated **11:10** Resolved the issue by reverting to the previous version of Sendwithus **12:40** Underlying issue identified **13:00** Engineering team began an investigation to determine total user impact **17:00** First emails to personally reach out to impacted users sent # Root Cause Analysis **Problem:** A user has reported that some of their emails contain an unexpected attachment. Why? **Reason:** Recent changes were made to our systems to support sending an arbitrary number of inline attachments. This change introduced a bug which resulted in a variable being reused for subsequent emails sent by the same process. Why? **Reason:** An optional parameter in a function definition had a default reference-type value. The issue was not identified before deploying to production. Why? **Reason:** Our current testing, linting, and code review processes were not sufficient to identify the problem. # What can be improved * Changes are being made to our testing and linting processes to catch this category of bug in our CI pipeline * The transient nature of inline attachments and a lack of thread/process level identification in our archived transaction logs has made it difficult to identify exactly which customers and emails were affected. Due to this only the maximum number of _potential_ impacted emails can be identified * Increased controls to improve cross-tenant isolation of information # Corrective actions * The problematic change was quickly identified and reverted after we became aware of the issue * Changes have since been made to the codebase to reduce the probability of similar issues manifesting elsewhere

1. 2023년 5월 23일 오후 2:45 UTCinvestigating

   Our engineering team is currently investigating an issue with the Email logs page. Email sending may be delayed.

2. 2023년 5월 23일 오후 2:56 UTCidentified

   The issue has been identified and a fix is being implemented.

3. 2023년 5월 23일 오후 3:02 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

4. 2023년 5월 23일 오후 5:33 UTCresolved

   This incident has been resolved.

1. 2023년 5월 4일 오후 9:45 UTCinvestigating

   Some customers are running into issues connecting to the Sendwithus API and dashboard. Our engineers are currently looking into the issue.

2. 2023년 5월 4일 오후 10:17 UTCidentified

   The issue has been identified and a fix is being implemented.

3. 2023년 5월 4일 오후 10:43 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

4. 2023년 5월 4일 오후 11:17 UTCresolved

   This incident has been resolved.

1. 2023년 4월 27일 오후 5:53 UTCinvestigating

   Some customers may be experiencing elevated error rates on the Sendwithus API. We are investigating.

2. 2023년 4월 27일 오후 6:20 UTCidentified

   The issue has been identified and a fix is being implemented.

3. 2023년 4월 27일 오후 6:22 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

4. 2023년 4월 27일 오후 6:40 UTCresolved

   This incident has been resolved.

1. 2023년 3월 15일 오후 10:52 UTCinvestigating

   We are investigating the issue where users are seeing incorrect data in the customer's panel

2. 2023년 3월 15일 오후 11:01 UTCinvestigating

   We have temporarily disabled the feature while we investigate.

3. 2023년 3월 16일 오전 12:18 UTCidentified

   The issue has been identified and a fix is being implemented.

4. 2023년 3월 16일 오전 2:49 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

5. 2023년 3월 16일 오전 4:36 UTCresolved

   The root problem has been resolved. Sendwithus support will be reaching out to impacted customers.

1. 2023년 3월 15일 오후 10:48 UTCidentified

   We're experiencing issues with forwarding Mailgun webhooks to accounts. The team has identified the problem and are working on a resolution.

2. 2023년 3월 16일 오전 3:44 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

3. 2023년 3월 16일 오전 4:31 UTCresolved

   This incident has been resolved.

1. 2022년 11월 7일 오후 8:33 UTCidentified

   Some customers emails are failing to send correctly due to a rendering change. We've identified the problem and are working on a fix. We will resend any failures once the solution is live.

2. 2022년 11월 7일 오후 8:43 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

3. 2022년 11월 7일 오후 9:40 UTCresolved

   The underlying problem has been resolved and all failed emails have been resent.

1. 2022년 9월 29일 오후 7:22 UTCinvestigating

   Some customers may be experiencing slower dashboard loading times. The Sendwithus team is investigating. Email is continuing to send normally.

2. 2022년 9월 29일 오후 7:23 UTCinvestigating

   We are continuing to investigate this issue.

3. 2022년 9월 29일 오후 8:26 UTCidentified

   The issue has been identified and a fix is being implemented.

4. 2022년 9월 29일 오후 8:57 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

5. 2022년 9월 30일 오전 6:05 UTCresolved

   This incident has been resolved.

1. 2022년 7월 28일 오후 4:44 UTCinvestigating

   Some customers may be experiencing a slower than normal dashboard. The Sendwithus team is investigating. Email is continuing to send normally.

2. 2022년 7월 28일 오후 5:15 UTCidentified

   The issue has been identified and a fix is being implemented.

3. 2022년 7월 28일 오후 5:51 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

4. 2022년 7월 28일 오후 6:54 UTCresolved

   This incident has been resolved.

1. 2022년 6월 23일 오후 4:21 UTCinvestigating

   Some customers have reported connection issues with the Sendwithus dashboard. Our engineering team is currently investigating.

2. 2022년 6월 23일 오후 5:50 UTCresolved

   This incident has been resolved.

1. 2022년 4월 21일 오후 6:48 UTCinvestigating

   Some customers may be seeing Sendwithus API and application errors. We are investigating the issue and will have an update soon.

2. 2022년 4월 21일 오후 6:54 UTCidentified

   The issue has been identified and we are implementing a fix.

3. 2022년 4월 21일 오후 6:57 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

4. 2022년 4월 21일 오후 7:05 UTCmonitoring

   Sendwithus performance has returned to normal. We are continuing to monitor the results.

5. 2022년 4월 21일 오후 7:27 UTCresolved

   This incident has been resolved.

1. 2022년 4월 11일 오후 10:00 UTCidentified

   We've identified an issue with email sending and some messages may be delayed. A fix has been implemented.

2. 2022년 4월 11일 오후 10:05 UTCmonitoring

   A fix has been implemented and we are monitoring the results. Emails are sending normally again.

3. 2022년 4월 11일 오후 10:25 UTCresolved

   This incident has been resolved.

1. 2022년 3월 23일 오후 6:35 UTCidentified

   Our team has identified an issue where emails may be delayed for customers using SendGrid. We have released some changes to help mitigate this issue and we are actively working with the SendGrid team to get this resolved.

2. 2022년 3월 23일 오후 7:34 UTCmonitoring

   We have seen a significant decrease in delayed emails from SendGrid accounts and our team will continue to monitor.

3. 2022년 3월 23일 오후 9:28 UTCmonitoring

   We have not seen any errors in the last several hours. Our team is continuing to monitor.

4. 2022년 3월 23일 오후 11:37 UTCresolved

   This incident has been resolved.

1. 2022년 3월 23일 오후 2:29 UTCidentified

   We have identified an issue that is causing intermittent errors on the Sendwithus API and are working to resolve the issue. HTTP 5xx errors are safe to retry.

2. 2022년 3월 23일 오후 3:06 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

3. 2022년 3월 23일 오후 3:35 UTCresolved

   This incident has been resolved.

1. 2022년 3월 21일 오후 5:51 UTCinvestigating

   We are currently investigating this issue.

2. 2022년 3월 21일 오후 6:02 UTCidentified

   The issue has been identified and a fix is being implemented.

3. 2022년 3월 21일 오후 6:04 UTCmonitoring

   A fix has been implemented, dashboard access has been restored, and we are monitoring the results.

4. 2022년 3월 21일 오후 9:38 UTCresolved

   The dashboard has been running smoothly for the last several hours.

1. 2022년 3월 10일 오후 10:11 UTCidentified

   We have identified a problem with the reliability of the Sendwithus API and were working on a resolution. HTTP 5xx errors are safe to retry.

2. 2022년 3월 10일 오후 11:09 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

3. 2022년 3월 11일 오전 1:16 UTCresolved

   This incident has been resolved.