Kickbox 장애 기록

1. 2026년 8월 27일 오전 3:49 UTCidentified

   문제는 확인되었습니다. 우리는 수정 작업을하고 있습니다

2. 2026년 8월 27일 오전 3:50 UTCidentified

   이 문제를 해결하기 위해 계속 노력하고 있습니다.

3. 2026년 8월 27일 오전 5:04 UTCresolved

   이 사건은 해결되었습니다.

공식 인시던트 업데이트를 자동 번역했습니다.

1. 2026년 1월 19일 오후 3:17 UTCresolved

   This incident has been resolved.

1. 2026년 1월 16일 오전 2:13 UTCinvestigating

   Currently investigating issue with web application and batch API.

2. 2026년 1월 16일 오전 2:21 UTCidentified

   The issue has been identified and a fix is being implemented.

3. 2026년 1월 16일 오전 3:01 UTCidentified

   We are continuing to work on a fix for this issue.

4. 2026년 1월 16일 오전 3:24 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

5. 2026년 1월 16일 오전 4:23 UTCresolved

   This incident has been resolved.

1. 2025년 8월 21일 오후 7:30 UTCinvestigating

   Currently investigating issue.

2. 2025년 8월 21일 오후 8:26 UTCinvestigating

   We are still investigating an intermittent issue with list downloads.

3. 2025년 8월 21일 오후 10:02 UTCresolved

   Incident resolved. Monitoring will take place for a time to ensure functionality.

1. 2025년 6월 2일 오후 2:42 UTCidentified

   We are actively addressing an issue causing inconsistencies in the verification results for Yahoo and AOL emails. This problem has been traced to a breaking change in a recent update from Yahoo/AOL that affects our results. We have identified the issue and we are working towards a resolution.

2. 2025년 6월 2일 오후 9:03 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

3. 2025년 6월 3일 오후 1:17 UTCresolved

   This incident has been resolved.

1. 2025년 2월 21일 오후 12:24 UTCinvestigating

   Currently investigating an issue that is causing an outage on our EU instances.

2. 2025년 2월 21일 오후 1:14 UTCinvestigating

   We are continuing to investigate this issue.

3. 2025년 2월 21일 오후 1:14 UTCresolved

   This incident has been resolved.

1. 2024년 11월 17일 오전 10:54 UTCinvestigating

   Issue currently under investigation.

2. 2024년 11월 17일 오후 6:07 UTCmonitoring

   Fix implemented and results being monitored.

3. 2024년 11월 18일 오후 5:41 UTCresolved

   Incident resolved.

1. 2024년 10월 24일 오후 3:25 UTCinvestigating

   We are currently investigating and issue causing slower than normal API responses for a fraction of requests

2. 2024년 10월 24일 오후 3:26 UTCidentified

   Issue identified. Preparing deployment for fix.

3. 2024년 10월 24일 오후 3:51 UTCmonitoring

   A fix has been deployed. We are now monitoring services.

4. 2024년 10월 29일 오후 5:18 UTCresolved

   This incident has been resolved.

1. 2024년 9월 14일 오후 11:37 UTCinvestigating

   Partial outage. Investigating

2. 2024년 9월 14일 오후 11:38 UTCinvestigating

   We are continuing to investigate this issue.

3. 2024년 9월 15일 오전 12:07 UTCresolved

   This incident has been resolved.

1. 2024년 8월 21일 오후 8:40 UTCresolved

   Team is investigating the cause of a minor service degradation occurring between 2:30pm and 3:00pm (central time). Details to come.

2. 2024년 8월 22일 오후 5:15 UTCpostmortem

   # August 21, 2024 Service Degradation Investigation **Date**: 08/21/2024 **Status**: Mitigated ### Summary A service degradation was detected by our monitoring services between 12:34 PST and 12:48 PST. We removed the impacted servers from rotation restoring full-service. We investigated and solved the issue before bringing the impacted servers back into production rotation. ### Impact API service degradation in 45-second bursts for a 12 minute period, beginning at 12:34 PST causing some requests to respond with a 502 ‘Bad Gateway’. The issue was limited to a small subset of our production servers, so fewer than 3% of global API requests during this time were affected. ### Root Cause\(s\) During a routine canary upstream service update, a small number of canary servers became unable to connect to an upstream internal service. Due to this problem, a fraction of requests to these servers caused a connection timeout, resulting in 502 Bad Gateway errors from our proxies.This problem was sporadic, so the proxies did not automatically remove these servers from rotation. ### Resolution Our internal monitoring systems detected this degradation and we worked to manually remove these servers from production rotation. This restored the API to full health. After our investigation, we resolved the communication issues with a full code re-deploy and service restart. We then ran a full suite of API health checks, confirmed the problem was resolved, and re-added the servers into production. ‌ ### Action Items We will perform a review of our internal monitoring systems to decrease the time to alert, and review our proxy health-check endpoints to see if they can be improved for partial service degradation. ‌ ## Timeline

1. 2024년 3월 11일 오후 8:23 UTCidentified

   We’re currently experiencing reduced performance with several of our integrations, including Iterable, HubSpot, and Klaviyo within Kickbox, specifically affecting functionality; our team is actively working to resolve these issues to restore full service. Users can still actively leverage our email verification services through list verification or API. If you have any additional questions, please reach out to [email protected].

2. 2024년 6월 12일 오후 12:56 UTCresolved

   This incident has been resolved.

1. 2024년 3월 5일 오후 6:47 UTCresolved

   A 7 minute unexpected partial outage occurred. Root cause still under investigation.

1. 2024년 3월 5일 오후 3:08 UTCinvestigating

   We are currently investigating an issue with the EU verification instance

2. 2024년 3월 5일 오후 3:10 UTCinvestigating

   We are continuing to investigate this issue.

3. 2024년 3월 5일 오후 3:29 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

4. 2024년 3월 5일 오후 4:05 UTCresolved

   A networking error caused a severed connection to the EU verification engine. Manual reconnection has restored services.

1. 2024년 1월 21일 오후 6:01 UTCresolved

   A DNS change with an Amazon service temporarily broke sign-in for some users Sunday morning (API unaffected.

1. 2024년 1월 8일 오후 9:38 UTCresolved

   There was a small outage 10 minute outage due to an unforeseen network issue during a routine deployment.

2. 2024년 1월 10일 오후 3:15 UTCpostmortem

   A minor configuration change \(adding an alias to a Redis instance\) caused an unexpected error and prevented the Kickbox app from restarting after deployment. The issue was quickly detected and resolved by the engineering team. The total outage was 11 minutes. The misconfiguration was identified quickly and a rollback tag was deployed to restore service within minutes.

1. 2023년 9월 22일 오후 1:04 UTCinvestigating

   We're investigating a period of degraded service beginning at 5:54AM CST and lasting until 7:30AM CST.

2. 2023년 9월 22일 오후 2:41 UTCresolved

   We had degraded service due to a partial datastore service degradation. We repaired and restarted the datastore. No data was lost.

1. 2023년 9월 20일 오후 8:38 UTCresolved

   An issue with a upstream service provider created a window of unplanned downtime for a small fraction of EU users (smaller than 1%). The partial service outage lasted from 3pm September 19th until about 3pm the next day.

1. 2023년 9월 5일 오후 5:43 UTCinvestigating

   In the deliverablity tools suite, DMARC data has stopped populating as of September 1st. Engineering is investigating the issue and we hope to have more information and a resolution to this issue soon.

2. 2023년 9월 5일 오후 9:02 UTCinvestigating

   Still investigating the case.

3. 2023년 9월 6일 오후 7:53 UTCresolved

   This incident has been resolved.

1. 2023년 8월 17일 오후 5:10 UTCidentified

   Our system failed to properly log and summarized inbound aggregate DMARC reports. Our engineers are aware of the issue and are actively working to address this issue as soon as possible.

2. 2023년 8월 17일 오후 7:50 UTCidentified

   We are continuing to work on a fix for this issue.

3. 2023년 8월 17일 오후 7:50 UTCresolved

   We’ve corrected the issue with DMARC that was preventing ongoing data collection. Ongoing, inbound DMARC reports should be properly received, parsed, logged and displayed in reporting. If you have any questions, please don’t hesitate to reach out to us at [email protected]

1. 2023년 8월 8일 오후 5:26 UTCresolved

   Just after 11:00 am (central) a routine TLS certificate update caused a misconfiguration. Some users experienced a partial outage.

2. 2023년 8월 8일 오후 6:16 UTCpostmortem

   The new SSL cert that was implemented did not support `TLS1.2` . This resulted in issues for only some users where that support was required.