Zonos 장애 기록

1. 2026년 9월 5일 오전 12:08 UTCinvestigating

   우리는 현재이 문제를 조사하고 있습니다.

2. 2026년 9월 5일 오전 12:09 UTCidentified

   문제는 식별되고 수정이 구현됩니다.

3. 2026년 9월 5일 오전 12:21 UTCmonitoring

   수정이 구현되어 결과를 모니터링하고 있습니다.

4. 2026년 9월 5일 오전 12:56 UTCresolved

   이 사건은 해결되었습니다.

공식 인시던트 업데이트를 자동 번역했습니다.

1. 2026년 9월 4일 오후 5:29 UTCinvestigating

   우리는 현재이 문제를 조사하고 있습니다.

2. 2026년 9월 4일 오후 5:30 UTCidentified

   문제는 식별되고 수정이 구현됩니다.

3. 2026년 9월 4일 오후 5:30 UTCmonitoring

   수정이 구현되어 결과를 모니터링하고 있습니다.

4. 2026년 9월 4일 오후 7:47 UTCresolved

   이 사건은 해결되었습니다.

공식 인시던트 업데이트를 자동 번역했습니다.

1. 2026년 8월 3일 오후 9:23 UTCinvestigating

   우리는 현재이 문제를 조사하고 있습니다.

2. 2026년 8월 3일 오후 9:24 UTCidentified

   문제는 식별되고 수정이 구현됩니다.

3. 2026년 8월 3일 오후 9:57 UTCidentified

   이 문제를 해결하기 위해 계속 노력하고 있습니다.

4. 2026년 8월 3일 오후 9:57 UTCmonitoring

   수정이 구현되어 결과를 모니터링하고 있습니다.

5. 2026년 8월 3일 오후 10:36 UTCmonitoring

   우리는 더 많은 문제를 모니터링하고 있습니다.

6. 2026년 8월 3일 오후 10:36 UTCresolved

   이 사건은 해결되었습니다.

7. 2026년 9월 9일 오후 8:06 UTCpostmortem

   ##**공지 사항** 8 월 3, 2026, Zonos는 착륙 비용 및 관련 서비스에 영향을 미치는 높은 오류율을 경험했습니다. 사건은 대략 2:52 PM MDT에서 시작되고 3:51 PM MDT에, degraded 서비스의 대략 59 분 결과로 해결되었습니다. 사건은 부분적 정전이었다: 요청은 이벤트 전반에 걸쳐 처리되었지만, 영향을받는 서비스로 재해하는 중요한 요청은 오류를 반환합니다. ## ** 중요 ** 1 차적인 영향을받는 서비스는 사건 창 도중 GraphQL 요구에 대략 50% 과실 비율을 경험했습니다. 그 결과, Landed Cost를 사용한 고객 경험있는 간헐적 인 요청 실패, 다른 요청이 성공적으로 처리하는 동안. Landed Cost는 공유 플랫폼 서비스에 의존하기 때문에 관련 작업 흐름을 통해 높은 오류율이 발생합니다. ## ** 루트 원인 ** 이 사건은 우리 플랫폼 내에서 새로 도입 된 데이터 액세스 경로를 사용하여 시작된 생산 배포에 의해 트리거되었습니다. 생산 요청 볼륨 아래, 이 액세스 패턴은 예상보다 더 많은 데이터베이스 활동을 생성. 이 지원 서비스 내에서 사용 가능한 데이터베이스 연결 및 그 서비스에 따라 발생하는 요청이 실패합니다. 영향을받는 기능이 아직 생산 규모에서 운동되지 않았기 때문에 우선적으로 발생하지 않았습니다. ## **해설 ** 우리의 엔지니어링 팀은 처음에 관련된 서비스 배포를 압연하면서 오류의 소스를 조사했습니다. 이 문제를 해결하지 못했을 때, 문제 접근 패턴을 활성화 한 Landed Cost 배포가 다시 구출되었습니다. 서비스 즉시 그 rollback에 따라 복구. 영구 보정은 데이터베이스 액세스 패턴을 최적화하고 영향을받는 작업 부하를 적절하게 경로를 배포했습니다. ##**이전 액션** 사건에 따라, 우리는 즉각적인 실패를 모두 검토하고 종속 서비스를 통해 전파하는 방법. 우리는 다음과 같은 유사 사건의 likelihood 및 잠재적 영향을 줄이기 위해 몇 가지 조치를 취하고 있습니다. * 데이터베이스 연결 포화에 대한 추가 모니터링 및 경고 추가. * 분할된 데이터에 대한 데이터베이스 쿼리에 대한 안전한 보호 및 검토 관행 강화. * 새로 활성화 된 데이터 액세스 패턴의 확장. * 평가 서비스 고립과 실패 처리는 다른 플랫폼 서비스에 충격을 감소시키기 위하여 의존성 할 수 있습니다. * 이 사건 중 확인된 조건에 대한 플랫폼에 대한 유사한 데이터 접근 패턴을 검토. 우리는이 사건이 발생하는 붕괴에 대해 사과합니다. Zonos 플랫폼의 탄력과 관찰력을 향상시키기 위해이 이벤트에서 발견을 계속합니다.

공식 인시던트 업데이트를 자동 번역했습니다.

1. 2026년 7월 8일 오후 7:41 UTCinvestigating

   We are currently investigating this issue.

2. 2026년 7월 8일 오후 7:42 UTCidentified

   The issue has been identified and a fix is being implemented.

3. 2026년 7월 8일 오후 7:42 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

4. 2026년 7월 8일 오후 9:00 UTCresolved

   This incident has been resolved.

1. 2026년 5월 26일 오후 9:27 UTCinvestigating

   An issue with an authentication provider is currently affecting some users.

2. 2026년 5월 26일 오후 9:29 UTCidentified

   The issue has been identified and a fix is being implemented.

3. 2026년 5월 26일 오후 9:41 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

4. 2026년 5월 26일 오후 9:54 UTCmonitoring

   We are continuing to see increased error rates and are monitoring for a fix from our authentication provider.

5. 2026년 5월 26일 오후 9:59 UTCmonitoring

   We are seen error rates decline and will continue monitoring.

6. 2026년 5월 26일 오후 10:31 UTCresolved

   This incident has been resolved.

1. 2026년 3월 11일 오후 9:19 UTCinvestigating

   We are currently investigating this issue.

2. 2026년 3월 11일 오후 9:19 UTCidentified

   The issue has been identified and a fix is being implemented.

3. 2026년 3월 11일 오후 9:20 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

4. 2026년 3월 11일 오후 9:21 UTCresolved

   This incident has been resolved.

5. 2026년 3월 11일 오후 9:25 UTCpostmortem

   **What products were affected and what was the impact?** Classify, Country of Origin Impact: DEGRADED PERFORMANCE   **What timeframe did this issue occur?** | **Date** | **Time** | | --- | --- | | Mar 11, 2026 | 2026-03-11 10:12 - 2026-03-11 12:06 |   **How was the issue detected?** Zonos observed elevated error rates affecting Classify and Country of Origin APIs. ‌ **What functionality was affected?** A portion of Classify and Country of Origin inference requests did not return a valid response. ‌ **What was the root cause?** A firmware issue on a subset of GPU nodes caused network connectivity failures. This caused the nodes to enter an unhealthy state. Our reconnect logic did not properly remove unhealthy nodes from the pool, so they continued to receive requests. ‌ **What was the resolution of the problem, and what steps are being taken to prevent future issues?** Shortly after discovering the issue, we restarted nodes to resolve the connectivity issues and restore service. We then identified the reconnect logic issue, and released a fix to prevent future issues. We have scheduled a firmware upgrade on all nodes to resolve the underlying network issue. We have also refined our alerting to catch errors more quickly in the future.

1. 2026년 2월 22일 오후 6:25 UTCinvestigating

   We are currently investigating this issue.

2. 2026년 2월 22일 오후 6:28 UTCinvestigating

   We are continuing to investigate this issue.

3. 2026년 2월 22일 오후 7:16 UTCidentified

   The issue has been identified and a fix is being implemented.

4. 2026년 2월 23일 오후 3:45 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

5. 2026년 2월 23일 오후 4:22 UTCresolved

   This incident has been resolved.

1. 2026년 2월 19일 오후 9:58 UTCinvestigating

   We are currently investigating this issue.

2. 2026년 2월 19일 오후 9:59 UTCidentified

   The issue has been identified and a fix is being implemented.

3. 2026년 2월 19일 오후 9:59 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

4. 2026년 2월 19일 오후 10:24 UTCresolved

   This incident has been resolved.

1. 2026년 2월 19일 오후 4:29 UTCinvestigating

   We are currently investigating this issue.

2. 2026년 2월 19일 오후 4:32 UTCidentified

   The issue has been identified and a fix is being implemented.

3. 2026년 2월 19일 오후 4:58 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

4. 2026년 2월 19일 오후 5:00 UTCmonitoring

   We are continuing to see some issues and are continuing to monitor.

5. 2026년 2월 19일 오후 5:26 UTCmonitoring

   We are continuing to monitor for any further issues.

6. 2026년 2월 19일 오후 5:48 UTCmonitoring

   The issue has been resolved and operations have returned to normal. We are actively monitoring to ensure continued stability.

7. 2026년 2월 19일 오후 6:57 UTCresolved

   This incident has been resolved.

1. 2025년 10월 20일 오후 9:57 UTCinvestigating

   We are currently investigating this issue.

2. 2025년 10월 20일 오후 9:57 UTCidentified

   The issue has been identified and a fix is being implemented.

3. 2025년 10월 20일 오후 9:57 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

4. 2025년 10월 20일 오후 9:57 UTCresolved

   This incident has been resolved.

1. 2025년 9월 4일 오후 7:59 UTCinvestigating

   We are currently investigating this issue.

2. 2025년 9월 4일 오후 8:17 UTCresolved

   Users logging in may have experienced a flashing screen. The change was rolled back and the issue has been resolved.

1. 2025년 6월 10일 오후 2:54 UTCinvestigating

   An issue with an upstream infrastructure provider is impacting our BigCommerce Plugin

2. 2025년 6월 10일 오후 2:54 UTCidentified

   The issue has been identified and we are continuing to monitor the impact.

3. 2025년 6월 10일 오후 2:55 UTCidentified

   We are continuing to work on a fix for this issue.

4. 2025년 6월 10일 오후 4:00 UTCresolved

   This incident has been resolved.

1. 2025년 2월 13일 오후 9:52 UTCinvestigating

   We are currently investigating this issue.

2. 2025년 2월 13일 오후 9:53 UTCresolved

   This incident has been resolved.

3. 2025년 2월 13일 오후 9:56 UTCpostmortem

   **What products were affected and what was the impact?** * Checkout \(Custom Integrations\) Impact: * Major outage for affected merchants **What timeframe did this issue occur?** |   | **Date** | **Time** | | --- | --- | --- | | From: | Jan 24, 2025 | 12:00 PM MST | | To: | Jan 27, 2025 | 07:45 AM MST |   **How was the issue detected?** Our team was notified that Checkout domains were not being allowed as expected, leading to an immediate investigation by the engineering team. **What functionality was affected?** Online store settings \(allowed store domain\) were not loading properly for all stores using custom integrations. This prevented affected stores from loading checkout. ‌ **What problems did this cause?** Stores relying on custom integrations could not load their checkout correctly. ‌ **What was the resolution of the problem and steps that are being taken for continued follow-up?** * A rollback to a previous version immediately resolved the issue. * A new synthetic test has been implemented to ensure store settings caching is store-specific and does not mix settings between stores. ‌ ‌ **What mitigation solutions will we put in place to prevent this issue from occurring in the future?** * **Improved Testing:** A new synthetic test has been introduced to verify caching behavior and prevent incorrect store settings from being cached. * **Staging Environment:** Setting up a dedicated test environment for custom integrations to catch similar issues earlier.

1. 2025년 2월 13일 오후 8:52 UTCinvestigating

   We are currently investigating this issue.

2. 2025년 2월 13일 오후 8:53 UTCresolved

   This incident has been resolved.

3. 2025년 2월 13일 오후 9:27 UTCpostmortem

   ### What products were affected and what was the impact? Shopify Checkout Partial outage: * Affected some Shopify merchants ### What timeframe did this issue occur? | | **Date** | **Time** | | --- | --- | --- | | From: | Jan 23, 2025 | 05:30 PM MDT | | To: | Jan 24, 2025 | 03:30 PM MDT | ### How was the issue detected? * A discrepancy was identified when some orders in Shopify were missing tracking numbers. * Additionally, merchants using older versions of the Shopify app experienced order import failures. ‌ ### What functionality was affected? * Order fulfillment ‌ ### What problems did this cause? * **Order Processing Issues:** Orders processed without tracking numbers caused confusion and delays for merchants and customers. * **Order Import Failures:** Some merchants experienced missing orders due to import failures. ‌ ### What was the resolution of the problem and steps that are being taken for continued follow-up? * We patched the root cause of the issue, resolving the tracking number and order import failures. * We updated all the affected orders with tracking numbers and manually imported orders that had failed to import. * API headers and logging enhancements were implemented to improve issue detection and response time. ### What mitigation solutions will we put in place to prevent this issue from occurring in the future? * Expand QA coverage to address downstream impacts of refactors, with a focus on comprehensive testing of critical flows, including label creation. * Expand synthetic test suite to validate end-to-end order processing, with particular emphasis on fulfillment workflows. * Establish a consistent testing environment that simulates real-world Shopify store configurations, reducing the risk of silent failures.

1. 2025년 2월 13일 오후 8:30 UTCinvestigating

   We are currently investigating this issue.

2. 2025년 2월 13일 오후 8:31 UTCresolved

   This incident has been resolved.

3. 2025년 2월 13일 오후 8:38 UTCpostmortem

   **What products were affected and what was the impact?** * All Magento Checkout merchants Impact: * Partial Outage **What timeframe did this issue occur?** | | **Date** | **Time** | | --- | --- | --- | | From: | Jan 6, 2025 | 2:27 PM MST | | To: | Jan 7, 2025 | 09:11 AM MST | ‌ ### How was the issue detected? Our team was notified that carts were not being created for some Magento shoppers, leading to an immediate investigation by the engineering team. ‌ ### What functionality was affected? The checkout process for Magento merchants encountered validation failures due to incorrect UUID handling. This disrupted the checkout flow, impacting merchants using the Magento Checkout. ‌ ### What problems did this cause? * Magento merchants experienced disruptions in checkouts being loaded at times. ‌ ### What was the resolution of the problem and steps that are being taken for continued follow-up? * The issue was identified as stemming from changes made to TSID validation. * A rollback of the problematic changes was performed promptly, restoring checkout functionality. * A synthetic test was created to validate UUID behavior and detect future occurrences of this issue. * Code annotations have been added to document known issues and improve future debugging. ‌ ### What mitigation solutions will we put in place to prevent this issue from occurring in the future? * Establishing alerts detecting deviations in carts-to-checkout sessions. This will help flag potential checkout issues early. * Increasing test coverage to identify gaps in synthetic testing.

1. 2025년 2월 13일 오후 6:51 UTCinvestigating

   We are experiencing an issue with orders not being created despite payments being processed successfully.

2. 2025년 2월 13일 오후 6:53 UTCresolved

   This incident has been resolved.

3. 2025년 2월 13일 오후 6:54 UTCpostmortem

   **What products were affected and what was the impact?** * Zonos Checkout Impact: * Major Outage **What timeframe did this issue occur?** | | **Date** | **Time** | | --- | --- | --- | | From: | Jan 3, 2025 | 01:30 PM MST | | To: | Jan 3, 2025 | 07:06 PM MST | ‌ ### How was the issue detected? Our team was notified that payments were being processed, but orders were not being created. ‌ ### What functionality was affected? * Ability to create orders * Order fulfillment ‌ ### What problems did this cause? * Customer frustration due to failed order creation despite successful payment processing. * Potential financial reconciliation issues for merchants and customers. ‌ ### What was the resolution of the problem and steps that are being taken for continued follow-up? * The root cause was identified as a backend change, which introduced a bug related to the separation of payment intent and checkout session. * A rollback was implemented to restore functionality, and error logging for Stripe intent failures has now been enabled. ‌ ### What mitigation solutions will we put in place to prevent this issue from occurring in the future? * Synthetic tests have been updated to run every 10 minutes to narrow down problematic PRs and detect failures earlier.

1. 2024년 11월 12일 오후 8:51 UTCinvestigating

   We are currently investigating elevated error rates.

2. 2024년 11월 12일 오후 9:00 UTCidentified

   The issue has been identified and a fix is being implemented.

3. 2024년 11월 12일 오후 9:39 UTCidentified

   We are continuing to work on a fix for this issue.

4. 2024년 11월 12일 오후 9:55 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

5. 2024년 11월 12일 오후 10:20 UTCresolved

   This incident has been resolved.

1. 2024년 11월 8일 오후 7:49 UTCinvestigating

   We are currently investigating this issue.

2. 2024년 11월 8일 오후 7:49 UTCidentified

   The issue has been identified and a fix is being implemented.

3. 2024년 11월 8일 오후 7:50 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

4. 2024년 11월 8일 오후 7:52 UTCresolved

   This incident has been resolved.

5. 2024년 11월 8일 오후 7:53 UTCpostmortem

   ### What products were affected and what was the impact? Landed Cost API, Checkout Impact: CRITICAL ### What timeframe did this issue occur? | **Date** | **Time** | | --- | --- | | November 8, 2024 | 8:50am - 10:26am MST | ### How was the issue detected? A spike in error logs triggered an alert to our Engineering team, who responded immediately to the issue. ### What functionality was affected? Landed Cost quotes that use our automated item classification service failed. ### What problems did this cause? If an HS Code was not provided in the API request to Landed Cost, and the automatic classification service was enabled, then the landed cost quote would fail. When the landed cost quote fails, shoppers may not be able to place their order. ### What was the resolution of the problem and steps that are being taken for continued follow-up? The root cause of the issue was a deployment issue with the item service used for automatic classification. While the issue was detected immediately, resolution required rebuilding and redeploying services, which took longer than expected. After services were rebuilt and redeployed, the system health was validated and normal operations resumed. ### What mitigation solutions will we put in place to prevent this issue from occurring in the future? We discovered that this issue was due, in part, to a deficiency in our deployment procedures. We are working to update the procedure to prevent any future issues. We are also creating a synthetic test in our lower environments that will catch similar issues before they can be deployed into production.

1. 2024년 10월 14일 오후 9:21 UTCinvestigating

   We are currently investigating this issue.

2. 2024년 10월 14일 오후 9:22 UTCidentified

   The issue has been identified and a fix is being implemented.

3. 2024년 10월 14일 오후 9:23 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

4. 2024년 10월 14일 오후 9:23 UTCresolved

   This incident has been resolved.

5. 2024년 10월 14일 오후 9:24 UTCpostmortem

   **What products were affected and what was the impact?** On **Saturday, October 12, 2024, at 22:30 UTC**, we experienced an outage that impacted the ability of some Shopify merchants to retrieve landed cost quotes for international orders. This outage lasted until **Sunday, October 13, 2024, at 21:33 UTC**, during which time affected merchants may have been unable to display accurate landed cost calculations to their customers, potentially affecting international checkouts and purchase completions. The root cause of the outage was an API permission issue that was introduced during a routine deployment. Unfortunately, the synthetic tests for the Landed Cost API flow didn’t immediately flag the issue, which caused an extremely uncharacteristic delay to the resolution. It's important to note that this issue was caused by a very unique set of circumstances. We have just finished an extremely complex and expansive migration project to move all of our merchants to a robust and secure new system. This issue was directly caused by processes related to this now complete migration. Impact: critical ‌ **What timeframe did this issue occur?** | **Date** | **Time** | | --- | --- | | October 12, 2024 | 22:30 UTC | | October 13, 2024 | 21:33 UTC | ‌ **How was the issue detected?** The engineering team noticed a warning of failed quotes in the log files. ‌ **What functionality was affected?** All landed cost requests during the outage failed for affected merchants. ‌ **What problems did this cause?** Shoppers of affected merchants were unable to get landed cost quotes, and in many cases complete their checkout. ‌ **What was the resolution of the problem and steps that are being taken for continued follow-up?** Once the issue was identified, our team acted swiftly to permission issue. After thorough testing, we confirmed the API was fully restored at **21:33 UTC** on Sunday. ‌ **What mitigation solutions will we put in place to prevent this issue from occurring in the future?** * **Expanded test coverage**: We are enhancing our synthetic tests to include additional token and permission checks, ensuring that potential issues like this are caught earlier. * **Improved log classifications**: We’re updating our logging processes to better distinguish between warnings and errors, which will enable faster detection and resolution of similar problems in the future.

1. 2024년 10월 8일 오전 1:48 UTCinvestigating

   We are currently investigating this issue.

2. 2024년 10월 8일 오전 3:00 UTCresolved

   This incident has been resolved.