Ιστορικό διακοπών του Zonos

1. 5 Σεπτεμβρίου 2026 στις 12:08 π.μ. UTCinvestigating

   Αυτή τη στιγμή ερευνούμε αυτό το ζήτημα.

2. 5 Σεπτεμβρίου 2026 στις 12:09 π.μ. UTCidentified

   Το θέμα έχει εντοπιστεί και υλοποιείται μια λύση.

3. 5 Σεπτεμβρίου 2026 στις 12:21 π.μ. UTCmonitoring

   Εφαρμόστηκε μια ρύθμιση και παρακολουθούμε τα αποτελέσματα.

4. 5 Σεπτεμβρίου 2026 στις 12:56 π.μ. UTCresolved

   Το περιστατικό λύθηκε.

Αυτόματη μετάφραση από την επίσημη ενημέρωση του συμβάντος.

1. 4 Σεπτεμβρίου 2026 στις 5:29 μ.μ. UTCinvestigating

   Αυτή τη στιγμή ερευνούμε αυτό το ζήτημα.

2. 4 Σεπτεμβρίου 2026 στις 5:30 μ.μ. UTCidentified

   Το θέμα έχει εντοπιστεί και υλοποιείται μια λύση.

3. 4 Σεπτεμβρίου 2026 στις 5:30 μ.μ. UTCmonitoring

   Εφαρμόστηκε μια ρύθμιση και παρακολουθούμε τα αποτελέσματα.

4. 4 Σεπτεμβρίου 2026 στις 7:47 μ.μ. UTCresolved

   Το περιστατικό λύθηκε.

Αυτόματη μετάφραση από την επίσημη ενημέρωση του συμβάντος.

1. 3 Αυγούστου 2026 στις 9:23 μ.μ. UTCinvestigating

   Αυτή τη στιγμή ερευνούμε αυτό το ζήτημα.

2. 3 Αυγούστου 2026 στις 9:24 μ.μ. UTCidentified

   Το θέμα έχει εντοπιστεί και υλοποιείται μια λύση.

3. 3 Αυγούστου 2026 στις 9:57 μ.μ. UTCidentified

   Συνεχίζουμε να εργαζόμαστε πάνω σε μια λύση για αυτό το ζήτημα.

4. 3 Αυγούστου 2026 στις 9:57 μ.μ. UTCmonitoring

   Εφαρμόστηκε μια ρύθμιση και παρακολουθούμε τα αποτελέσματα.

5. 3 Αυγούστου 2026 στις 10:36 μ.μ. UTCmonitoring

   Συνεχίζουμε να παρακολουθούμε για τυχόν περαιτέρω ζητήματα.

6. 3 Αυγούστου 2026 στις 10:36 μ.μ. UTCresolved

   Το περιστατικό λύθηκε.

7. 9 Σεπτεμβρίου 2026 στις 8:06 μ.μ. UTCpostmortem

   Περίληψη συστατικού Στις 3 Αυγούστου 2026, η Zonos παρουσίασε αυξημένα ποσοστά σφαλμάτων που επηρεάζουν το κόστος γης και τις συναφείς υπηρεσίες. Το περιστατικό ξεκίνησε περίπου στις 2:52 μμ MDT και επιλύθηκε στις 3:51 μμ MDT, με αποτέλεσμα περίπου 59 λεπτά υποβαθμισμένης υπηρεσίας. Το περιστατικό ήταν μερική διακοπή: οι αιτήσεις συνέχισαν να υποβάλλονται σε επεξεργασία καθ' όλη τη διάρκεια της εκδήλωσης, αλλά ένα σημαντικό μέρος των αιτήσεων που βασίζονταν στην επηρεαζόμενη υπηρεσία επέστρεψε λάθη. Impact ** Impact ** Η κύρια επηρεαζόμενη υπηρεσία παρουσίασε περίπου 50% ποσοστό σφάλματος για τις αιτήσεις GraphQL κατά τη διάρκεια του παραθύρου συμβάντος. Ως αποτέλεσμα, οι πελάτες που χρησιμοποιούσαν το Landed Cost παρουσίασαν διαλείπουσες αποτυχίες στο αίτημα, ενώ άλλα αιτήματα συνέχισαν να επεξεργάζονται επιτυχώς. Επειδή το Landed Cost βασίζεται σε κοινές υπηρεσίες πλατφόρμας, το υποκείμενο ζήτημα είχε επίσης ως αποτέλεσμα την αύξηση των ποσοστών σφάλματος σε ορισμένες σχετικές ροές εργασίας. Αιτίες για τα σκουπίδια Το περιστατικό πυροδοτήθηκε από μια ανάπτυξη παραγωγής που άρχισε να χρησιμοποιεί μια νέα διαδρομή πρόσβασης δεδομένων μέσα στην πλατφόρμα μας. Στο πλαίσιο του όγκου των αιτήσεων παραγωγής, το εν λόγω πρότυπο πρόσβασης δημιούργησε σημαντικά μεγαλύτερη δραστηριότητα βάσης δεδομένων από ό,τι αναμενόταν. Αυτό εξάντλησε τις διαθέσιμες συνδέσεις βάσης δεδομένων μέσα σε μια υποστηρικτική υπηρεσία και προκάλεσε αιτήματα ανάλογα με την υπηρεσία αυτή να αποτύχει. Το υποκείμενο ζήτημα δεν είχε εκδηλωθεί προηγουμένως επειδή η πληγείσα λειτουργικότητα δεν είχε ακόμη ασκηθεί σε κλίμακα παραγωγής. ¶ ** Λύση ** Η ομάδα μηχανικών μας αρχικά γύρισε πίσω μια σχετική ανάπτυξη υπηρεσιών κατά τη διερεύνηση της πηγής των σφαλμάτων. Όταν αυτό δεν επιλύθηκε το ζήτημα, η ανάπτυξη του κόστους προσγείωσης που ενεργοποίησε το προβληματικό μοτίβο πρόσβασης ανακλήθηκε. Η υπηρεσία ανακτήθηκε αμέσως μετά την αναστροφή. Στη συνέχεια εφαρμόστηκε μόνιμη διόρθωση για να βελτιστοποιηθεί το πρότυπο πρόσβασης στη βάση δεδομένων και να δρομολογηθεί κατάλληλα ο φόρτος εργασίας που επηρεάστηκε. ** Προληπτικές ενέργειες ** Μετά το περιστατικό, επανεξετάσαμε τόσο την άμεση αποτυχία όσο και τον τρόπο με τον οποίο διαδόθηκε μέσω των εξαρτώμενων υπηρεσιών. Παίρνουμε διάφορες δράσεις για να μειώσουμε την πιθανότητα και τον πιθανό αντίκτυπο παρόμοιων περιστατικών, συμπεριλαμβανομένων: * Προσθήκη πρόσθετης παρακολούθησης και ειδοποίησης για κορεσμό σύνδεσης βάσης δεδομένων. * Ενίσχυση των διασφαλίσεων και των πρακτικών αναθεώρησης για ερωτήματα βάσης δεδομένων κατά χωρισμένα δεδομένα. * Επέκταση των δοκιμών των νέων ενεργοποιημένων προτύπων πρόσβασης δεδομένων υπό την παραγωγή-όπως το φορτίο. * Επανεξέταση της απομόνωσης των υπηρεσιών και της αποτυχίας χειρισμού για τη μείωση των επιπτώσεων μια υποβαθμισμένη εξάρτηση μπορεί να έχει σε άλλες υπηρεσίες πλατφόρμας. * Επανεξέταση παρόμοιων προτύπων πρόσβασης δεδομένων σε όλη την πλατφόρμα για τις συνθήκες που προσδιορίζονται κατά τη διάρκεια αυτού του συμβάντος. Ζητάμε συγγνώμη για την αναστάτωση που προκάλεσε το περιστατικό. Θα συνεχίσουμε να χρησιμοποιούμε τα ευρήματα από αυτό το γεγονός για να βελτιώσουμε την ανθεκτικότητα και την παρατηρησιμότητα της πλατφόρμας Ζώνος.

Αυτόματη μετάφραση από την επίσημη ενημέρωση του συμβάντος.

1. 8 Ιουλίου 2026 στις 7:41 μ.μ. UTCinvestigating

   We are currently investigating this issue.

2. 8 Ιουλίου 2026 στις 7:42 μ.μ. UTCidentified

   The issue has been identified and a fix is being implemented.

3. 8 Ιουλίου 2026 στις 7:42 μ.μ. UTCmonitoring

   A fix has been implemented and we are monitoring the results.

4. 8 Ιουλίου 2026 στις 9:00 μ.μ. UTCresolved

   This incident has been resolved.

1. 26 Μαΐου 2026 στις 9:27 μ.μ. UTCinvestigating

   An issue with an authentication provider is currently affecting some users.

2. 26 Μαΐου 2026 στις 9:29 μ.μ. UTCidentified

   The issue has been identified and a fix is being implemented.

3. 26 Μαΐου 2026 στις 9:41 μ.μ. UTCmonitoring

   A fix has been implemented and we are monitoring the results.

4. 26 Μαΐου 2026 στις 9:54 μ.μ. UTCmonitoring

   We are continuing to see increased error rates and are monitoring for a fix from our authentication provider.

5. 26 Μαΐου 2026 στις 9:59 μ.μ. UTCmonitoring

   We are seen error rates decline and will continue monitoring.

6. 26 Μαΐου 2026 στις 10:31 μ.μ. UTCresolved

   This incident has been resolved.

1. 11 Μαρτίου 2026 στις 9:19 μ.μ. UTCinvestigating

   We are currently investigating this issue.

2. 11 Μαρτίου 2026 στις 9:19 μ.μ. UTCidentified

   The issue has been identified and a fix is being implemented.

3. 11 Μαρτίου 2026 στις 9:20 μ.μ. UTCmonitoring

   A fix has been implemented and we are monitoring the results.

4. 11 Μαρτίου 2026 στις 9:21 μ.μ. UTCresolved

   This incident has been resolved.

5. 11 Μαρτίου 2026 στις 9:25 μ.μ. UTCpostmortem

   **What products were affected and what was the impact?** Classify, Country of Origin Impact: DEGRADED PERFORMANCE   **What timeframe did this issue occur?** | **Date** | **Time** | | --- | --- | | Mar 11, 2026 | 2026-03-11 10:12 - 2026-03-11 12:06 |   **How was the issue detected?** Zonos observed elevated error rates affecting Classify and Country of Origin APIs. ‌ **What functionality was affected?** A portion of Classify and Country of Origin inference requests did not return a valid response. ‌ **What was the root cause?** A firmware issue on a subset of GPU nodes caused network connectivity failures. This caused the nodes to enter an unhealthy state. Our reconnect logic did not properly remove unhealthy nodes from the pool, so they continued to receive requests. ‌ **What was the resolution of the problem, and what steps are being taken to prevent future issues?** Shortly after discovering the issue, we restarted nodes to resolve the connectivity issues and restore service. We then identified the reconnect logic issue, and released a fix to prevent future issues. We have scheduled a firmware upgrade on all nodes to resolve the underlying network issue. We have also refined our alerting to catch errors more quickly in the future.

1. 22 Φεβρουαρίου 2026 στις 6:25 μ.μ. UTCinvestigating

   We are currently investigating this issue.

2. 22 Φεβρουαρίου 2026 στις 6:28 μ.μ. UTCinvestigating

   We are continuing to investigate this issue.

3. 22 Φεβρουαρίου 2026 στις 7:16 μ.μ. UTCidentified

   The issue has been identified and a fix is being implemented.

4. 23 Φεβρουαρίου 2026 στις 3:45 μ.μ. UTCmonitoring

   A fix has been implemented and we are monitoring the results.

5. 23 Φεβρουαρίου 2026 στις 4:22 μ.μ. UTCresolved

   This incident has been resolved.

1. 19 Φεβρουαρίου 2026 στις 9:58 μ.μ. UTCinvestigating

   We are currently investigating this issue.

2. 19 Φεβρουαρίου 2026 στις 9:59 μ.μ. UTCidentified

   The issue has been identified and a fix is being implemented.

3. 19 Φεβρουαρίου 2026 στις 9:59 μ.μ. UTCmonitoring

   A fix has been implemented and we are monitoring the results.

4. 19 Φεβρουαρίου 2026 στις 10:24 μ.μ. UTCresolved

   This incident has been resolved.

1. 19 Φεβρουαρίου 2026 στις 4:29 μ.μ. UTCinvestigating

   We are currently investigating this issue.

2. 19 Φεβρουαρίου 2026 στις 4:32 μ.μ. UTCidentified

   The issue has been identified and a fix is being implemented.

3. 19 Φεβρουαρίου 2026 στις 4:58 μ.μ. UTCmonitoring

   A fix has been implemented and we are monitoring the results.

4. 19 Φεβρουαρίου 2026 στις 5:00 μ.μ. UTCmonitoring

   We are continuing to see some issues and are continuing to monitor.

5. 19 Φεβρουαρίου 2026 στις 5:26 μ.μ. UTCmonitoring

   We are continuing to monitor for any further issues.

6. 19 Φεβρουαρίου 2026 στις 5:48 μ.μ. UTCmonitoring

   The issue has been resolved and operations have returned to normal. We are actively monitoring to ensure continued stability.

7. 19 Φεβρουαρίου 2026 στις 6:57 μ.μ. UTCresolved

   This incident has been resolved.

1. 20 Οκτωβρίου 2025 στις 9:57 μ.μ. UTCinvestigating

   We are currently investigating this issue.

2. 20 Οκτωβρίου 2025 στις 9:57 μ.μ. UTCidentified

   The issue has been identified and a fix is being implemented.

3. 20 Οκτωβρίου 2025 στις 9:57 μ.μ. UTCmonitoring

   A fix has been implemented and we are monitoring the results.

4. 20 Οκτωβρίου 2025 στις 9:57 μ.μ. UTCresolved

   This incident has been resolved.

1. 4 Σεπτεμβρίου 2025 στις 7:59 μ.μ. UTCinvestigating

   We are currently investigating this issue.

2. 4 Σεπτεμβρίου 2025 στις 8:17 μ.μ. UTCresolved

   Users logging in may have experienced a flashing screen. The change was rolled back and the issue has been resolved.

1. 10 Ιουνίου 2025 στις 2:54 μ.μ. UTCinvestigating

   An issue with an upstream infrastructure provider is impacting our BigCommerce Plugin

2. 10 Ιουνίου 2025 στις 2:54 μ.μ. UTCidentified

   The issue has been identified and we are continuing to monitor the impact.

3. 10 Ιουνίου 2025 στις 2:55 μ.μ. UTCidentified

   We are continuing to work on a fix for this issue.

4. 10 Ιουνίου 2025 στις 4:00 μ.μ. UTCresolved

   This incident has been resolved.

1. 13 Φεβρουαρίου 2025 στις 9:52 μ.μ. UTCinvestigating

   We are currently investigating this issue.

2. 13 Φεβρουαρίου 2025 στις 9:53 μ.μ. UTCresolved

   This incident has been resolved.

3. 13 Φεβρουαρίου 2025 στις 9:56 μ.μ. UTCpostmortem

   **What products were affected and what was the impact?** * Checkout \(Custom Integrations\) Impact: * Major outage for affected merchants **What timeframe did this issue occur?** |   | **Date** | **Time** | | --- | --- | --- | | From: | Jan 24, 2025 | 12:00 PM MST | | To: | Jan 27, 2025 | 07:45 AM MST |   **How was the issue detected?** Our team was notified that Checkout domains were not being allowed as expected, leading to an immediate investigation by the engineering team. **What functionality was affected?** Online store settings \(allowed store domain\) were not loading properly for all stores using custom integrations. This prevented affected stores from loading checkout. ‌ **What problems did this cause?** Stores relying on custom integrations could not load their checkout correctly. ‌ **What was the resolution of the problem and steps that are being taken for continued follow-up?** * A rollback to a previous version immediately resolved the issue. * A new synthetic test has been implemented to ensure store settings caching is store-specific and does not mix settings between stores. ‌ ‌ **What mitigation solutions will we put in place to prevent this issue from occurring in the future?** * **Improved Testing:** A new synthetic test has been introduced to verify caching behavior and prevent incorrect store settings from being cached. * **Staging Environment:** Setting up a dedicated test environment for custom integrations to catch similar issues earlier.

1. 13 Φεβρουαρίου 2025 στις 8:52 μ.μ. UTCinvestigating

   We are currently investigating this issue.

2. 13 Φεβρουαρίου 2025 στις 8:53 μ.μ. UTCresolved

   This incident has been resolved.

3. 13 Φεβρουαρίου 2025 στις 9:27 μ.μ. UTCpostmortem

   ### What products were affected and what was the impact? Shopify Checkout Partial outage: * Affected some Shopify merchants ### What timeframe did this issue occur? | | **Date** | **Time** | | --- | --- | --- | | From: | Jan 23, 2025 | 05:30 PM MDT | | To: | Jan 24, 2025 | 03:30 PM MDT | ### How was the issue detected? * A discrepancy was identified when some orders in Shopify were missing tracking numbers. * Additionally, merchants using older versions of the Shopify app experienced order import failures. ‌ ### What functionality was affected? * Order fulfillment ‌ ### What problems did this cause? * **Order Processing Issues:** Orders processed without tracking numbers caused confusion and delays for merchants and customers. * **Order Import Failures:** Some merchants experienced missing orders due to import failures. ‌ ### What was the resolution of the problem and steps that are being taken for continued follow-up? * We patched the root cause of the issue, resolving the tracking number and order import failures. * We updated all the affected orders with tracking numbers and manually imported orders that had failed to import. * API headers and logging enhancements were implemented to improve issue detection and response time. ### What mitigation solutions will we put in place to prevent this issue from occurring in the future? * Expand QA coverage to address downstream impacts of refactors, with a focus on comprehensive testing of critical flows, including label creation. * Expand synthetic test suite to validate end-to-end order processing, with particular emphasis on fulfillment workflows. * Establish a consistent testing environment that simulates real-world Shopify store configurations, reducing the risk of silent failures.

1. 13 Φεβρουαρίου 2025 στις 8:30 μ.μ. UTCinvestigating

   We are currently investigating this issue.

2. 13 Φεβρουαρίου 2025 στις 8:31 μ.μ. UTCresolved

   This incident has been resolved.

3. 13 Φεβρουαρίου 2025 στις 8:38 μ.μ. UTCpostmortem

   **What products were affected and what was the impact?** * All Magento Checkout merchants Impact: * Partial Outage **What timeframe did this issue occur?** | | **Date** | **Time** | | --- | --- | --- | | From: | Jan 6, 2025 | 2:27 PM MST | | To: | Jan 7, 2025 | 09:11 AM MST | ‌ ### How was the issue detected? Our team was notified that carts were not being created for some Magento shoppers, leading to an immediate investigation by the engineering team. ‌ ### What functionality was affected? The checkout process for Magento merchants encountered validation failures due to incorrect UUID handling. This disrupted the checkout flow, impacting merchants using the Magento Checkout. ‌ ### What problems did this cause? * Magento merchants experienced disruptions in checkouts being loaded at times. ‌ ### What was the resolution of the problem and steps that are being taken for continued follow-up? * The issue was identified as stemming from changes made to TSID validation. * A rollback of the problematic changes was performed promptly, restoring checkout functionality. * A synthetic test was created to validate UUID behavior and detect future occurrences of this issue. * Code annotations have been added to document known issues and improve future debugging. ‌ ### What mitigation solutions will we put in place to prevent this issue from occurring in the future? * Establishing alerts detecting deviations in carts-to-checkout sessions. This will help flag potential checkout issues early. * Increasing test coverage to identify gaps in synthetic testing.

1. 13 Φεβρουαρίου 2025 στις 6:51 μ.μ. UTCinvestigating

   We are experiencing an issue with orders not being created despite payments being processed successfully.

2. 13 Φεβρουαρίου 2025 στις 6:53 μ.μ. UTCresolved

   This incident has been resolved.

3. 13 Φεβρουαρίου 2025 στις 6:54 μ.μ. UTCpostmortem

   **What products were affected and what was the impact?** * Zonos Checkout Impact: * Major Outage **What timeframe did this issue occur?** | | **Date** | **Time** | | --- | --- | --- | | From: | Jan 3, 2025 | 01:30 PM MST | | To: | Jan 3, 2025 | 07:06 PM MST | ‌ ### How was the issue detected? Our team was notified that payments were being processed, but orders were not being created. ‌ ### What functionality was affected? * Ability to create orders * Order fulfillment ‌ ### What problems did this cause? * Customer frustration due to failed order creation despite successful payment processing. * Potential financial reconciliation issues for merchants and customers. ‌ ### What was the resolution of the problem and steps that are being taken for continued follow-up? * The root cause was identified as a backend change, which introduced a bug related to the separation of payment intent and checkout session. * A rollback was implemented to restore functionality, and error logging for Stripe intent failures has now been enabled. ‌ ### What mitigation solutions will we put in place to prevent this issue from occurring in the future? * Synthetic tests have been updated to run every 10 minutes to narrow down problematic PRs and detect failures earlier.

1. 12 Νοεμβρίου 2024 στις 8:51 μ.μ. UTCinvestigating

   We are currently investigating elevated error rates.

2. 12 Νοεμβρίου 2024 στις 9:00 μ.μ. UTCidentified

   The issue has been identified and a fix is being implemented.

3. 12 Νοεμβρίου 2024 στις 9:39 μ.μ. UTCidentified

   We are continuing to work on a fix for this issue.

4. 12 Νοεμβρίου 2024 στις 9:55 μ.μ. UTCmonitoring

   A fix has been implemented and we are monitoring the results.

5. 12 Νοεμβρίου 2024 στις 10:20 μ.μ. UTCresolved

   This incident has been resolved.

1. 8 Νοεμβρίου 2024 στις 7:49 μ.μ. UTCinvestigating

   We are currently investigating this issue.

2. 8 Νοεμβρίου 2024 στις 7:49 μ.μ. UTCidentified

   The issue has been identified and a fix is being implemented.

3. 8 Νοεμβρίου 2024 στις 7:50 μ.μ. UTCmonitoring

   A fix has been implemented and we are monitoring the results.

4. 8 Νοεμβρίου 2024 στις 7:52 μ.μ. UTCresolved

   This incident has been resolved.

5. 8 Νοεμβρίου 2024 στις 7:53 μ.μ. UTCpostmortem

   ### What products were affected and what was the impact? Landed Cost API, Checkout Impact: CRITICAL ### What timeframe did this issue occur? | **Date** | **Time** | | --- | --- | | November 8, 2024 | 8:50am - 10:26am MST | ### How was the issue detected? A spike in error logs triggered an alert to our Engineering team, who responded immediately to the issue. ### What functionality was affected? Landed Cost quotes that use our automated item classification service failed. ### What problems did this cause? If an HS Code was not provided in the API request to Landed Cost, and the automatic classification service was enabled, then the landed cost quote would fail. When the landed cost quote fails, shoppers may not be able to place their order. ### What was the resolution of the problem and steps that are being taken for continued follow-up? The root cause of the issue was a deployment issue with the item service used for automatic classification. While the issue was detected immediately, resolution required rebuilding and redeploying services, which took longer than expected. After services were rebuilt and redeployed, the system health was validated and normal operations resumed. ### What mitigation solutions will we put in place to prevent this issue from occurring in the future? We discovered that this issue was due, in part, to a deficiency in our deployment procedures. We are working to update the procedure to prevent any future issues. We are also creating a synthetic test in our lower environments that will catch similar issues before they can be deployed into production.

1. 14 Οκτωβρίου 2024 στις 9:21 μ.μ. UTCinvestigating

   We are currently investigating this issue.

2. 14 Οκτωβρίου 2024 στις 9:22 μ.μ. UTCidentified

   The issue has been identified and a fix is being implemented.

3. 14 Οκτωβρίου 2024 στις 9:23 μ.μ. UTCmonitoring

   A fix has been implemented and we are monitoring the results.

4. 14 Οκτωβρίου 2024 στις 9:23 μ.μ. UTCresolved

   This incident has been resolved.

5. 14 Οκτωβρίου 2024 στις 9:24 μ.μ. UTCpostmortem

   **What products were affected and what was the impact?** On **Saturday, October 12, 2024, at 22:30 UTC**, we experienced an outage that impacted the ability of some Shopify merchants to retrieve landed cost quotes for international orders. This outage lasted until **Sunday, October 13, 2024, at 21:33 UTC**, during which time affected merchants may have been unable to display accurate landed cost calculations to their customers, potentially affecting international checkouts and purchase completions. The root cause of the outage was an API permission issue that was introduced during a routine deployment. Unfortunately, the synthetic tests for the Landed Cost API flow didn’t immediately flag the issue, which caused an extremely uncharacteristic delay to the resolution. It's important to note that this issue was caused by a very unique set of circumstances. We have just finished an extremely complex and expansive migration project to move all of our merchants to a robust and secure new system. This issue was directly caused by processes related to this now complete migration. Impact: critical ‌ **What timeframe did this issue occur?** | **Date** | **Time** | | --- | --- | | October 12, 2024 | 22:30 UTC | | October 13, 2024 | 21:33 UTC | ‌ **How was the issue detected?** The engineering team noticed a warning of failed quotes in the log files. ‌ **What functionality was affected?** All landed cost requests during the outage failed for affected merchants. ‌ **What problems did this cause?** Shoppers of affected merchants were unable to get landed cost quotes, and in many cases complete their checkout. ‌ **What was the resolution of the problem and steps that are being taken for continued follow-up?** Once the issue was identified, our team acted swiftly to permission issue. After thorough testing, we confirmed the API was fully restored at **21:33 UTC** on Sunday. ‌ **What mitigation solutions will we put in place to prevent this issue from occurring in the future?** * **Expanded test coverage**: We are enhancing our synthetic tests to include additional token and permission checks, ensuring that potential issues like this are caught earlier. * **Improved log classifications**: We’re updating our logging processes to better distinguish between warnings and errors, which will enable faster detection and resolution of similar problems in the future.

1. 8 Οκτωβρίου 2024 στις 1:48 π.μ. UTCinvestigating

   We are currently investigating this issue.

2. 8 Οκτωβρίου 2024 στις 3:00 π.μ. UTCresolved

   This incident has been resolved.