Zid outage history

We are facing and have identified a partial outage in our Merchant Imports & Exports service and are working on resolving it as soon as possible.

This incident has been resolved.

# Incident Post-mortem Report **Dashboard exports and imports delayed for all Stores** | | | | --- | --- | | **Date** | Oct 25, 2026, 11:25 AM until 2:26 PM \(UTC\+3, KSA time\). Duration: 3 hours 1 minute | | **Type of Incident** | Service degradation: background processing | | **Title** | A very large customer import stalled dashboard exports and imports for all Stores | | **Impact** | 132 export requests from 66 Stores were delayed, by 81 minutes on average and up to 3 hours 43 minutes. 3 of them timed out and must be requested again. The customer imports of the Store that started the large import were delayed until 3:12 PM. | | **Total impact on uptime** | No outage. Stores, checkout and the dashboard worked the whole time. Only exports and imports waited. | ## Summary Dashboard exports \(orders, customers, wallet, abandoned carts and others\) and dashboard imports are built in the background by one shared pool of workers. At 11:23 AM a Merchant started a customer import with about 1.05 million rows. The platform splits every import into parts of 100 rows, so this one became 10,486 parts. After each part, the platform checks whether the whole import has finished, so it can send the Merchant a summary. That check loaded every part of the import, rows included, into memory: about 200 MB per check for this import, with up to five checks running at once. The workers ran out of memory and restarted again and again. Export requests from every Store waited behind the import parts. We shipped a code-level change so the check only asks whether any part is still pending and adds up the final totals in the database. It no longer loads the rows, so its memory use no longer grows with the size of the import. ## Timeline \(KSA time\) | Time | Event | | --- | --- | | 11:23 AM | A Merchant starts a customer import of about 1.05 million rows. By 11:27 AM it is split into 10,486 parts. | | 11:25 AM | The first export request that has to wait is created. Later requests queue behind the import parts. | | ~11:34 AM | The workers start running out of memory and restarting. From here they finish about 5 import parts per restart and almost no exports. | | 12:26 PM | A Merchant reports to support that 3 exports requested since 11:30 AM are still processing. | | ~1:15 PM | Product Engineering starts the investigation and traces the restarts to the import completion check. | | 1:54 PM | Product Engineering prepares the change, with a test that reproduces the memory growth. | | 2:08 PM | The workers start clearing the backlog, although they still restart from time to time. | | 2:26 PM | The last delayed export finishes. Exports are back to normal. | | 2:34 PM | The change is live. | | ~3:00 PM | The worker restarts stop. | | 3:09 PM | 3 export requests that had waited more than 3 hours are marked as timed out. | | 3:10 PM | The large import finishes, except one part of 100 rows that the platform retries automatically. | ## What was affected | Area | State during the incident | How we know | | --- | --- | --- | | Dashboard exports, all types and all Stores | **Delayed:** 132 requests from 66 Stores, 81 minutes on average, up to 3 hours 43 minutes | Export request records. Orders 72, detailed orders 21, wallet 11, abandoned carts 6, customers 6, reviews 5, order preparation 4, coupons 3, other 4. | | Exports that timed out | **Failed:** 3 requests from 2 Stores | Export request records. These Merchants need to request the export again. | | Exports still processing | **1 request** | Export request records. The platform retries it automatically. | | Customer imports of the Store that started the large import | **Delayed until 3:12 PM:** 9 imports | Import records. All parts finished except one, which is retried automatically. | | Imports from other Stores | **Not affected** | No other Store started an import during the incident | | Export file contents | **Not affected** | Each file is built from live data when it runs, so delayed files are complete | | Stores, checkout and the dashboard | **Not affected** | Only the background workers for exports and imports restarted | ## Action taken 1. Found the cause: the workers that build exports and imports were restarting because the import completion check ran out of memory. 2. Shipped a code-level change: the completion check counts pending parts and adds up the totals in the database instead of loading every part. A test fails on the old behaviour and passes on the new one. 3. Confirmed that the delayed exports finished, and identified the 3 timed-out requests so support can ask those Merchants to request them again. ## Conclusions **One Merchant's import should not hold up every other Store's exports.** Exports and imports share the same workers, so a heavy import delays everyone. The change removes this cause. **Work that runs once per part must not grow with the size of the whole import.** The completion check read the entire import after every part. That was fine for typical imports and failed only for a very large one. With the change, an import of this size no longer exhausts the workers' memory.

تجارنا الأعزاء، نواجه حالياً اضطراباً جزئياً في بعض خدماتنا نتيجة مشكلة تقنية لدى أحد مزودي الخدمة الخارجيين. يعمل فريقنا التقني بشكل مكثف وبالتنسيق المباشر مع مزود الخدمة لمعالجة المشكلة واستعادة جميع الخدمات إلى وضعها الطبيعي في أسرع وقت ممكن. نعتذر عن أي إزعاج قد يسببه هذا الانقطاع، وسنوافيكم بآخر المستجدات فور توفرها. نشكركم على صبركم وتفهمكم. Dear Merchants, We are currently experiencing a partial service disruption due to an issue affecting one of our third-party service providers. Our engineering team is actively investigating and working with the provider to restore full service availability as quickly as possible. We apologize for any inconvenience and will provide further updates as more information becomes available. Thank you for your patience and understanding.

تم حل المشكلة: عادت تصاميم المتاجر إلى العمل تجارنا الأعزاء، عادت الخدمة إلى العمل، وتظهر واجهات المتاجر الآن بتصاميمها المعتادة. إن ظهر متجرك لأحد العملاء دون تنسيق، فيكفي تحديث الصفحة أو فتحها من جديد. لا يلزمك أي إجراء آخر. سنواصل متابعة الخدمة للتأكد من استقرارها. نعتذر عن الإزعاج، ونشكر لكم صبركم وتفهمكم. --- Resolved: Store designs are loading again Dear Merchants, The service is back, and storefronts are loading with their usual designs. If a shopper still sees a store without styling, refreshing the page will fix it. You don't need to do anything else. We will keep monitoring to make sure the service stays stable. We apologize for the disruption and thank you for your patience.

ملخص الحادثة: خلل في ظهور الثيمات وتخصيصها للمتاجر — ٩ أكتوبر ٢٠٢٦ ماذا حدث ظهرت بعض واجهات المتاجر يوم ٩ أكتوبر ٢٠٢٦ دون تنسيقاتها، فبدت صفحاتها نصوصاً وروابط متتابعة بلا تصميم. بدأ الخلل قرابة الساعة 1:06 ظهراً بتوقيت السعودية، وانتهى قرابة الساعة 2:48 عصراً. السبب أوقف أحد مزودي البنية التحتية خدمة التخزين التي تُحمَّل منها ملفات تصاميم المتاجر، والسبب خطأ في الفوترة والمحاسبة لدى المزوّد. لم يكن الخلل تقنياً، ولم يكن من جهة زد. الأثر • تأثر شكل واجهات المتاجر لدى العملاء طوال مدة الخلل. • لم تتأثر بيانات المتاجر ولا المنتجات ولا الطلبات، ولم يُفقد منها شيء. • ظلت صور المنتجات تعمل طوال مدة الخلل. الحل تواصلنا مع المزوّد مباشرة حتى صحّح الخطأ لديه وأعاد تفعيل الخدمة، فعادت المتاجر إلى شكلها الطبيعي دون أن يلزم التجار أي إجراء. ما سنفعله لمنع تكراره سنبني مصدراً بديلاً لملفات تصاميم المتاجر، تنتقل إليه المتاجر تلقائياً إذا تعطّل المصدر الأساسي، فلا يتوقف ظهور المتاجر بسبب مزوّد واحد. نعتذر عن الإزعاج الذي سببه هذا الخلل لكم ولعملائكم، ونشكر لكم صبركم وثقتكم. --- Incident summary: Store themes and customizations not loading — October 9, 2026 What happened On October 9, 2026, some storefronts loaded without their styling, so pages showed as plain text and links. The issue began at about 1:06 PM Saudi time and ended at about 2:48 PM. Cause One of our infrastructure providers suspended the storage service that serves store theme files because of a billing and accounting error on their side. This was not a technical failure, and it was not caused by Zid. Impact • Storefronts looked unstyled to shoppers for the duration of the incident. • Store data, products and orders were not affected, and nothing was lost. • Product images kept working throughout. Resolution We worked with the provider directly until they corrected their error and re-enabled the service. Stores returned to normal without merchants having to do anything. Preventing a repeat We will add a backup source for store theme files. If the primary source fails, stores will switch to the backup automatically, so a single provider can no longer take down how stores look. We apologize for the disruption to you and your customers, and we thank you for your patience and trust.

أعزاؤنا التجار، واجهت متاجرنا مشكلة في صفحة تسجيل الدخول ابتداءً من الساعة 3:17 مساءً وحتى 3:37 مساءً بتوقيت السعودية. خلال هذه الفترة، لم يتمكن المستخدمون مؤقتًا من الوصول إلى صفحة تسجيل الدخول. لقد تم تحديد المشكلة والسيطرة عليها، وعاد النظام إلى العمل بشكل طبيعي. تبيّن أن السبب كان عملية نشر غير سليمة لتطبيق الواجهة الأمامية React، وتم التراجع عن التغيير ومعالجة المشكلة بشكل فوري. كما نعمل حاليًا على مراجعة عمليات النشر والتحقق الخاصة بالواجهة الأمامية للمساعدة في منع تكرار مثل هذه المشكلات مستقبلًا. نأسف للإزعاج، ونقدّر تفهمكم. مع خالص التحيات، Dear Merchants, Our stores experienced an issue affecting the login page from 3:17 PM to 3:37 PM KSA, during which users were temporarily unable to access the login page. The issue has been identified and resolved, and the system is now operating normally. The issue was caused by a faulty frontend React deployment. The change was promptly reverted and remediated. We are currently reviewing our frontend deployment and validation processes to help prevent similar issues from recurring. We apologize for the inconvenience and appreciate your understanding.

Some stores were experiencing slower than usual response times.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

The team released an incident fix for an issue where the count of products under categories was not calculated correctly, and was instead showing 0 for parent categories whose products are all placed inside child categories. As a result of this fix, there was an increase in DB queries, leading to a spike in CPU utilization of our database. This happened mainly due to cache warming jobs used in the old catalog system. While the fix was intended for the Dashboard, an inadvertent consequence happened on “Old Catalog” – not Vitrin \(stores on Vitrin were browsable safely\). During that period, some users who did not have the store cached on their end may have experienced downtime with some stores. Others experienced degraded performance only on cart actions. ‌ Action Taken: 1. We immediately reverted the troublesome deployment. 2. The congested queues for store cashing were purged to avoid having blocked jobs 3. Assembled a rapid response team to decouple this dependency and prevent such incidents in the future.

Start Time: 2025-11-18 2:48 PM Impact: Some users may experience intermittent errors or inability to access parts of the Zid platform. Cloudflare has reported a global network issue and is currently investigating. As Zid’s traffic flows through Cloudflare’s network, this incident may impact accessibility for some users, depending on their region and routing. Our systems and infrastructure remain healthy, and the issue is external. We are actively monitoring Cloudflare’s updates and will provide further details as soon as they are available. أعلنت كلاودفلاير عن وجود مشكلة في شبكتها العالمية وهي حالياً في مرحلة التحقيق. ونظرًا لاعتماد منصة زد على شبكة كلاودفلاير لتوجيه حركة المرور، قد يتأثر جزء من المستخدمين حسب المنطقة ومسار الاتصال. جميع خوادمنا تعمل بشكل طبيعي، والمشكلة خارج نطاق منصة زد. نقوم بمتابعة الموقف بشكل مستمر، وسيتم نشر أي تحديثات جديدة فور توفرها من كلاودفلاير.

This incident has been resolved.

Dears, We experienced a brief service disruption due to a DDoS attack, which caused temporary unavailability of our stores for approximately 3 minutes. The issue was promptly identified and successfully mitigated by our security team. All systems are now operating normally. Thank you for your patience and understanding. اعزائنا التجار لقد واجهنا انقطاعًا مؤقتًا في الخدمة نتيجة لهجوم (DDoS) مما أدى إلى تعذر الوصول إلى متاجرنا لمدة تقارب 3 دقائق. وقد تم التعرف على المشكلة بسرعة والتعامل معها بنجاح من قبل فريق الأمن لدينا. جميع الأنظمة تعمل الآن بشكل طبيعي. نشكر لكم تفهمكم.

Our stores are currently experiencing downtime. We’re actively working on resolving the issue and will keep you updated on the progress. Thank you for your patience! 🙏

Our stores are back online! Our team has successfully handled the issue, and we are now in the monitoring phase to ensure everything runs smoothly.

أعزاؤنا التجار، واجهت متاجرنا بطئًا في عربة التسوق استمر لحوالي 20 دقيقة، مما تسبب في تأخير تأكيد بعض الطلبات . لقد تم تحديد المشكلة والسيطرة عليها، وعاد النظام إلى العمل بشكل طبيعي. نحن نواصل مراقبة الوضع لضمان استقرار المنصة. نأسف للإزعاج، ونقدّر تفهمكم. مع خالص التحيات، Dear Merchants, Our stores experienced slowness in the cart for about 20 minutes, which caused some orders to take longer to confirm. The issue has been identified and resolved, and the system is now operating normally. We continue to monitor the situation to ensure platform stability. We apologize for the inconvenience and appreciate your understanding.

Dear Merchants, We are currently facing an issue impacting the POS and MyZid applications on both iOS and Android platforms, which may result in unexpected logouts. Our team is actively working on a resolution and will provide updates shortly. Thank you for your patience and understanding as we work to resolve this issue promptly. أعزاءنا التجار نواجه حاليًا مشكلة تؤثر على تطبيقات نقاط البيع و ماي زد، وقد تؤدي إلى تسجيل خروج غير متوقع يعمل فريقنا بنشاط على إيجاد حل وسوف يقدم تحديثات قريبًا نشكرك على صبركم وتفهمكم بينما نعمل على حل هذه المشكلة على الفور.

Here’s the latest update: • The issue with ZIDPOS has been resolved. - Users should force close the app, and it will automatically load the new configuration. • MyZID is still under investigation. - On iOS, the update is still in progress. - For Android, the update is still in progress. آخر التحديثات: • تم حل مشكلة ZIDPOS ينبغي على المستخدمين إغلاق واعادة تشغيل التطبيق ، وسيتم تحميل الإعدادات الجديدة تلقائيًا. • مشكلة MyZID لا تزال قيد التحقق

Dear Merchants, We are pleased to inform you that the issue impacting the POS and MyZid applications on both iOS and Android platforms, which resulted in unexpected logouts, has been resolved. السادة التجار الكرام، نود إعلامكم بأنه قد تم حل المشكلة التي أثرت على تطبيقات POS و MyZid والتي تسببت في عمليات تسجيل خروج غير متوقعة.

Dears, We experienced a brief disruption in our stores frontend service that lasted approximately 3 minutes due to an unexpected technical issue. The disruption affected random stores, not all. Our team immediately identified the problem, took full control, and resolved the issue swiftly. The service is now fully operational. أعزاؤنا التجار لقد واجهنا انقطاعًا قصيرًا في خدمة الواجهة الأمامية للمتاجر استمر لمدة حوالي 3 دقائق نتيجة لمشكلة تقنية غير متوقعة. تأثرت بعض المتاجر بشكل عشوائي وليس جميعها. قام فريقنا بتحديد المشكلة على الفور، وتم حل المشكلة بسرعة. الخدمة الآن تعمل بشكل كامل، ونحن نراقب لضمان استقرارها. نعتذر عن أي إزعاج قد تسبب به هذا الأمر و نقدر تفهمكم.

The system experienced a brief outage lasting approximately 4 minutes due to an unexpected technical issue. Our team promptly identified and resolved the problem, ensuring minimal disruption to services. We are taking steps to prevent similar occurrences in the future.

On September 22, 2024, at around 1:42 AM KSA time (GMT+3), our stores encountered a brief outage lasting approximately one minute, caused by an unexpected technical issue that was not managed by our service provider.

Dear Merchants, Some of our stores have encountered a brief 2 min 404 error message as a result of a current deployment we have done. All systems are now operating successfully. Thank you أعزاؤنا التجار واجهت بعض متاجرنا رسالة خطأ (404) لفترة قصيرة مدتها دقيقتين نتيجة لعملية التحديث الأخيرة التي قمنا بها جميع الأنظمة تعمل الآن بنجاح. شكرًا لك

Partial outage for some stores from 5:24:03 on 6th April 2024 till 5:31:06 the same day. Total partial downtime 7 minutes and 3 seconds.

**What happened?** We suffered from a similar issue as the day before. This issue has been a hardware issue that affected our auto-scaling \(done when traffic increases\). ‌ **How long it lasted for?** 7 minutes and 3 seconds ‌ **What have we done to ensure it does not happen again?** We have rebuild the hardware and re-build the cluster in which our gateway runs. This will ensure that there is a brand new platform supporting it.

Some stores were not opening and showing error messages shown to their customers from 9:43:57 pm to 9:57:59 PM. The issue caused the platform to be unstable and it was related to a our gateway cluster used to secure the traffic and cache the stores to ensure faster response to end consumers.

**What happened?**  A set of improvements have been undertaken on the platform in the last weeks to absorb the high volume of traffic generated by the stores associated with the campaigns period of Ramadan. A subset of these changes has been done on one of the critical components use to accelerate the speed of showing the website to end consumers. This component is called Caddy. This is used for 2 main reasons, to control unwanted traffic to the platform and filter it and also cache the stores so that people that are browsing the same store can be shown with the different pages faster \(an acceleration of around 200% compared to not using this component\).  At 9:43:57 pm on 4th April 2024 Caddy scaled up failed. This issue caused intermittently some stores to show errors to end consumers not allowing them to continue browsing and forcing them to refresh.  **How long it latest for?**  6 minutes and 2 seconds At 9:57:59 pm on 4th April 2024 all pods went back to work effectively and the issue was resolved.  **Did it affect orders?**  * Ratio of created orders in the next 15 min after the issue happened is higher by 10 or 15 orders more than average per minute \(before the issue occurring\). This suggest that all orders were not lost considering the traffic after the issue and the orders after the issue. **How are we preventing from this happening again?**  * Review proactively the Caddy configuration to ensure we don’t have bigger issues like the one reported in the future and ensure that all issues are resolved in the Caddy gateway  * We have added additional monitoring however this was a hardware failure that we continuing to investigate with hosting supplier in cloud.

أعزاؤنا التجار واجهت متاجرنا بطء ملحوظ. تم حل المشكلة ونحن نراقب الوضع لضمان استقرار المنصة. مع خالص التحيات Dear Merchants, Our stores have experienced noticeable slowdowns. The issue has been resolved, and we are monitoring the situation to ensure the stability of the platform. Sincerely,

أعزاؤنا التجار نحن نواجه حاليًا بطء في عملية الدفع عبر متاجرنا بسبب حجم كبير غير متوقع من الزيارات. يرجى الثقة بأن فريقنا يعمل بنشاط على حل هذه المشكلة في أقرب وقت ممكن. إلى جانب ذلك، يرجى ملاحظة أن منصتنا بشكل عام لا تزال تعمل بكامل طاقتها. نحن ملتزمون بضمان تجربة سلسة في أماكن أخرى على موقعنا ونركز على حل تأخير عملية الدفع. سنوفر تحديثات بشكل دوري. مع خالص التحيات Dears, We are currently experiencing a slowdown in the checkout process across our stores due to an unexpectedly high volume of traffic. Please be assured that our team is actively working to resolve this issue as quickly as possible. Apart from this, please note that our overall platform remains fully operational. We are dedicated to ensuring a seamless experience elsewhere on our site and are focused on resolving the checkout delay. We will provide regular updates. Best regards

أعزاؤنا التجار لقد تمكنا من تحديد سبب البطء في عملية الدفع عبر متاجرنا. فريقنا يعمل الآن بجد لحل هذه المشكلة في أسرع وقت ممكن. نقدر صبركم بينما نقوم بالتعديلات اللازمة Dears, We have identified the cause of the slowdown in the checkout process across our stores. Our team is now working diligently to resolve this issue as swiftly as possible. We appreciate your patience as we make the necessary adjustments.

أعزاؤنا التجار منصتنا تعمل تحت ظروف مرورية طبيعية وبسرعة طبيعية الان . قد نواجه بعض التأخيرالطفيف في إضافة المنتجات إلى السلة في حال زيادة الزيارات، ونحن نراقب الوضع لضمان استقرار المنصة. نعتبر المشكلة قد تم حلها، الفريق سيواصل العمل ومراقبة المنصة. Dears, Our platform is operating under normal traffic and normal speed. At times we might have some delays on add to cart if the volume of traffic increase and we are monitoring to ensure the stability of the platform. We consider the issue resolve but rest assure that the team will continue to work and monitor the platform. Best regards

Dear all, Our stores and associated services experienced multiple instances of complete downtime, starting at 2:57:21 PM and lasting until 3:12:05 PM, totaling 9 minutes and 54 seconds. Our team promptly addressed the issue and successfully resolved it. We are now operating smoothly تجارنا الأعزاء تعرضت متاجرنا والخدمات المرتبطة بها لعدة حالات من تعطل كامل، بدءًا من الساعة 2:57:21 مساءً وحتى الساعة 3:12:05 مساءً، بمجموع 9 دقائق و 54 ثانية. فريقنا استجاب للمشكلة على الفور ونجح في حلها بشكل صحيح. نعمل الآن بسلاسة. نقدر تفهمكم الكريم

We are currently investigating this issue.

We are continuing to investigate this issue.

The issue has been identified and a fix is being implemented.

This incident has been resolved.

تجارنا الأعزاء تعرضت متاجرنا لبطء شديد غير متوقع لمدة دقيقة واحدة، وانقطعت الخدمة عن بعض المتاجر من الساعة 7:20 مساءً حتى الساعة 7:21 مساءً، قام فريق الدعم لدينا بحل المشكلة بسرعة. نقدر تفهمكم الكريم Our stores experienced an unexpected sever slowness/ partial outage and got a timeout from 7:20 pm to 7:21 pm. Our support team promptly resolved the issue. Appreciating your kind understanding