37 Deliverect incidents · luty 2021 — official updates, affected components, duration and resolution details.
Między 13: 13 a 13: 42 UTC w dniu 26 września niektórzy klienci doświadczyli błędów lub opóźnień w przetwarzaniu zamówień. Nasz zespół zidentyfikował problem i przywrócił normalną obsługę o 13: 42 UTC. Przeglądamy ten incydent i podejmujemy kroki, aby zapobiec ponownemu wystąpieniu.
Automatyczne tłumaczenie oficjalnej aktualizacji incydentu.
Między 16: 55 a 17: 14 UTC w dniu 15 września niektórzy klienci doświadczyli błędów lub opóźnień w przetwarzaniu zamówień. Nasz zespół zidentyfikował problem i przywrócił normalną obsługę o 17: 14 UTC. Przeglądamy ten incydent i podejmujemy kroki, aby zapobiec ponownemu wystąpieniu.
Automatyczne tłumaczenie oficjalnej aktualizacji incydentu.
We are currently experiencing an issue where menu pushes are failing. Our team is aware and actively investigating the root cause. We will provide updates as more information becomes available.
We are continuing to investigate this issue.
The issue has been identified and a fix is being implemented.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We are currently investigating a major, global outage reported by our cloud provider, Cloudflare. This outage is affecting their network infrastructure, which is severely impact the availability, performance, and accessibility of some of our products and services.
A fix has been implemented by Cloudflare and our systems are recovering, we are monitoring closely.
The incident has been resolved and our services and products are fully operational again.
We noticed a service interruption impacting our Retail Order Processing and related front-end UIs in several regions. The root cause was identified as a major global outage affecting our critical third-party provider, Cloudflare. The incident started at roughly 1:00 PM UTC. Importantly, our restaurant order processing channels were unaffected and continued to function normally throughout this period. Our engineering teams were immediately focused on developing and implementing a service workaround. As of 2:21 PM UTC, all Retail Order Processing services and front-end UIs were fully functional and stable. We apologize for the disruption caused by this external event.
We are currently experiencing a service outage due to an incident with our cloud provider. This disruption is impacting multiple systems, and our team is actively working to assess the situation and restore services as quickly as possible
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We are currently investigating an issue within our API that is causing delays in order processing. Our engineering team is actively working to identify the root cause and resolve the issue as quickly as possible.
A fix has been implemented and we are monitoring the results.
This incident has been resolved
Some of our customers experienced temporary delays in order processing due to an issue with a new feature. This feature, designed to improve how we handle old data, unintentionally created extra work for our system during a busy period. While our team resolved the issue, we understand how important it is for your operations to run smoothly. To prevent this from happening again, we’ve made several improvements, including better monitoring, stricter testing for new features, and adjustments to how our system handles errors. We sincerely apologize for any inconvenience this may have caused and thank you for your patience as we continue to make Deliverect more reliable for your business.
Deliverect experienced degraded performance between 20:08 UTC and 20:53 UTC. Most customers should not have been impacted, but a small number may have experienced dropped orders, delayed orders, or other issues with Deliverect functionalities. The issue has been resolved and we are working to prevent future occurrences.
We are currently investigating the issue.
A fix has been implemented and we are monitoring the results.
The overall degradation in API response times has been solved. The fix we implemented has fixed the issue.
Start Time: April 30, 2024, 06:41 AM CEST (04:41 AM UTC) End Time: April 30, 2024, 06:48 AM CEST (04:48 AM UTC) We are backfilling an incident that occurred earlier this morning where the Deliverect API experienced degraded performance for a short period of time (approximatively 7 minutes). This resulted in slower response times and potential delayed order processing. Deliverect API is now functioning normally and we apologize for any inconvenience this may have caused.
Customers might have experienced delay on order processing from 18:12 to 18:36 UTC+0 The incident is now resolved and we are back to normal order processing activities.
Between 00:18 and 00:27 UTC+1, our system degraded and performance slowly declined. The incident was caused by a third-party provider experiencing issues. We sincerely apologize for the inconvenience.
We are currently investigating this issue.
We experienced order processing delays, but we can confirm that now all processing activities are back to normal. As a way to improve order processing performance, the menu-push feature has been paused until we have further information about its impact on the order flow.
We've identified the cause of the incident and applied a fix for it. We'll keep monitoring.
This incident has been resolved.
We are currently investigating this issue.
The issue has been identified and a fix is being implemented.
A fix has been implemented and we are monitoring the results
We are continuing to monitor for any further issues.
This incident has been resolved.
Our monitoring system detected a sudden traffic decrease on one of our systems, which was impacting order processing. Requests are not correctly arriving to our systems we are actively investigating and setting all efforts to quickly resolve it.
We are continuing to investigate this issue. The impact was that our queueing systems was struggling to process orders to the POS systems, and restarting them only partially helped recovery.
After a thorough investigation the issue has been identified and a fix is being implemented.
A fix has been deployed and we are monitoring the results
Operations were restored completely for affected customers. We are currently actively monitoring our systems.
Starting at 10:28 (UTC+0), we faced potential delays over a period of 45min. All orders were eventually processed and the incident has been addressed and fixed.
We are monitoring the fix that was applied, and are implementing changes to avoid this from occurring in the future.
We are continuing to monitor for any further issues.
This incident has been resolved.
Starting May 21, 19:00 CET time we have been experiencing degraded performance with our API, which is impacting the speed and responsiveness of our frontend.
You may notice that our Frontend is slower than usual at certain times, when it's the case our API takes more time than usual to respond to requests. This can lead to some orders being rejected or duplicated. Our team is actively working on resolving the problem and restoring normal operations as quickly as possible.
A fix has been implemented and we are monitoring the results.
We are continuing to monitor for any further issues.
This incident has been resolved
Customers and partners might have experienced slowness on the Deliverect Frontend and non-critical API calls between 8:00 PM CET and 9:45 PM CET. A fix has been implemented and the incident is now resolved.
An issue with a TLS certificate impacted customers using the Tunnel but did not affect customers using DMA. We experienced system instability for around 50 minutes. The issue is related to the expiry of a TLS certificate, which is normally automatically renewed by our processes. Our Infrastructure and Support teams were notified of the incident and immediately began investigating the cause. The automation responsible for renewing this specific certificate was identified as the root cause of the incident. The team worked to replace manually the expired certificate with a new one so customers can use the tunnel again without disruption. The code responsible for the certificate renewal is now being reviewed to avoid this bug from happening again.
We’ve noticed some order updates did not reach the DMA. This issue was noticed between 15:19:45 (UTC+0) and 18:40:00 (UTC+0). We’ve identified the issue and applied a fix to it. We will keep monitoring.
