История сбоев Dixa

1. 23 октября 2026 г. в 10:05 UTCresolved

   23 октября API Dev был недоступен с 11:15 до 11:31 CEST (16 минут). Отключение было вызвано ошибкой на этапе развертывания одного из наших компонентов. Вскрытие будет опубликовано в течение 5 рабочих дней.

2. 27 октября 2026 г. в 07:37 UTCpostmortem

   ##Резюме 23 октября 2026 года, в 11:15 утра по восточному времени, все запросы на Dixa Public API (dev.dixa.io) начались с ошибкой. Отключение было вызвано неправильным развертыванием компонента, который аутентифицирует входящие запросы API. Наша команда обнаружила проблему в 11:16 утра по восточному времени, определила причину и развернула исправление. Полное обслуживание было восстановлено в 11:31 утра по восточному времени. # # Влияние Между 11:15 и 11:31 AM CEST общедоступный API Dixa был недоступен для всех клиентов. Каждый запрос API в этом окне, во всех версиях API, получил ошибку HTTP 500. Запросы были отклонены до того, как они достигли наших систем, поэтому ни один запрос не был частично обработан, и никакие данные не были потеряны или изменены. Интеграции, которые автоматически возвращают неудавшиеся запросы, будут восстановлены сами по себе. Интеграциям, которые не повторяются, возможно, потребуется повторно отправлять любые звонки, сделанные во время этого окна. ##Коренная причина Отключение было вызвано ручным развертыванием компонента, который аутентифицирует каждый запрос в Dixa Public API. Новая версия этого компонента была упакована неправильно, поэтому запустить ее не удалось. Без рабочего этапа аутентификации каждый входящий запрос API не удался. Сам код был правильным; проблема заключалась только в том, как он был упакован для развертывания. ## Резолюция Наша команда обнаружила проблему в 11:16 утра по восточному времени и начала расследование пострадавших служб. К 11:26 утра по восточному времени первопричина была прослежена до неправильного развертывания. Правильно упакованная версия того же кода была развернута в 11:30 утра по восточному времени, и запросы API снова стали успешными. Показатели ошибок вернулись к нормальному уровню к 11:32 утра по восточному времени. ##Превентивные меры Чтобы предотвратить подобные инциденты, мы: Замена ручного развертывания этого компонента автоматизированным конвейером развертывания • Проверка каждого изменения в условиях производства до того, как оно достигнет производственного уровня Улучшение версий релизов, чтобы мы могли немедленно вернуться к известной хорошей версии Добавление дополнительных автоматических оповещений об ошибках аутентификации API и скоростях ошибок для более быстрого обнаружения проблем ## Закрыть записку Мы понимаем, насколько важен надежный доступ к Dixa Public API для интеграции и рабочих процессов. Мы приносим извинения за любые неудобства, причиненные в течение этого периода, и стремимся к укреплению устойчивости этой службы.

Автоматический перевод официального обновления инцидента.

1. 28 июля 2026 г. в 10:47 UTCinvestigating

   Мы испытываем некоторые трудности с Advanced Insight & Miuros. Мы расследуем этот вопрос. Обновление в 13:30 CEST

2. 28 июля 2026 г. в 11:15 UTCresolved

   Все известные проблемы с этим инцидентом решены. Мы благодарим вас за терпение и сотрудничество.

3. 5 августа 2026 г. в 06:45 UTCpostmortem

   ###**Пост-Мортем: Продвинутое понимание и ухудшение производительности Miuros** ** Дата инцидента: ** 28 июля 2026 г. ** Продолжительность:** 12:47 PM - 1:15 PM CEST \(28 минут\) **Жесткость: ** Средняя — ухудшение производительности, проблемы с частичным доступом #### ** Резюме** 28 июля 2026 года в 12:47 вечера CEST пользователи начали испытывать ухудшение производительности и периодические трудности с доступом к Advanced Insight & Miuros. Наша команда определила проблему и начала расследование вскоре после ее обнаружения. Реализовано исправление, а полное обслуживание подтверждено восстановленным к 1:15 ПМ КЕСТ. ####**Влияние** В период с 12:47 до 1:15 вечера некоторые пользователи испытывали медленное время загрузки или не могли надежно получить доступ к Advanced Insight & Miuros. Потеря данных не произошла, и доступ был полностью восстановлен после развертывания исправления. ####*Корневая причина** Пониженная производительность была прослежена до изменения, введенного во время недавнего развертывания в среде Advanced Insight & Miuros, которое оказало непреднамеренное влияние на производительность обслуживания. После пометки наша команда инженеров исследовала пораженные компоненты и внедрила корректирующее исправление для восстановления нормальной производительности. ####**Резолюция** Обнаружив проблему в 12:47 вечера по восточному времени, наша команда начала расследование пострадавших служб. Корневая причина была прослежена до недавнего развертывания, и исправление было развернуто, разрешая ухудшенную производительность. Полная функциональность была подтверждена в 1:15 PM CEST. ####*Превентивные меры** Чтобы предотвратить подобные инциденты, мы: Усиление предварительной проверки развертывания и тестирования производительности для Advanced Insight & Miuros Улучшение пороговых значений мониторинга и оповещения для обнаружения ухудшения производительности ранее * Совершенствование процесса отката для сокращения времени на решение аналогичных проблем ####*Закрыть записку** Мы понимаем, насколько важен надежный доступ к Advanced Insight & Miuros для ваших рабочих процессов. Мы приносим извинения за любые неудобства, причиненные в течение этого периода, и стремимся к укреплению устойчивости этой службы.

Автоматический перевод официального обновления инцидента.

1. 28 июля 2026 г. в 08:18 UTCresolved

   В 8:53 по восточному времени 28/07/2026 наши услуги по обработке конверстаций и входу в Dixa не были полностью работоспособны. Наши инженеры были немедленно уведомлены о проблеме и начали работу по выявлению первопричины. В 9:03 AM CEST было развернуто решение этой проблемы, и пострадавшие службы снова полностью функционировали. Мы провели анализ первопричин этого инцидента и внедряем решения для предотвращения подобных случаев в будущем. Мы искренне извиняемся за любые неудобства, которые это могло вызвать.

2. 5 августа 2026 г. в 06:43 UTCpostmortem

   ### **Post-Mortem: Conversation Handling & Login Service Disruption** **Date of Incident:** July 28, 2026 **Duration:** 8:53 AM – 9:03 AM CEST \(10 minutes\) **Severity:** High — partial service disruption #### **Summary** On July 28, 2026, at 8:53 AM CEST, Dixa experienced a service disruption affecting conversation handling and user login functionality. The issue was identified quickly by our engineering team, a fix was deployed at 9:03 AM CEST, and full functionality was restored within 10 minutes of the initial impact. #### **Impact** Users may have experienced difficulty logging into Dixa or handling conversations during the affected window. No data loss occurred, and all systems returned to normal operation by 9:03 AM CEST. #### **Root Cause** The disruption was traced to a change introduced during a routine deployment to our production environment, which had an unintended effect on the services responsible for conversation handling and authentication. Our monitoring systems flagged the anomaly shortly after it occurred, allowing our engineering team to identify and roll back the change quickly. #### **Resolution** Engineers on-call were alerted immediately once the anomaly was detected. The team traced the issue to the recent deployment and deployed a corrective fix at 9:03 AM CEST, which resolved the disruption. #### **Preventive Measures** To reduce the likelihood of similar incidents going forward, we are: * Strengthening our pre-deployment validation checks to catch this class of issue before it reaches production * Improving automated rollback triggers so similar anomalies are reverted even faster * Enhancing monitoring coverage on conversation handling and authentication services specifically #### **Closing Note** We know reliability is critical to how you run your business, and we take incidents like this seriously. We apologize for any inconvenience this may have caused and remain committed to continuously improving the resilience of our platform.

Автоматический перевод официального обновления инцидента.

1. 29 июня 2026 г. в 09:04 UTCinvestigating

   В настоящее время мы расследуем проблему, затрагивающую доступ к Advanced Insights в Dixa. Некоторые пользователи могут быть не в состоянии получить доступ к этой функции в это время. Наша команда активно изучает первопричину и предоставит обновление, как только будет доступна дополнительная информация. Приносим извинения за неудобства.

2. 29 июня 2026 г. в 09:08 UTCinvestigating

   Мы продолжаем расследование этого вопроса.

3. 29 июня 2026 г. в 09:30 UTCmonitoring

   Мы определили проблему, затрагивающую доступ к Advanced Insights, и внедрили исправление. В настоящее время мы следим за ситуацией, чтобы подтвердить полную резолюцию. Обратите внимание, что данные за последние 3 часа могут отсутствовать. Мы работаем над восстановлением этого и предоставим дальнейшее обновление, как только проблема будет полностью решена. Приносим извинения за неудобства.

4. 29 июня 2026 г. в 12:30 UTCmonitoring

   Мы все еще постепенно перераспределяем данные, поэтому данные могут быть несвежими с сегодняшнего утра для некоторых клиентов. Мы обновим эту страницу, когда все будет обновлено.

5. 29 июня 2026 г. в 15:11 UTCresolved

   Все данные теперь передислоцированы, и мы полностью обновлены для всех.

Автоматический перевод официального обновления инцидента.

1. 23 апреля 2026 г. в 11:42 UTCmonitoring

   We're currently aware of an increased delay with data showing in the Conversation view and Real Time dashboard. Conversation offers are fully operational, but you might see conversations on the dashboard that have been offered already and/or outdated information on the Conversations view. We'll notify here when data ingestion has caught up.

2. 23 апреля 2026 г. в 11:51 UTCresolved

   Data ingestion has fully caught up and the Real-Time Dashboard, Search and Conversations pages are showing live data again.

1. 8 апреля 2026 г. в 16:32 UTCinvestigating

   We've received reports of Dixa not loading. We're investigating and will come back to you with an update as soon as we know more. Next update at 16:45 UTC (18:45 CEST)

2. 8 апреля 2026 г. в 16:37 UTCinvestigating

   We are continuing to investigate this issue.

3. 8 апреля 2026 г. в 16:50 UTCinvestigating

   We're still trying to find the cause. We'll update you again in 15 minutes, or sooner if we have identified the problem.

4. 8 апреля 2026 г. в 17:00 UTCidentified

   We've identified the cause of the issue, and are actively working towards a solution. While we do so, we also confirmed that this only impacts connections towards the agent interface. Other parts of Dixa are not impacted by this partial outage. We'll report back at 17:15 UTC (19:15 CEST), or earlier as soon as we have more information to share.

5. 8 апреля 2026 г. в 17:12 UTCmonitoring

   We are happy to inform, that our teams have deployed a fix for the issue. We are seeing agents successfully reconnect to agent interface. We will continue to monitor the results. Next update at 17:30 UTC (19:30 CEST)

6. 8 апреля 2026 г. в 17:24 UTCresolved

   This incident has now been resolved. Our sincere apologies for the disruption. We identified the cause of the issue being a configuration issue on a new entry point for agent connections towards agent interface that was introduced earlier today. More information will be shared in the post mortem, which will be posted here within 5 business days. If you have questions about today's outage, feel free to reach out to [email protected]. We'll be happy to help.

7. 10 апреля 2026 г. в 12:21 UTCpostmortem

   ## **Summary** On April 8, 2026, Dixa experienced a partial outage lasting approximately 58 minutes. New WebSocket connections were unable to be established, which meant that new logins failed, and any agents who refreshed their browser or lost their connection could not reconnect. Agents who remained on an existing session were unaffected during the incident. The root cause was a TLS certificate misconfiguration introduced during a planned migration of our ingress controller infrastructure. The issue was identified, fixed, and fully resolved within the hour. ## **Impact** Between 18:26 and 19:24 CEST, customers attempting to log in to Dixa or re-establish a WebSocket connection \(e.g., after a page reload\) were unable to do so. Browsers rejected the connection due to an invalid TLS certificate being served. Agents who were already logged in with an active WebSocket session continued to operate normally throughout the incident. The impact was limited to new or reconnecting sessions. A small number of customers were affected and reported the issue to our support team. No conversations or data have been lost during the incident. ## **Timeline \(CEST\)** * 18:26 - Internal reports that Dixa is not loading for some users * 18:30 - Issue escalated to engineering via our critical support channel * 18:32 - Status page updated to **Investigating** * 18:40 - Engineering identifies a TLS certificate error \(`ERR_CERT_AUTHORITY_INVALID`\) * 18:52 - Root cause identified, a self-signed default certificate was being served instead of the correct one * 19:00 - Status page updated to **Identified** * 19:11 - Fix deployed; WebSocket connections begin recovering * 19:12 - Status page updated to **Monitoring** * 19:24 - Full recovery confirmed; status page updated to **Resolved** ## **Root Cause** As part of our ongoing WebSocket resilience work to ensure a more stable platform, we migrated our ingress controller \(the component that routes incoming traffic to internal services\) from an end-of-support solution to a new one. This migration was tested in our staging environment before being applied to production. However, there was a configuration discrepancy between staging and production for the ingress class that handles WebSocket traffic. When DNS was switched to the new ingress controller on the morning of April 8, existing connections continued to work through cached DNS entries still pointing to the old controller. Hours later, as DNS caches expired across the internet, clients began resolving to the new controller, which, due to the misconfiguration, did not recognize the WebSocket routes. This caused it to serve a default self-signed TLS certificate instead of the valid one, leading browsers to reject the connection. The length of the partial outage per customer varies depending on when the DNS cache expired, and if WebSocket connections started to connect to the new load balancer. ## **Resolution** Once the root cause was identified, a configuration update was deployed to the new ingress controller so that it could correctly handle WebSocket traffic. Connections began recovering immediately after the fix was applied. ## **Preventive Measures** We have taken the following steps to reduce the likelihood and impact of similar issues in the future: * **Environment parity:** All non-production environments have been aligned with production configuration conventions, eliminating the discrepancy that caused this incident. * **Endpoint monitoring:** We have added external monitoring checks that validate both the availability and TLS certificate validity of our WebSocket endpoints. This will enable faster detection if a similar issue occurs. * **WebSocket isolation**: We will isolate the platform's WebSocket requirement to be optional, so if a similar issue should happen in the future, the disruption will be less intrusive for users. ‌ ‌ We sincerely apologize for the disruption this caused. Reliability is a top priority for us, and we are committed to learning from every incident to make Dixa more resilient. If you have any questions, please don't hesitate to reach out to your account team or our support at [[email protected]](mailto:[email protected]).

1. 18 марта 2026 г. в 15:02 UTCinvestigating

   We are experiencing some difficulties with the AiCoPilot feature "Smart Replies". We are investigating the matter. Next update at 3:30 PM UTC

2. 18 марта 2026 г. в 15:02 UTCinvestigating

   We are continuing to investigate this issue.

3. 18 марта 2026 г. в 15:48 UTCinvestigating

   We are further investigating the matter. Thank you for your patience.

4. 18 марта 2026 г. в 17:17 UTCinvestigating

   Our engineering team is actively investigating the root cause. This might take some time to fully resolve and we will continue to provide updates here on daily basis. We apologize for any inconvenience caused and thank you for your patience.

5. 18 марта 2026 г. в 19:52 UTCresolved

   All known issues to this incident have been resolved. We thank you for your patience and cooperation. Post mortem about this incident will be posted within 5 business days.

6. 26 марта 2026 г. в 08:54 UTCpostmortem

   _Dixa experienced issues with the Smart Reply feature in AI CoPilot on 18 March 2026._ _**Summary**_ _18 March 2026 at 14:37 CET - 20:20 CET, customers experienced issues with the Smart Reply feature in AI CoPilot. Agents were unable to use Smart Reply._ _**Root cause**_ _A code change introduced a regression that caused agents' browsers to block completing the request to load Smart Reply suggestions._ _**Timeline**_ _At 14:37 CET on 18 March 2026: Issue flagged internally after several customers reported the issue_ _At 15:33 CET: Engineering investigates and attempts an initial rollback, which does not resolve the issue._ _At 16:02 CET: Status page updated to "Investigating" — customers notified of difficulties with AiCoPilot Smart Replies._ _At 16:48 CET: Status page updated — engineering continues to investigate._ _At 18:17 CET: Status page updated — root cause investigation ongoing_ _At 20:20 CET: The knowledge-backend service is successfully rolled back to a stable version, restoring Smart Reply._ _At 20:52 CET: Status page updated to "Resolved" — all known issues confirmed resolved._ _**Solution**_ _The immediate solution was to roll back to a stable version from prior to the breaking change, restoring Smart Reply for all affected customers._ _Long term, the team will review the offending commit before re-deploying to prevent similar regressions from reaching production undetected._ _We sincerely apologise for the inconvenience caused by this issue._

1. 11 марта 2026 г. в 08:34 UTCinvestigating

   We are receiving reports of slowness in the agent interface. We are investigating the issue. Next update at 09:50 CET

2. 11 марта 2026 г. в 08:51 UTCinvestigating

   We have received reports of instability in the platform. We are investigating the issue. Updates will follow

3. 11 марта 2026 г. в 08:54 UTCmonitoring

   Our teams has now identified the issue. We are working on a fix, that will resolve the issue. We will provide more information soon. Next update at 10:10 CET

4. 11 марта 2026 г. в 09:14 UTCmonitoring

   We are continuing to monitor system metrics and are observing a recovering trend. Next update: 10:30 CET

5. 11 марта 2026 г. в 09:31 UTCmonitoring

   We continue to observe improvements. We can confirm there was no data loss. Please note that as a side effect, some conversations may have been routed to the default queue. We apologize for any inconvenience caused. Next update: 11:00 CET

6. 11 марта 2026 г. в 09:59 UTCidentified

   We have identified some additional disruptions in the service. Our team is actively working on a fix. Next update: 11:30hs

7. 11 марта 2026 г. в 10:35 UTCidentified

   We continue to actively work on resolving this incident with the highest priority. Our team remains fully engaged and further updates will follow as our investigation progresses. Next update: 12:00 hs

8. 11 марта 2026 г. в 10:54 UTCmonitoring

   A fix has been deployed and we are observing improvements across system metrics. Our team will continue to actively monitor the situation until full recovery is confirmed. Next update: 12:30 CET

9. 11 марта 2026 г. в 11:31 UTCresolved

   All known issues linked to this incident have been fixed and full recovery of the platform has been confirmed. We thank you for your patience and cooperation. A postmortem will be published within 5 business days.

10. 13 марта 2026 г. в 12:22 UTCpostmortem

    **Summary:** On March 11, 2026, Dixa experienced platform-wide degraded performance lasting approximately 3 hours \(08:30 - 12:35 CET\). Customers experienced slow or failed conversation loading, timeouts on email sending, conversation transfers, assignments, and flow processing. No data was lost, and there were no security issues at any point. **Impact:** * _Availability_: Platform-wide slowness and partial inaccessibility for ~3 hours. * _Affected functionality_: Conversation loading, email sending, conversation transfers, conversation assignments, and flow processing - all experienced significant slowness and intermittent failures. * _Data integrity:_ All emails were fully processed after the fix. No data was lost, and no security issues occurred at any point. **Root Cause:** The incident was caused by an atypical traffic pattern in inbound email processing that resulted in repeated internal retries - retries are a normal part of email distribution, accounting for factors such as sending delays and server availability, but this expanded exponentially. The sustained retry volume placed excessive load on a central platform component, causing cascading timeouts across dependent services and resulting in platform-wide degradation. **Timeline \(CET\):** * Mar 11, 06:00 - First signs of email processing errors detected * Mar 11, 08:30 - Platform degradation begins; customer impact starts * Mar 11, 10:50 - First mitigation deployed; partial improvement * Mar 11, 12:30 - Root cause fully identified; final fix applied * Mar 11, 12:35 - Platform stability confirmed **Resolution:** We identified and addressed the source of the abnormal email volume, which resulted in an immediate reduction in error rates, and the platform to recover. **What We Have Done Since This Incident:** Following this incident, we have already implemented the following improvements: 1. Added validation to reject invalid email addresses early in the pipeline, preventing them from entering retry loops. 2. Optimised internal lookups to fetch only necessary data instead of the full conversation history, significantly reducing load during email processing. 3. Added deduplication logic to prevent redundant data fetches during email processing. 4. Enforced concurrency limits: platform components now shed excess traffic when saturated, allowing requests to be redistributed rather than queued indefinitely. 5. Added deadline checking: expired requests are now discarded immediately instead of consuming resources on work that is no longer needed. 6. Reduced internal timeout thresholds to fail fast under contention rather than blocking for extended periods. **What We're Continuing to Work On:** 1. Loop detection and interruption - Introduce mechanisms to detect and automatically halt email processing anomalies before they can accumulate significant load. 2. Improved alerting and escalation - Ensure processing anomalies are detected and escalated with appropriate urgency. **Closing Note:** We sincerely apologize for the disruption this caused. These improvements are our highest priority. If you have any questions, please reach out to [[email protected]](mailto:[email protected]).

1. 10 марта 2026 г. в 12:38 UTCinvestigating

   We are currently investigating an issue affecting the Help Center (Dixa Knowledge). We will provide a further update by 14:00 CET.

2. 10 марта 2026 г. в 12:57 UTCidentified

   Our teams has now identified the issue. We are working on a fix, that will resolve the issue. We will provide more information soon. Next update at 14:10 CET

3. 10 марта 2026 г. в 13:13 UTCmonitoring

   A fix has been deployed and we are seeing significant improvements to the Help Center (Dixa Knowledge). Some images and custom styling may still not display correctly yet. Our team is actively working to resolve this. Next update: 14:30 CET.

4. 10 марта 2026 г. в 13:48 UTCmonitoring

   We are continuing to monitor the situation. While the Help Center (Dixa Knowledge) has largely recovered, some custom CSS styling issues remain. We will provide a further update shortly

5. 10 марта 2026 г. в 17:03 UTCresolved

   We can confirm that the issue has been resolved. A postmortem report will be published within 5 business days.

6. 16 марта 2026 г. в 17:02 UTCpostmortem

   **Summary**: On March 10, 2026, all public Dixa Knowledge Help Centers were unavailable following a production deployment performed during scheduled maintenance. **Impact** * Dixa Knowledge Public Help Centers remained inaccessible for 6 hours \(approximately\) * Customers with advanced custom CSS styling were contacted directly with instructions on how to update their selectors to prevent recurrence. **Root Cause:** After the production deployment performed during a scheduled maintenance, all public Help Centers became unavailable. Visitors saw an error page \(500\) instead of help content. **Timeline \(CET\)** Mar 10, 08:00 - Maintenance completed. No anomalies detected. Mar 10, 11:52 - Initial customer reports received. Considered isolated cases at the time. Mar 10, 13:38 - Incident process triggered. Reported on Status page. Mar 10, 13:57 - Root cause fully identified. Mar 10, 14:13 - Fix deployed. Public Help Centers restored; some styling issues persisted. Mar 10, 14:48 - Root cause for styling anomalies identified as linked to build-time generated class names. Affected customers notified with instructions. Mar 10, 18:03 - Incident resolved. **Resolution**: We identified a misconfiguration in the deployment that caused Help Centers to attempt to load from a test environment rather than production. A fix was deployed, and an immediate reduction in error rates confirmed it was effective. **What We Have Done Since This Incident:** Following this incident, we have already implemented the following improvements: 1. Proactively contacted all affected customers with clear instructions on how to update their custom CSS styling. 2. Documented additional validations for future migrations and action plans to address similar errors. 3. External documentation updated to offer alternatives to the use of auto-generated classes for styling Help Centers. **What We're Continuing to Work On** * Improve monitoring and alerting, ensure availability issues are detected and escalated promptly, independent of customer reports. * Investigating a solution to expose stable, named elements for Help Center customization, reducing dependency on build-generated class names. **Closing Note:** We sincerely apologize for the disruption this caused. We appreciate your patience throughout this disruption. If you have any questions, please reach out to [email protected].

1. 2 марта 2026 г. в 18:08 UTCinvestigating

   We have received reports of instability in the platform. We are investigating the issue. Updates will follow

2. 2 марта 2026 г. в 18:16 UTCinvestigating

   We've received reports of slowness and problems with responsiveness across Dixa's agent interface. We're investigating.

3. 2 марта 2026 г. в 18:45 UTCinvestigating

   We are continuing to investigate this issue.

4. 2 марта 2026 г. в 19:06 UTCinvestigating

   We've taken a few measures trying to improve stability while we investigate the cause of the issues. You may notice slight improvement, but we haven't pinned down the root cause yet.

5. 2 марта 2026 г. в 19:18 UTCidentified

   We've identified the cause of the issues and are taking measures to resolve the problems you're experiencing as soon as possible. Our sincere apologies for the inconvenience caused.

6. 2 марта 2026 г. в 20:09 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

7. 2 марта 2026 г. в 20:21 UTCresolved

   The incident has been resolved. During the incident, inbound emails may have had trouble processing. You may therefore experience some blank, queue-less emails that didn't properly go through a flow, followed by an inbound email that did go through your email flow and does have the correct message. You can safely close these conversations or merge them into the correctly processed inbound email. We sincerely apologize for the inconvenience and encourage you to contact [email protected] if you have further questions. A Post Mortem will be posted within 5 business days.

8. 4 марта 2026 г. в 15:21 UTCpostmortem

   **Summary** On March 2, 2026, Dixa experienced platform-wide degraded performance lasting approximately 3 hours and 45 minutes. Customers experienced slow or failed conversation loading, timeouts on email sending, conversation transfers, assignments, and flow processing. No data was lost, and there were no security issues at any point. **Impact** * **Availability:** Platform-wide slowness and partial inaccessibility for ~3 hours 45 minutes. * **Affected functionality:** Inbound email processing, conversation loading, conversation transfers, conversation assignments, and flow processing — all experienced significant slowness and intermittent failures. * **Side effect:** Some inbound emails resulted in empty, queue-less conversations being created. These are safe to close or merge with the correctly processed follow-up conversation. * **Data integrity:** All emails were fully processed after the fix. No data was lost and no security issues occurred at any point. **Root Cause** The incident was caused by a significant and atypical spike in inbound conversations that fell well outside expected operational parameters. This unexpected volume put pressure on a central database component, which began throttling requests. The throttling then cascaded to other parts of the system, resulting in platform-wide slowness and temporary inaccessibility. **Timeline \(CET\)** Mar 1, 02:14 Significant and atypical spike in inbound conversations begins Mar 2, ~17:45 Database throttling begins; connection pools saturate Mar 2, 19:08 Incident declared; investigation begins Mar 2, 20:06Mitigation measures applied; investigation ongoing Mar 2, 20:18 Root cause identified Mar 2, 21:09 Fix deployed; error rates drop Mar 2, 21:17 Resolution confirmed Mar 2, 21:21Incident resolved **Resolution** We identified and addressed the source of the abnormal conversation volume, which immediately caused error rates to drop and the platform to recover. **What We're Doing to Prevent Recurrence** We have identified several systemic improvements and are actively working on them: 1. **Detect and suppress atypical inbound volume patterns** — Introduce early filtering to prevent abnormal spikes from reaching core platform components. 2. **Improve retry and backoff behaviour** — Reduce the risk of compounding load during high-traffic failure scenarios. 3. **Reduce inter-service dependencies** — Limit the blast radius of a single overloaded component affecting other parts of the platform. 4. **Improve database resilience** — Add circuit breakers and timeouts to prevent database pressure from cascading across services. 5. **Atomic conversation creation** — Ensure conversations are never persisted without their initial message, eliminating orphaned empty conversations as a failure side effect. **Closing Note** We sincerely apologize for the disruption this caused. We take platform reliability seriously and are committed to the systemic improvements outlined above. If you have any questions, please reach out to [[email protected]](mailto:[email protected]).

1. 20 февраля 2026 г. в 14:41 UTCinvestigating

   We have received reports of instability in the platform. We are investigating the issue. Updates will follow

2. 20 февраля 2026 г. в 14:49 UTCresolved

   All known issues related to this incident have been resolved. We experienced a brief incident that caused slowness and intermittent timeouts across the platform. During this period, some customers may have experienced degraded performance when working in the interface. The issue was identified and resolved quickly, and the platform is now operating normally. We apologize for the inconvenience and appreciate your patience. A post-mortem about this incident will be posted within 5 business days.

3. 25 февраля 2026 г. в 07:05 UTCpostmortem

   **Incident Summary** On Feb 20, 2026 - 15:24 CET, we experienced a brief period of degraded performance across the platform. Some customers may have noticed slower response times and intermittent timeouts while using the interface. **Impact** During the incident window, a subset of requests were delayed or failed, which may have affected normal usage for some customers. **Root Cause** The issue was caused by an internal service experiencing elevated load, which temporarily impacted request processing across the platform. **Resolution** Our engineering team identified the issue quickly and restored normal performance. The platform has been operating normally since the fix was applied. **Prevention** We are reviewing monitoring and scaling controls for the affected service to reduce the likelihood of similar incidents in the future.

1. 27 января 2026 г. в 11:41 UTCinvestigating

   We are experiencing some difficulties with AI Voice transcripts. We are investigating the matter. Next update at 13:00 CET

2. 27 января 2026 г. в 12:10 UTCidentified

   We have identified the issue as related to one of our providers. We continue to monitor the situation closely. Next update: 14:00 CET

3. 27 января 2026 г. в 12:36 UTCresolved

   All known issues to this incident have been resolved by our partner. We thank you for your patience and cooperation. Post mortem about this incident will be posted within 5 business days.

4. 2 февраля 2026 г. в 12:37 UTCpostmortem

   Dixa experienced issues with transcription on Tuesday 27th January 2026 between 11:27 and 13:15 CET **Timeline** * At 10:22 CET / 4:22 EST Azure reports issues on the Azure OpenAI Service * At 11:27 CET / 5:27 EST the first issues with transcriptions start to happen at Dixa. At 11:46 CET / 5:46 EST Dixa investigates unrelated issues with live chats and calls. * At 12:16 CET / 6:16 EST While investigating the unrelated issues, Dixa noticed the issues with transcriptions. * At 13:15 CET / 7:15 EST Transcription issues disappear. **Impact** Transcriptions were not available or delayed in the specified timeframe. **Root cause** Our provider for AI voice transcriptions, Azure, reported issues. According to their [status](https://azure.status.microsoft/en-us/status/history/): _Between 09:22 UTC and 16:12 UTC on 27 January 2026, and again between 11:14 UTC and 13:35 UTC on 29 January 2026, a platform issue resulted in an impact to the Azure OpenAI Service in the Sweden Central region. Impacted customers may have experienced HTTP 500/503 errors, failed inference requests, and/or issues with model deployment metadata. This issue also affected the Agent Service and other downstream AI Services dependent on Azure OpenAI in this region._ **Immediate solution** We monitored the situation until it was fixed. **Long-term solution \(where applicable\)** We could consider changing region for our service if the issues become more frequent. Otherwise, Azure has been reliable enough to keep the current infrastructure. Furthermore, Azure made improvements on their side to prevent this situation from happening again.

1. 6 января 2026 г. в 15:24 UTCinvestigating

   We are experiencing some difficulties with searching for contacts when making outbound calls. We are investigating the matter. In the short term, we recommend pasting phone numbers in directly.

2. 6 января 2026 г. в 15:32 UTCidentified

   Our teams has now identified the issue. We are working on a fix, that will resolve the issue.

3. 6 января 2026 г. в 15:35 UTCmonitoring

   We are happy to inform, that our teams have deployed a fix for the issue. We will continue to monitor the results.

4. 6 января 2026 г. в 15:40 UTCresolved

   All known issues to this incident have been resolved. We thank you for your patience and cooperation.

1. 22 декабря 2025 г. в 10:30 UTCinvestigating

   We have received reports of instability in the platform. We are investigating the issue. Updates will follow

2. 22 декабря 2025 г. в 10:33 UTCinvestigating

   We are continuing to investigate this issue.

3. 22 декабря 2025 г. в 11:00 UTCidentified

   We have identified the root cause of the issue and are working actively on implementing the fix. We will keep updating status page with the progress.

4. 22 декабря 2025 г. в 11:26 UTCidentified

   We are still working on a full resolution to this issue. Additionally, we currently experience a slight delay in certain webhooks events which will fully caught up once the issue is resolved.

5. 22 декабря 2025 г. в 11:54 UTCidentified

   The issue with AI Voice transcripts is now fully resolved. All missing transcripts were now created and the new ones are created as expected. We are still working on an issue with delays of some webhook events.

6. 22 декабря 2025 г. в 12:47 UTCidentified

   We are still experiencing delays in some webhook events, we will update the status page once this issue is resolved. We truly apologise for the inconvenience.

7. 22 декабря 2025 г. в 12:59 UTCmonitoring

   All issues related to this incident have been fixed. Webhooks events are being processed as expected. We will monitor the situation before we move the status to Resolved.

8. 22 декабря 2025 г. в 13:41 UTCresolved

   This incident has been resolved. We will publish postmortem within five business days. We are truly sorry for the inconvenience.

9. 13 января 2026 г. в 11:48 UTCpostmortem

   # **Degraded performance - AI Voice Transcription and webhooks** ## **Summary** On 22 December 2025, Dixa experienced an issue with the AI Voice Transcription feature. Transcriptions were not being generated for phone calls. Once the issue was resolved, a temporary period of degraded performance on Webhooks occurred as the system processed a backlog of pending transcriptions. ## **Root cause** Following a scheduled release, a component responsible for processing voice transcription requests stopped functioning correctly. When the issue was resolved, the system began processing all pending transcription requests. The high volume of these requests temporarily affected Webhook delivery times, causing delays in notifications to third-party integrations. ## **Timeline** **Monday 22 December at 11:30 CET:** Issue reported and investigation started. **Monday 22 December at 12:00 CET:** The issue was identified and a fix was deployed. AI Voice Transcription began processing again. **Monday 22 December at 12:26 CET:** Degraded Webhook performance identified, affecting some third-party integrations, including chatbots. **Monday 22 December at 12:54 CET:** AI Voice Transcription fully restored. **Monday 22 December at 14:35 CET:** All services fully recovered. Incident closed. ## **Preventive measures** Dixa will implement the following measures to prevent similar issues in the future: * Enhanced monitoring for the AI voice transcription feature. * Improved handling of backlog processing to minimise impact on dependent services. We sincerely apologise for the inconvenience this has caused.

1. 16 декабря 2025 г. в 19:49 UTCinvestigating

   We are observing challenges with loading Co-pilot features. We are working on identifying the root case and will update the status page as soon as the issue is identified.

2. 16 декабря 2025 г. в 19:55 UTCinvestigating

   We are continuing to investigate this issue.

3. 16 декабря 2025 г. в 20:03 UTCidentified

   We have identified the root cause and applying the fix.

4. 16 декабря 2025 г. в 20:12 UTCmonitoring

   We deployed a fix and can see that the issue is resolved. We will monitor the situation for a some time now until we fully close the incident.

5. 16 декабря 2025 г. в 20:34 UTCresolved

   This incident has been resolved. We will publish postmortem within 5 business days.

6. 23 декабря 2025 г. в 15:44 UTCpostmortem

   **Summary** On the 16th of December 2025 between 8:49 PM CET and 9:34 PM CET we experienced intermittent issues with AI features such as Co-pilot and intent detection automation on some of the Dixa instances. **Root cause** The incident was caused by the increased usage of the above mentioned features which caused hitting the set usage limit. **Timeline** At 8:49 PM CET: We started observing usage spikes which resulted in errors that started occurring around AI related features. At 8:55 PM CET: Status page was updated with the areas that were impacted by the issue. At 9:03 PM CET: We found the root cause and moved the incident status to Identified. At 9:12 PM CET: The fix was deployed and the incident status changed to Monitoring. At 9:34 PM CET: After a period of monitoring we did not observed any more errors and the incident status was moved to Resolved. **Solution** The immediate solution was to increase the usage limits as well as implement the separation of the limits per each AI feature in the backend. Long term we are planning to implement proactive alerting and monitoring around AI features usage and set limits to prevent the errors that occurred when the usage spiked. ‌ We sincerely apologise for the inconvenience caused by the above issue.

1. 11 декабря 2025 г. в 06:52 UTCresolved

   Summary Parts of Dixa’s search functionality malfunctioned during a 30-minute window, following a deployment including some deficient code, affecting, amongst others, the search feature within the Email and Phone composer features, respectively. Root Cause Starting at 2.28 PM (CET) on 8 Dec, the faulty application started serving requests, some of which it was unable to handle. This was due to an insufficiently tested code path that got invoked under certain feature flag configurations, resulting in the search request not being handled correctly. Action Items On-call engineers were alerted about elevated error levels within 15 minutes of the first failing requests, and shortly after, were able to identify the impaired deployment, rolling it back to a previous version while investigations continued. At 3.07 PM (CET), error rates related to failing search requests had completely levelled off. A fix to the root issue has been released since, further improving the overall safety and handling of code dictated by feature flag configurations. This includes more thorough runtime parameter checks as well as adding several regression tests to ensure the issue won’t be reintroduced.

1. 18 ноября 2025 г. в 11:59 UTCinvestigating

   We are experiencing some difficulties with Dixa Knowledge an AI Co-Pilot functionality. We are investigating the matter. Next update at 12.15 PM UTC

2. 18 ноября 2025 г. в 12:06 UTCidentified

   We've identified that the problems are caused by an outage at a third party, Cloudflare. Dixa only uses Cloudflare for Dixa Knowledge and is therefore largely unaffected. However, some of our sub-processors may also use Cloudflare for some or all of their services. Currently, we've identified that our AI-powered features are not, or only partially, working. We will keep you informed of any changes in the situation. Next update in half an hour.

3. 18 ноября 2025 г. в 12:31 UTCidentified

   We are seeing signs of improvement, and services are slowly recovering. Dixa Knowledge help centers appear to be intermittently available again. The outage is related to an ongoing issue with a third-party provider (Cloudflare). We will continue to monitor the situation and provide updates as full service is restored. We will update you again in 30 minutes.

4. 18 ноября 2025 г. в 12:44 UTCidentified

   We've identified that the problems are caused by an outage at a third party, Cloudflare. Dixa only uses Cloudflare for Dixa Knowledge and is therefore largely unaffected. However, some of our sub-processors may also use Cloudflare for some or all of their services. Currently, we've identified that our AI-powered features are not, or only partially, working. Dixa Knowledge Help Centers may also be throwing intermittent errors. We will keep you informed of any changes in the situation.

5. 18 ноября 2025 г. в 13:02 UTCidentified

   We are continuing to monitor service disruptions affecting our AI-related features due to ongoing issues with Cloudflare infrastructure. We have observed intermittent improvements in service availability and Dixa Knowledge KBs appear to be mostly working as of right now, though KB visitors may still experience intermittent issues. Cloudflare provides critical internet infrastructure for thousands of services worldwide. When they experience issues, it may have a serious impact on services that depend on their infrastructure. Our team is closely monitoring the situation alongside both Cloudflare and our AI subprocessor as they work toward resolution. We apologize for any inconvenience and will continue to provide updates as the situation develops.

6. 18 ноября 2025 г. в 13:49 UTCidentified

   Our AI sub-processor reports seeing a small number of requests being processed successfully with the majority still failing while Cloudflare continues to work on restoring their services. We continue to monitor both our AI provider as well as Cloudflare for any changes, and will update you here when there is more information to share. Thank you for your patience and our apologies once again for the disruptions you may be experiencing today.

7. 18 ноября 2025 г. в 14:48 UTCmonitoring

   The issues at Cloudflare appear to have been fully resolved. As a result, all related impact on Dixa services has also been mitigated. Dixa Knowledge and AI-powered features such as Co-Pilot, Mim, Intent detection and smart replies are now operating normally again. We will continue to monitor performance closely, but no further disruption is expected. Our sincere apologies for any inconvenience this may have caused.

8. 18 ноября 2025 г. в 15:43 UTCresolved

   Although Cloudflare is still reporting problems, we are confident that the issues impacting Dixa services have been resolved. Dixa Knowledge Bases and AI-powered features have been fully operational as of 15:45 CET (14:45 UTC). A post-mortem will be posted within 5 days.

9. 20 ноября 2025 г. в 08:20 UTCpostmortem

   **Summary** On the 18th of November 2025 at 12:55 PM CET we received reports of Dixa Knowledge help centers not being available due to a Cloudflare error. At roughly the same time, we were made aware internally that we were receiving a lot of errors on AI-related functionality from one of our upstream sub-processors handling AI requests.‌ **Root cause** The incident originated from a major Cloudflare service disruption \([https://www.cloudflarestatus.com/incidents/8gmgl950y3h7\)](https://www.cloudflarestatus.com/incidents/8gmgl950y3h7)) caused by a configuration file used for Cloudflare’s Bot Management system. The file was auto-generated and due to a bug became too large, subsequently crashing Cloudflare services and causing major disruption across the globe. Dixa is using Cloudflare solely for the purpose of serving Dixa Knowledge's Knowledge Bases. Cloudflare helps us handle verification of custom host names and generating certificates for these custom host names, as well as protect the knowledge bases from malicious third parties. We quickly learned that one of our upstream sub-processors for AI-related features also uses Cloudflare for similar reasons, therefore also breaking AI Co-Pilot, Mim, Intent Detection and other AI-powered features inside Dixa. **Timeline** At 12:55 pm CET: We started observing increased errors on services handling AI-related features inside Dixa and received the first report about Dixa Knowledge KBs being inaccessible. We triggered our internal incident process and communicated here at 12:59 PM CET. At 1:06 pm CET: The full impact on Dixa services started to become clear, and we were confident only AI-related features and Knowledge Bases were affected. For the next hour and a half, we monitored the situation closely and saw alternating service recovery and disruption on the before-mentioned services while Cloudflare was recovering. At 2:49 pm CET: We were made aware by our AI sub-processor that they saw slight improvements in connectivity. At 3:46 pm CET: The impacted systems have fully recovered. The incident status was moved to Monitoring. At 5.43 pm CET: We changed the incident on our status page from 'Monitoring' to 'Resolved' as no further problems have been reported nor detected. At 6:44 pm CET: Cloudflare confirmed that the incident is fully resolved on their end. ‌ We sincerely apologize for the inconvenience this has caused.

1. 20 октября 2025 г. в 16:55 UTCidentified

   We continue seeing interruptions with the telephony service. Inbound and outbound traffic on some numbers might be affected. The issue is caused by the problem with Amazon AWS services (https://health.aws.amazon.com/health/status) and services affected by it like Twilio (https://status.twilio.com/). We will keep monitoring status of the issues and adding updates to the status page. We will publish the next update in 60 minutes.

2. 20 октября 2025 г. в 17:52 UTCidentified

   Some customers might still experience intermittent issues with inbound and outbound calls as an upstream incident causing these issues is not fully resolved. We will keep updating the status of this issue within the next 60 minutes.

3. 20 октября 2025 г. в 18:51 UTCidentified

   The telephony inbound and outbound traffic looks stable at the moment, however the main AWS incident causing this issue is still not fully resolved therefore temporary and intermittent issues might be expected. We will post the next update within 60 minutes.

4. 20 октября 2025 г. в 19:46 UTCmonitoring

   The services at AWS and Twilio that caused the telephony inbound and outbound intermittent issues have been mostly recovered. We do not see signs of issues on our service. We are moving the incident to Monitoring status.

5. 20 октября 2025 г. в 22:56 UTCresolved

   The issue is now fully resolved.

6. 28 октября 2025 г. в 15:46 UTCpostmortem

   **Summary**  On the 20th of October 2025 at 6:55 PM CEST due to the ongoing AWS incident Dixa experienced intermittent issues with Telephony Inbound and Outbound connectivity. On some Dixa instances the calls were not being placed or the connection between agents and users was not established.  ‌ **Root cause**  The incident originated from a **major AWS service disruption in the us-east-1 region** caused by a **DNS race condition within Amazon DynamoDB’s endpoint management system**. The fault temporarily removed valid DNS records for several AWS services, resulting in widespread API resolution failures across multiple dependent systems. Dixa is using Twilio’s APIs for **telephony services.** When DynamoDB’s DNS records were invalidated, Twilio’s **regional load balancers lost access to key internal routing data**, which caused inbound and outbound voice requests to fail.  Because Twilio’s APIs are globally routed through Twilio’s us-east-1 infrastructure, Dixa’s platform became unable to establish or maintain voice sessions.  ‌ **Timeline** At 06:55 pm CEST: We started observing intermittent interruptions with Telephony Inbound and Outbound service. Only some Dixa instances and phone numbers were affected. The incident was reported with the Identified status.  At 07:52 pm CEST: The update is posted stating that some of the services are still impacted.  At 08:51 pm CEST: There are no signs of issues on Dixa services anymore, however the third party services are still recovering and minimal interruptions could be expected.  At 09:46 pm CEST: The impacted systems have fully recovered. The incident status was moved to Monitoring.  At 00:56 am CEST on the 21st of October: AWS confirmed that the incident is fully resolved on their end. Dixa’s incident status was changed to Resolved.  ‌ **Preventive measures**  Dixa will evaluate implementing the following measures to mitigate the risk of similar issues in the future:  * evaluate the feasibility of enabling outbound and inbound routing via multi-regional providers in case the us-east-1 Twilio region fails. * Independent external health check monitoring outside of AWS and Twilio. * Service dependency mapping and redundancy revisit. * Incident communication redundancy  ‌ We sincerely apologise for the inconvenience this has caused.

1. 20 октября 2025 г. в 07:16 UTCinvestigating

   We have received reports of instability in the platform. We are investigating the issue. Updates will follow

2. 20 октября 2025 г. в 08:07 UTCidentified

   We are currently experiencing significant issues with our telephony channel, which is impacting our ability to provide telephony services to our customers. Both inbound and outbound traffic are being affected, meaning that you will encounter difficulties when trying to make or receive calls. Additionally, our SMS and Whatsapp channels may also experience disruptions as a result of this ongoing issue, which could hinder your ability to communicate via those platforms as well. The root cause of these problems is related to an incident with our telephony services provider, which is completely outside of our control. We want to assure you that we are actively collaborating with them to address and resolve the issue at the earliest possible opportunity. Our team is working diligently to ensure that normal service is restored as quickly as possible. We believe that the current issues are likely tied to a larger incident that a major cloud service provider is experiencing in one of its regions. This situation has unfortunately impacted multiple services, including ours, complicating the process further. We sincerely apologize for any inconvenience this may cause you and appreciate your patience and understanding during this time. Our priority is to keep you updated as we work through this situation, and we are committed to restoring service as swiftly as possible.

3. 20 октября 2025 г. в 08:20 UTCidentified

   We are currently experiencing significant issues with our telephony channel, which is impacting our ability to provide telephony services to our customers. Both inbound and outbound traffic are being affected, meaning that you will encounter difficulties when trying to make or receive calls. Additionally, our SMS and Whatsapp channels may also experience disruptions as a result of this ongoing issue, which could hinder your ability to communicate via those platforms as well. The root cause of these problems is related to an incident with our telephony services provider, which is completely outside of our control. We want to assure you that we are actively collaborating with them to address and resolve the issue at the earliest possible opportunity. Our team is working diligently to ensure that normal service is restored as quickly as possible. We believe that the current issues are likely tied to a larger incident that a major cloud service provider is experiencing in one of its regions. This situation has unfortunately impacted multiple services, including ours, complicating the process further. We sincerely apologize for any inconvenience this may cause you and appreciate your patience and understanding during this time. Our priority is to keep you updated as we work through this situation, and we are committed to restoring service as swiftly as possible

4. 20 октября 2025 г. в 08:55 UTCidentified

   The issues are still continuing and telephony is still down. We're following the issue which is related to an outage for our telephony network supplier Twilio (https://status.twilio.com/). This in turn is caused by an Incident at AWS (https://health.aws.amazon.com/health/status). As a consequence, other services dependent on AWS might experience downtime, such as Elevio and JIRA. We'll keep monitoring and will keep you posted once we get an update. Next update is at latest in one hour

5. 20 октября 2025 г. в 09:52 UTCidentified

   We can see that log in issues to Elevio and Jira (including the Jira integration) are now resolved, however we are still experiencing issues with the telephony channel due to the outage at Twilio (https://status.twilio.com/). We will keep updating the status of this incident, in one hour at the latest.

6. 20 октября 2025 г. в 10:38 UTCidentified

   Telephony traffic is slowly recovering. The inbound calls are entering the Dixa platform and agents can accept offers but can't connect to the call. Outbound calls are still affected. We continue to follow the status of this issue with Twilio and we will keep posting updates. We will post the next update in 30 minutes.

7. 20 октября 2025 г. в 11:05 UTCidentified

   We continue to see connection issues with phone calls. Calls are getting through to the Dixa platform but when agents accept the call, agents are not being connected to customers due to Twilio's network services which are still experiencing issues. SMS is fully functional again. We continue to monitor the situation and will update you again in the next 30 minutes, or sooner if there is news to share.

8. 20 октября 2025 г. в 11:26 UTCidentified

   We are happy to report inbound and outbound calls are operational again. If agents continue to experience problems, ask them to refresh the page before trying again. We continue to monitor the situation up close, and if anything changes we will inform you via this page. If you continue to experience issues, feel free to reach out to Dixa support. Our sincere apologies for the inconvenience you are experiencing today. Next update in 60 minutes, or earlier if there is an update to share.

9. 20 октября 2025 г. в 12:21 UTCmonitoring

   We're happy to report that all Whatsapp issues have been resolved, and messages sent out during the outage have caught up and have been sent to their receivers. We also got confirmation that telephony is working as expected. Again our sincere apologies for the inconvenience this has caused. If you still experience issues, please reach out to Dixa Support. We will continue to monitor if anything changes, but don't expect any further impact.

10. 20 октября 2025 г. в 13:19 UTCresolved

    All known issues reported as part of this incident have been fully resolved. We have been monitoring the system and all services are operational. We truly apologise for the inconvenience this incident cause and thank you for your patience while this was being solved. Post mortem about this incident will be posted within 5 business days.

11. 28 октября 2025 г. в 14:03 UTCpostmortem

    **Summary**  On the 20th of October 2025, starting from 08:58 am CEST, Dixa experienced widespread disruptions caused by the issues coming from third-party services.  **Impacted services** * Telephony Inbound and Outbound traffic - Calls were not being placed for the most part of the incident, and for a period of time, calls were placed, but the connection between agents and end users was not established.  * WhatsApp channel - conversations were queued but not sent during the incident. All queued messages were sent as soon as the issue was resolved. * SMS channel - conversations were queued but not sent during the incident. All queued messages were sent as soon as the issue was resolved. * Login to Elevio - login to Elevio was impacted. Help Centers were fully operational.  * Login to the Status page was impacted, which caused challenges with communicating incident updates once the incident was raised.  * Access to Dixa’s Public API documentation[ docs.dixa.io](http://docs.dixa.io) was also affected by the incident; Dixa’s API was fully operational. **Root cause**  The incident originated from a **major AWS service disruption in the us-east-1 region** caused by a **DNS race condition within Amazon DynamoDB’s endpoint management system**. The fault temporarily removed valid DNS records for several AWS services, resulting in widespread API resolution failures across multiple dependent systems. Dixa is using Twilio’s APIs for **telephony, WhatsApp, and SMS channels**. When DynamoDB’s DNS records were invalidated, Twilio’s **regional load balancers lost access to key internal routing data**, which caused inbound and outbound voice requests to fail. This also impacted dependent APIs for SMS and WhatsApp delivery. Because Twilio’s APIs are globally routed through Twilio’s us-east-1 infrastructure, Dixa’s platform became unable to establish or maintain voice sessions, send or receive SMS messages, or process WhatsApp traffic. Other parts of the Dixa platform \(login, chat, email\) remained operational, but all Twilio-dependent communication channels were impacted for the most part of the duration of the AWS event.  ### **Timeline** At 08:58 am CEST: We started observing errors in the logs coming from 3rd party systems, and shortly after, we started getting reports from customers about issues. At 09:16 am CEST: The Incident was reported on the status page with status: Investigating. The system we use for the status page was also impacted due to the same root cause at AWS, and we were unable to add further details regarding the incident. Between 9:00 and 10:00 am CEST: We kept updating the users who reached out to our support team about the status of the issue, while adding more details to the status page was not possible. At 10:00 am CEST: An updated incident message was posted in the Agent interface to notify users about the status of the incident  At 10:07 am CEST: We regained access to the status page and updated the status of the incident to: Identified and added the information about impacted services.  At 10:20 am CEST: The incident status update is posted as well, with the notification being sent to all status page subscribers.  At 10:55 am CEST: The next status page update is posted with reference to the incidents at AWS and Twilio, which were the root cause of the issues with Dixa services. We also reported issues with logging in to Elevio caused by the same incident.  At 11:52 am CEST: Update is posted. The Elevio login issue is resolved. The issue with telephony inbound and outbound, as well as SMS and WhatsApp service, continues.  At 12:58 pm CEST: The first calls were starting to get to Dixa infrastructure again, but the voice in the calls was still not operational.  At 13:05 pm CEST: The connection issue with telephony continues. SMS is confirmed to be fully operational again.  At 13:26 pm CEST: The inbound and outbound telephony is confirmed to be operational again. At 14:21 pm CEST: WhatsApp is confirmed to be operational as well. All messages that were queued during the incident got successfully delivered. The incident status is changed to: Monitoring At 15:19 pm CEST: All systems have been operational since the incident was moved to Monitoring, and the incident status was changed to: Resolved.  ‌ **Preventive measures**  Dixa will evaluate implementing the following measures to mitigate the risk of similar issues in the future:  * evaluate the feasibility of enabling outbound and inbound routing via multi-regional providers in case the us-east-1 Twilio region fails. * Independent external health check monitoring outside of AWS and Twilio. * Service dependency mapping and redundancy revisit. * Incident communication redundancy  ‌ We sincerely apologise for the inconvenience this has caused.

1. 14 октября 2025 г. в 08:24 UTCinvestigating

   We are experiencing some difficulties with translations for AI Co-pilot. We are investigating the matter.

2. 14 октября 2025 г. в 08:37 UTCidentified

   Our teams has now identified the issue and have deployed a fix. We're seeing progress and are monitoring.

3. 14 октября 2025 г. в 10:25 UTCidentified

   We're still experiencing instabilities relating to AI Co-pilot features and we're working on a fix for it

4. 14 октября 2025 г. в 10:54 UTCmonitoring

   A new fix has been deployed and we're seeing improvements. Please refresh the website and it should be back to normal. We're monitoring the progress

5. 14 октября 2025 г. в 12:41 UTCresolved

   All known issues to this incident have been resolved. We thank you for your patience and cooperation. Post mortem about this incident will be posted within 5 business days.

6. 17 октября 2025 г. в 11:56 UTCpostmortem

   ### Summary Dixa experienced partial degraded performance of AI-Co pilot features on October 14 2025 between 08:13-13:13 CEST. Users who never changed their translation settings were affected, experiencing unwanted auto-translations and non working translations. The issues were cause by an increased number of translation requests hitting our api, leading to overload. We eventually solved it by increasing the capacity for handing requests and by reverting the changes **Timeline** At 08:13 am CEST A change was deployed for AI copilot to default the Auto summary and translations to “Automatic” for users who never changed their preferences At 09:48 am CEST First reports of difficulties with translations for AI Co-pilot that didn’t work at all. At 10:23 am CEST A fix is deployed to support the increased amount of requests incoming. At 10:33 am CEST Fix is evaluated to have mitigated the issues. At 11:33 CEST The requests to our API increase and issues come back, more reports are coming in from customers about conversations being automatically translated or translations not working at all. At ~12:45 am CEST Fix is applied to support even more increased amount of requests to prevent translation failures and the quota on the model used by translation is increased to support the increased amount of requests. At 13:13 am CEST A final fix is deployed - changes to AI copilot settings defaults are reverted and requires agents to reload to take effect. This decreased the system overload. ### **Impact** The incident had a major impact on the users of the AI Co-pilot feature that never changed their settings who experienced degraded performance with conversations not translating or unwanted increased amount of translations performed. ## **Root cause** The frontend changes to settings defaults for AI auto summary and translate flipped the settings from on-demand to automatic. It was applied to existing customers that never changed their settings. It appeared to be a significant amount of customers who had not done any changes to their settings. The change took effect over the course of multiple hours, because it required agents to reload the UI for the settings to take effect. This caused many more translations being performed where customers did not expect. Additionally, this caused increased amount of translation requests via rest-api, which made rest-api incapable of performing all requests due to it having a capped number on max open requests ### **Resolution** * Support for and quota of translation requests were increased * The frontend change to settings defaults for AI Co-pilot were reverted We have taken steps to improve our processes for control and increased the capacity limit so that this won’t happen again. We apologize for the inconvenience this caused and thank you for you cooperation