История сбоев Batch

We are currently investigating an issue affecting profile update processing on our CEP platform. Since approximately 09:20 CEST today, profile data (including custom attributes and events) is no longer being updated in real-time. As a result: - Campaigns targeting users based on profile attributes may reach an inaccurate audience. - Trigger automations based on profile attribute changes are currently not firing. Message delivery for ongoing orchestrations (push, email, and SMS) is unaffected, as are device installs and tokens. No data is lost, profile updates will be applied once the issue is resolved and the backlog is processed.

Profile attribute and event updates sent via the Profile API have returned to real-time as of 12:14 CEST. The backlog accumulated since 09:20 CEST has been fully processed. Updates ingested via the mobile SDK are currently running approximately 45 minutes behind. Trigger automations based on profile attribute changes also remain affected, the root cause has been identified and a fix is in progress.

Profile data updates ingested via the mobile SDK are now running approximately 10 minutes behind, down from 45 minutes previously. Data used for campaign selection and attribute-based trigger automations remains approximately 1 hour behind schedule. Our team continues to work on a full resolution. We will provide a further update as soon as more information is available, or within the next hour.

This incident has been resolved. Profile data is now being updated in real-time across all ingestion channels. Campaign selections and attribute-based trigger automations are back to normal. No data was lost during the incident. Profile updates that occurred between June 19, 09:20 CEST and the resolution time have been fully processed.

Between May 8 at 01:00 CEST and 7:14 CEST, a subset of CEP recurring automations failed to execute as scheduled. Affected automations were not sent during this window and were not retried. The issue has been resolved. The service has been operating normally since 7:14 CEST.

We are investigating an issue affecting push notification delivery via the [MEP] Transactional API. Notifications submitted since 14:20 CEST may not be delivered. Our team is working to identify the cause.

We have identified the cause of the issue. A recent configuration change caused push notifications submitted via the Transactional API to not reach our delivery pipeline. A fix is being deployed. Once it is in place, affected notifications will be delivered and no data will be lost.

A fix has been deployed and the backlog of delayed notifications has been processed. New notifications are being sent as expected. We are monitoring the results.

All delayed notifications have been processed, and new notifications are now being delivered as expected. The incident is resolved, and all services are operating normally.

Between May 8 at 05:00 CEST and 11:15 CEST, a subset of CEP recurring automations failed to execute as scheduled. Affected automations were not sent during this window and were not retried. The issue has been resolved. The service has been operating normally since 11:30 CEST.

Our profile data indexation pipeline experienced instability starting at May 7, 02:00 CEST, causing it to fall behind on processing. As a result, profile data used for CEP one-time and recurring campaign targeting was delayed by up to 2 hours 30 minutes for a subset of traffic. The issue has been resolved. Processing has resumed and is recovering. We are monitoring the situation closely.

Profile data indexation has returned to normal. One-time and recurring CEP campaigns are now using up-to-date profile data for targeting and personalization. No data has been lost during this incident.

Between May 4, 2026 at 00:00 CEST and May 4, 2026 at 09:30 CEST, approximately 25% of requests serving the Web SDK were failing. This prevented the SDK from loading on some web pages, and as a result, some events may not have been tracked during this period. Push notifications were not affected and were delivered as expected throughout the incident.

We are currently experiencing an issue with our email composer provider that is impacting the email editor. Since 10:01 CEST, initializing the email composer may fail or be significantly delayed. The issue has been escalated to our third-party provider, and we are awaiting a fix on their side.

A fix has been deployed by our email composer provider and the email editor is initializing normally again. We are monitoring the situation to ensure stability before resolving the incident.

This incident has been resolved. Between 10:01 CEST and 10:30 CEST, our email composer provider experienced degraded performance that prevented or delayed the initialization of the email editor. A fix was deployed on the provider side and service is back to normal.

Since April 21 at 15:28 CEST, the Email Composer is unavailable due to an incident on our third-party provider's side (Stripo). Users are unable to access or edit email messages in the composer at this time. The issue has been escalated to Stripo, and we are actively monitoring their progress toward a fix. We will share further updates as soon as more information becomes available.

This incident has been resolved. Between 15:28 CEST and 15:57 CEST, our email composer provider experienced degraded performance on their infrastructure, which impacted plugin initialization in the email editor. A fix was deployed on the provider side and service is back to normal.

We are experiencing an infrastructure issue impacting parts of our platform. Campaign sending on MEP and the MEP Audiences API have been unavailable since 22:45 CET. CEP exports (recurring and API) have been experiencing slowdowns on some jobs since 23:10 CET. Other platform features are operating normally. Our team is actively working on restoring service.

We have identified the root cause of this incident. A failure on a subset of our infrastructure servers is preventing MEP campaigns from being sent and the MEP Audiences API from responding. CEP exports are also experiencing slowdowns on some jobs as a secondary effect. Our team is actively working on restoring the affected services.

MEP campaign delivery has been restored. Campaigns planned since 22:45 CET are currently being sent, and the MEP Audiences API is back to normal. We are monitoring the platform to ensure stability. CEP export slowdowns (recurring and API) remain under investigation.

This incident has been resolved. The exports have been back up since 09:45 CET, pending requests are still being processed.

We are investigating delays affecting CEP campaigns (one shot & recurring). As of writing, processing is approximately 1 hour behind schedule. In-app messages and automations are not affected. Campaigns will be sent to the expected users as the backlog clears.

CEP campaigns (one-shot and recurring automations) are back to normal. The backlog was fully processed at 16:25 CET, and campaigns sent as of now work as expected. No data has been lost.

The service has been restored and has been under monitoring for 48 hours with no regressions observed.

We are investigating a performance degradation within our profile processing pipeline. This issue specifically affects updates sent via our mobile SDKs. As of 21/02/2026 at 17:00 CET, processing times have increased, with current delays of up to 3 hours. This latency may result in delayed execution for triggered automations.

We have identified the issue and deployed a fix. The processing backlog is now beginning to clear, and delivery delays are steadily decreasing. We are monitoring the pipeline closely to ensure performance returns to nominal levels.

The processing backlog has been steadily decreasing over the last hour. We continue to monitor the situation closely until all updates are processed and real-time performance is restored.

This incident has been resolved.

We are currently experiencing an issue with our email composer provider that prevents image uploads from functioning correctly. The issue has been escalated to the provider, and we are awaiting a fix on their side. We will share further updates as soon as more information becomes available.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

We are investigating delays affecting CEP campaigns (one shot & recurring). As of writing, processing is approximately 3 hours behind schedule. In-app messages and automations are not affected. Campaigns will be sent to the expected users as the backlog clears.

We are still investigating the problem. CEP campaigns are slowly being sent but still approximately 3 hours behind schedule. Some trigger automations are delayed by 30 minutes. No data has been lost, messages will be sent to the expected users as the backlog clears. We will provide a status update at 10:00 CET.

We are still investigating the problem. CEP campaigns are still slowly being sent and approximately 2 hours behind schedule. Some trigger automations coming from SDK (mobile & web) events are delayed by 1:30 hours. No data has been lost, messages will be sent to the expected users as the backlog clears. We will provide a status update at 11:00 CET.

CEP campaigns (one-shot and recurring automations) are back to normal. The backlog was fully processed at 11:00 CET, and campaigns sent as of now work as expected. Trigger automations based on mobile & web SDK events are processing but remain delayed by ~2 hours. No data has been lost, messages will be sent to the expected users as the backlog clears.

Trigger automations based on mobile SDK events are now live. The ones based on webpush SDK events are currently delayed by 1:30h. We're continuing to monitor the situation and expect a full resolution by 14:00 CET

Trigger automations on webpush SDK are live since 13:00 CET. The incident is now resolved.

Since 06:24 CET, CEP Trigger automations have been running late. We identified the cause and deployed a fix. Deliveries may be delayed by up to 90 minutes while the backlog processes. We are monitoring recovery.

The backlog has been processed, the trigger automations are now live.

Summary: Between 15 January at 18:00 CET and 15 January at 19:00 CET, a small subset of CEP trigger automations was delayed for up to an hour. Only trigger automations using timers and with timers scheduled to fire between 18:00 and 19:00 CET were affected; this represents less than 2% of all trigger automations during this time period. Impact for the affected trigger automations: * Timer firing was delayed by up to one hour. * As a consequence, the delivery of emails, SMS, and push notifications was delayed by up to one hour. Root Cause: some timers were not accepted by our database system due to an edge case in their data, which blocked the entire timer scheduling pipeline. Resolution: a fix was deployed on 15 January at 18:55 CET, which resolved the issue. By 19:00 CET, the backlog was cleared.

Summary: Between 14 January at 21:30 CET and 15 January at 09:15 CET, a small subset of CEP trigger automations was delayed for up to 11 hours. This subset accounted for less than 2% of all CEP trigger automations. Impact for the affected trigger automations: * Event processing was delayed by up to 11 hours. * As a consequence, the delivery of emails, SMS, and push notifications was delayed by up to 11 hours. Root Cause: one instance of our event processing pipeline was unable to process certain events due to an edge case in their data, which slowed down the processing speed for that specific instance. Resolution: a mitigation was applied on 15 January at 08:36 CET, which restored normal processing performance. By 09:10 CET, the backlog was cleared and processing returned to real-time.

Trigger automations ran late between 07:00 and 08:40 CET. All messages were delivered; none were lost.

An incident affected CEP campaign delivery across Email, Push, and SMS between 11:00 and 14:30 CET. * Campaigns: Messages scheduled during this window were not sent and not retried. Analytics will show zero delivery and engagement. If impacted, consider resending your campaign. * Automations: Delays occurred between 11:40 and 12:15 CET. Affected automations were eventually sent, and the displayed analytics are accurate. The issue has been fixed, and we are actively monitoring the platform.

This incident has been resolved.

## Post-mortem: Partial Platform Outage ### Summary On December 22, 2025, between **10:17 and 13:40 UTC**, part of our CEP experienced service disruptions affecting message delivery, data ingestion, and dashboards for some customers. The incident was caused by a **hardware-related issue on a high-grade server** hosting part of our private virtualization infrastructure. The issue has since been fully resolved, and corrective actions have been implemented. ### Impact During the incident window, the following impacts were observed: * **Push and Email Campaigns \(CEP\):** Delivery was interrupted for a subset of campaigns. * **Data Ingestion & Analytics:** Temporary ingestion delays occurred. * **Dashboards:** The Batch dashboard was intermittently unavailable. No impact was observed on SMS delivery. The incident affected a limited subset of customers, depending on their usage at the time. ### Timeline \(UTC\) * **10:17** – First service degradation detected. * **10:20** – Investigation initiated. * **11:00** – Infrastructure issue identified on one hypervisor. * **11:30** – Mitigation and recovery actions started. * **13:40** – All impacted services fully operational. ### Root Cause The incident was caused by a **fault in the cooling system** of a high-grade server used in our private virtualization infrastructure. This cooling issue led to **overheating**, triggering a **protective shutdown of a disk group** on the affected server. As a result, the hypervisor abruptly lost access to multiple disks, causing all virtualized services hosted on that node to stop simultaneously. Although this class of hardware is designed to provide strong reliability guarantees, this cooling failure resulted in the loss of a single hypervisor and exposed the impact of service colocation on shared infrastructure. ### Resolution * The cooling issue and impacted hardware were fully repaired and validated by our infrastructure provider. * Affected services were restarted and resynchronized. * Data consistency was verified after recovery. ### Corrective and Preventive Actions Following this incident, we took the following actions: * The faulty hardware was repaired and removed from service until fully validated. * We reviewed service placement on our private virtualization infrastructure and **reduced the colocation of critical components** on single hypervisors. * Stateful services \(including Redis clusters\) were redistributed to **limit the blast radius** of a single host failure. * We **strengthened monitoring and alerting around service colocation**, allowing us to detect and act earlier when multiple critical components are unintentionally placed on the same underlying host. These actions aim to reduce the impact of similar infrastructure-level incidents in the future. ### Conclusion We apologize for the disruption this incident caused. While hardware failures of this nature are rare, this event highlighted areas where we could further improve infrastructure resilience and observability. We remain committed to transparency and continuous improvement. — **The Batch Engineering Team**

Between November 22nd at 10:30 CET and 12:08 CET, a performance issue on a database cluster impacted the delivery of Push notifications, SMS, and emails. Impact: * Event processing: Delayed by up to 1 hour and 30 minutes for a subset of users. * Campaign processing: Delayed by up to 2 hours for a subset of campaigns. * Delivery failures: A subset of campaign tasks timed out during processing and were not delivered. Resolution: A mitigation was applied at 12:08 CET, which restored normal processing performance. The resulting backlog was fully cleared by 12:25 CET.

Between November 14th at 10:17 (CET) and November 14th at 10:31 (CET), the Profile API and Catalog API experienced an incident during which HTTP 500 errors were returned on all requests, impacting 100% of the traffic during this period. Those requests have not been processed and can be safely retried.