Kustomer Status

WhatsApp has confirmed that all services are fully recovered. After careful monitoring, our team has determined that all affected areas are now fully restored. Please reach out to Kustomer support at support@kustomer.com if you have additional questions or concerns.

**Summary** On June 9, 2026, customers experienced delays in chat message delivery and related real-time updates for a portion of the morning. The issue was caused by a third-party service event affecting the infrastructure used to propagate real-time messaging updates. Service recovered the same morning, and current system health is stable. **Root cause** A third-party real-time messaging provider experienced elevated latency and intermittent publish failures in one of its regions. That disruption delayed delivery acknowledgements and other real-time updates in Kustomer. Messages continued to be stored successfully, but some updates were not reflected immediately in the client until the external service recovered or the client refreshed. **Timeline** * **9:05 AM EDT** — Customer impact began, including delayed chat message delivery and stale real-time updates. * **9:45 AM EDT** — The third-party provider reported active latency affecting its service. * **9:48 AM EDT** — The provider reported mitigation and recovery. * **Shortly after recovery** — Real-time behavior returned to normal and monitoring confirmed stability. **Lesson/improvements** * We added additional monitoring tied to the third-party provider's public service health signals so similar service events can be identified faster. * We are improving alerting around real-time messaging errors to reduce time to diagnosis. * We are continuing to review operational signals to better distinguish external service issues from internal platform issues.

**Summary** On June 2, 2026, customers in our prod1 environment experienced elevated latency affecting chat creation, email sending, and conversation routing. The event caused delayed processing for a subset of customer interactions during the incident window. Service performance recovered the same day after mitigation steps were applied, and the environment is currently operating normally. **Impact** * Scope: prod1 only * Customer effect: delayed chat creation, delayed email sends, and slower conversation routing * Duration: customer-visible impact began in the morning ET on June 2 and materially improved after mitigation later that afternoon * Other environments: prod2 and prod4 were not impacted by the customer-facing degradation **Timeline** * Early June 2: We detected elevated processing latency in services supporting chat, email, and routing in prod1. * Midday to afternoon ET: Customer reports confirmed delayed system behavior in prod1. * Afternoon ET: We increased service capacity and adjusted resource limits to reduce backlog and restore throughput. * Later that day: We completed an infrastructure rollback on affected worker hosts and confirmed stable recovery. * June 3: Monitoring confirmed the environment remained healthy. **Root cause** The incident was caused by infrastructure-level resource exhaustion on a set of worker hosts in prod1. That reduced the availability of a metadata-dependent service and created message backlog, which in turn increased latency for customer-facing workflows such as chat creation, email sending, and routing. The issue was isolated to prod1. **Resolution** We restored service by increasing available capacity, raising resource limits for the affected service, and rolling impacted worker infrastructure back to a stable configuration. After those changes were applied, backlog cleared and latency returned to expected levels. Current system health is stable. **Preventative actions** * Strengthen host-level capacity and disk safeguards before future infrastructure rollouts * Add earlier alerting for infrastructure resource pressure to reduce time to detection * Expand validation for production-scale logging and resource usage prior to promotion * Continue tuning service capacity thresholds and recovery procedures for metadata-dependent workloads

## Summary On May 26, 2026, some customers experienced failures when sending WhatsApp messages from existing conversations. The issue affected outbound message delivery for a subset of WhatsApp configurations. Service was restored after the responsible change was rolled back, and the issue is now resolved. ## Impact During the incident window, outbound WhatsApp messages failed for some existing conversations. Newly created conversations were less consistently affected, and impact varied by account configuration. Customers may have seen message-send failures or provider errors while attempting to send WhatsApp messages. ## Timeline All times EDT. * **4:05 PM:** The incident window is believed to have begun after a recent WhatsApp-related change. * **5:54 PM:** Active investigation began. * **5:56 PM:** The team identified a likely connection to recent WhatsApp sender-selection behavior and started a rollback. * **6:05 PM:** Rollback completed. * **6:17 PM – 7:47 PM:** The team validated recovery across affected examples and narrowed the underlying cause. * **8:21 PM:** The incident was declared resolved. ## Root cause The incident was caused by an issue in WhatsApp phone-number lookup and sender selection. A recent change expanded support for multiple valid phone-number formats in certain countries. In some cases, that allowed the system to resolve the wrong sender configuration for an existing conversation. When that happened, outbound sends could target an invalid or unavailable sender identity, causing WhatsApp message delivery to fail. ## Resolution We mitigated the incident by rolling back the relevant WhatsApp change. After the rollback, we verified recovery against affected examples and confirmed that the incident was resolved. ## Preventative actions * Move WhatsApp sender resolution toward more stable identifier-based matching rather than relying on ambiguous phone-number formatting. * Expand regression coverage for country-specific phone-number formatting edge cases. * Improve monitoring and alerting for message delivery failures and queue growth. * Improve testing workflows for WhatsApp changes before production rollout.

Kustomer has resolved an event affecting Chat on Prod 1 that caused messages to fail and assistants to not follow the configured flow. After careful monitoring, our team has determined that all affected areas are now fully restored. Please reach out to Kustomer support if you have additional questions or concerns.

## Summary On May 1, 2026, customers using the Gmail integration experienced a service event that caused Gmail connections to disappear and the email channel to stop functioning. The issue affected customers across different regions, more specifically: * EU-based clients, from 2:40 AM ET. * US-based clients, from 5 AM ET. The issue was fully resolved by 9:20 AM ET. No customer data was lost during this event. Messages that were delayed during the incident were fully recovered once service was restored. Current system health is stable. ## Root cause A configuration issue in the infrastructure used by the Gmail integration prevented replacement service tasks from starting correctly during routine task rotation. As running capacity declined over several hours, the Gmail integration became unavailable. ## Timeline | Time \(ET\) | Event | | --- | --- | | May 1, ~12:30 AM | Service task replacement failures began in production. | | May 1, ~2:40 AM | Engineering was alerted after service capacity dropped to zero in one production environment. | | May 1, 2:40–9:20 AM | Teams investigated the failure, identified the configuration problem, prepared a fix, and deployed it. | | May 1, 9:20 AM | Fix deployed and service restored. | ## Lessons and improvements * We are auditing related infrastructure configurations to identify and correct similar patterns in other services. * We are standardizing how service roles are managed so this class of configuration issue is less likely to recur. * We are improving alerting so teams are notified earlier when running service capacity drops below expected levels, before a full service interruption occurs. * We are adding additional checks to catch configuration drift and invalid service role references earlier in the deployment lifecycle.

## Summary On April 24, 2026, customers in our prod1 environment experienced a service event that caused elevated errors and latency in messaging-related workflows. The broad cross-customer impact was limited to approximately 41 minutes, from 3:26 PM ET to 4:07 PM ET. During that window, some customers saw failed or delayed messaging operations. The immediate platform impact was resolved the same day, and overall system health returned to normal. We then completed follow-up mitigation to stop the underlying event source and reduce the risk of recurrence. ## Impact * Customers in prod1 experienced elevated API errors and latency in messaging-related workflows. * The broad cross-customer impact lasted about 41 minutes. * A subset of messaging workflows failed or were delayed during that period. ## Timeline * **~3:25 PM ET** — A newly enabled automation began processing a large backlog of historical conversations for one tenant. * **3:26 PM ET** — Elevated errors and latency began affecting shared messaging workflows in prod1. * **4:01 PM ET** — We published a status update for the production issue. * **4:07 PM ET** — Broad cross-customer impact ended as the affected services stabilized. * **~5:17 PM ET** — We disabled the triggering automation configuration for the affected tenant. * **~5:23 PM ET** — The remaining retry activity stopped. ## Root cause The event was triggered when a newly enabled automation for one tenant processed a much larger set of eligible conversations than intended. That sudden volume overloaded a shared downstream service and caused elevated errors and timeouts in dependent workflows. The incident was amplified by missing safeguards in how this automation handled backlog volume and retries. In particular, the system did not sufficiently limit the number of conversations processed at once or prevent the same failed work from being retried too aggressively. ## Resolution We restored platform stability during the incident by allowing the affected services to recover under increased capacity, then disabled the triggering automation configuration and cleared the remaining retry backlog. System health is currently normal. ## Preventative actions We are treating the following preventative actions as a priority bug effort. These actions are expected to be resolved by the end of May in accordance with our SLOs: * Prevent newly enabled automation settings from processing large historical backlogs unintentionally. * Add stronger batch limits and tenant-level throttling for this workflow. * Reduce retry amplification by improving how failed work is tracked and re-queued. * Improve error handling so rate-limit conditions are classified correctly and handled with the right retry behavior.

## **Summary** ‌ On April 14, 2026, customers experienced errors accessing the Kustomer Knowledge Base, with all KB pages returning 500 errors. The issue was caused by a dependency upgrade in a recent deployment that introduced a TLS certificate mismatch in internal service communication. Customer impact began at 9:51 AM ET when the deployment reached the first production environment, and was fully resolved by 10:38 AM ET — a ~47 minute impact window. Engineers identified the root cause and completed a full rollback across all production environments within 12 minutes of the initial alert. ‌ ## **Root Cause** ‌ A recent deployment to the KB service included an upgrade to an internal library that contained a known issue with TLS hostname verification. This caused internal service-to-service requests to fail, resulting in 500 errors for all KB requests. The affected library version had previously been identified as problematic in a staging environment, but the fix had not been fully applied across all services before this deployment reached production. ## **Timeline** ‌ **Apr 14, 2026** ‌ 9:51 AM ET — Deployment reached the first production environment; customers began experiencing 500 errors when accessing the Knowledge Base ‌ 10:26 AM ET — Automated alerting fired; incident response began ‌ 10:31 AM ET — Engineers identified a recent deployment as the likely cause and began investigating rollback options ‌ 10:35 AM ET — Root cause confirmed as a problematic internal library version; rollback initiated across all production environments ‌ 10:37 AM ET — Rollback completed on prod1; KB restored for affected customers ‌ 10:38–10:39 AM ET — Rollback completed across remaining production environments ‌ 10:45 AM ET — Full KB functionality confirmed restored for all customers ‌ 12:09 PM ET — Corrected fix deployed to the Knowledge Base service; remediation completed across all other affected services ‌ ## **Lessons/Improvements** * Implementing stricter controls to prevent pre-release or beta library versions from being deployed to production * Improving the process for tracking and completing cross-service remediation work when an issue is identified in one service, to ensure all affected services are addressed * Enhancing our pre-production validation process to improve detection of this class of issue before it reaches production environments

## Summary On March 19, 2026, some Kustomer AI features were unavailable for customers hosted in one production environment \(Prod1\). The disruption affected AI-powered responses and related AI workflows for 41 customers in our Prod 1 instance. Other production environments were not affected. Service was fully restored within 4 hours, and the platform is operating normally. ## Root cause The incident was caused by a production database index change that was introduced during a service deployment. That change caused database performance to degrade significantly for a high-traffic dataset used by our AI services. As performance deteriorated, dependent services were unable to start or process requests normally, which led to broad disruption across affected AI features. Recovery was prolonged by duplicate records created during the incident window, which complicated restoration of normal database constraints and required additional remediation before services could be brought back cleanly. ## Timeline All times below are in EDT \(UTC-4\) on March 19, 2026. * **~10:27 AM** — A production deployment introduced a database index change that degraded performance for AI-related services. * **~10:37 AM** — Customer impact was confirmed and incident response began. * **~10:42 AM** — We published a public status update and began active mitigation. * **~11:37 AM** — We identified the primary cause and focused recovery on database stability and service restoration. * **~12:10 PM** — After scaling database capacity and reducing load, service recovery began. * **~1:43 PM** — AI services began recovering and queued work started draining. * **~2:19 PM** — Backlogged work had been processed and core functionality was restored. * **Later that afternoon** — Follow-up cleanup was completed and normal safeguards were re-applied. ## Lessons and improvements We take this incident seriously and are making changes to reduce the likelihood of recurrence. * We are tightening how database schema and index changes are deployed in production, including stronger pre-deployment validation and safer rollout sequencing. * We are removing index-management behavior from service startup paths where it can create unnecessary risk during deployment. * We are improving monitoring and alerting so service health failures and database stress are detected earlier. * We are refining incident response procedures, access readiness, and operational runbooks to speed mitigation during future incidents. * We are reviewing capacity and resilience safeguards for this part of the platform to better handle abnormal database load.

## Summary On February 20, 2026, a frontend change intended to improve performance unintentionally disrupted call setup in our Voice experience. As a result, agents could see an incoming call and click accept/decline, but audio would not connect and calls would time out. We rolled back the change and then deployed an additional fix to make startup more reliable and prevent recurrence. ## What happened When the Voice widget starts, it needs to receive a small set of configuration settings \(including feature-flag values\) from the main app before it can route and connect calls correctly. A pre-existing timing edge case meant that, in some cases, that initial configuration could be sent before the Voice widget was fully ready to receive it. Historically, the widget would receive the same configuration again shortly afterward, which masked a state issue. The performance change reduced those repeated configuration sends \(which is normally a good optimization\). But because the widget sometimes missed the first configuration during startup, it could start with incomplete settings and fall back to legacy call-routing behavior. In that fallback mode, calls could appear answerable in the UI but fail to fully connect audio. ## Root cause A performance optimization changed how/when configuration settings were delivered during Voice widget startup. Combined with a pre-existing startup timing edge case, some sessions did not receive the required configuration in time and fell back to legacy call-routing behavior, preventing audio from being bridged correctly. ## Timeline * 3:04 PM ET: Frontend change deployed * 4:38 PM ET: Reports received that calls could not be answered \(accept/decline visible, no audio\) * 4:44 PM ET: Initial rollback attempt of an unrelated change did not resolve the issue * 4:50 PM ET: Incident communication initiated via status page * 6:24 PM ET: Frontend change reverted/rolled back * 8:47 PM ET: Reports continued for some agents * 10:31 PM ET: Identified that affected sessions could persist without a browser refresh due to cached UI state; refresh/new session restored correct behavior * Saturday 9:00 AM ET: Confirmed reports that calls were successfully connecting ## Lessons / improvements * Harden Voice widget startup so required configuration can’t be missed during initialization. * Add automated smoke/E2E coverage for core Voice call flows \(answer/connect audio\) to detect regressions before production. * Improve safeguards and monitoring around call-connect failures to reduce time-to-detect and time-to-mitigate.

The scheduled maintenance has been completed.

The scheduled maintenance has been completed.

The scheduled maintenance has been completed.

Kustomer planned maintenance on Affected_Feature is now complete. Platform performance in all areas should now be fully restored. Please reach out to support if you have any questions or concerns.

The scheduled maintenance has been completed.

The scheduled maintenance has been completed.

The scheduled maintenance has been completed.

The scheduled maintenance has been completed.

The scheduled maintenance has been completed.

The scheduled maintenance has been completed.