Historia awarii Chili Piper

We are receiving multiple reports of HubSpot-based customers unable to load objects in their instances, which also affects their rule evaluation and routing. Our teams are investigating the source, as we're getting timeout errors from Hubspot API connections. This will cause issues routing, booking or properly updating Hubspot records. We have now identified this is a result of outage from HubSpot. This can be confirmed via their status page: https://status.hubspot.com/ Our team will continue to monitor this and provide an update.

We are continuing to investigate this issue.

HubSpot has now confirmed it was partially unavailable in North America from 09:34 AM to 10:25 AM (UTC -04:00). They investigating the cause and will update this page when they have more information. We are starting to see the issues resolved in Chili Piper now. We're also actively monitoring further updates from HubSpot.

HubSpot has now confirmed a fix to the issue. We have also confirmed restoration of services in Chili Piper.

We are aware of potential delays while routing records through Distro in one of our environments. As a result, you may notice that Distro is not identifying and, consequently, routing new records. Our Engineering Team is already aware and actively working on a resolution as their top priority. This is not impacting other products such as Concierge, Chat, Handoff, and their respective routing, nor ChiliCal and its browser extension. In case of any further questions, please reach out to our Customer Love Team at [email protected]

As of approximately 9:00 AM Eastern Time, our team is investigating reports of issues with rule processing across products. his is causing routing failures in Handoff, Concierge, and potentially other services that rely on rule and team assets. We are actively working to identify the root cause and will provide updates as more information becomes available.

This has now been resolved after our Engineering Team rolled back one of our services. Rules, Teams and routings are expected to be processed normally now across all products that rely on these assets. We have also already identified what caused it, so any new deployments done for this service should include a fix.

We see some customers have connectivity issues with Salesforce CRM. In the app, it will display "In Trouble" status when it is not connected. All CRM functionality, such as Contact/Account routing, email lookups, and rules logic may not work correctly while this is the status. We are internally working to reestablish the connection for affected customers, but in the meantime, if you see "In Trouble" or "Disconnected" status in the Admin Center's Integrations page, please try reconnecting to Salesforce from the appropriate integration user's Salesforce account, which should restore the connection. More updates to be shared soon.

We have deployed a fix for the Salesforce connectivity issues. We will also apply a retroactive update to proactively reestablish the Salesforce connection for affected tenants. The ETA for completion is approximately 30 minutes. In the meantime, if you see your Salesforce connection status currently set to "In Trouble," you can also have the integration user reconnect to Salesforce in the admin center, which will resolve the problem immediately.

The retroactive update has been completed, and all accounts should properly be in a connected status at this time. We do have a retry mechanism that will automatically re-attempt CRM Actions in Salesforce if they had been previously unsuccessful in most cases.

Heads up: Chili Piper has officially deprecated our Legacy service from the HubSpot Marketplace. This legacy app was retired last year and is no longer in use. There is no impact to any functionality or connectivity on the current Fire / Demand Conversion Platform. However, the removal may have triggered some automated emails that lack context. You may have received messages such as: Chili Piper Scheduling deleted by provider Log in with HubSpot deleted by provider Chili Piper - Legacy deleted by provider These notifications can be safely ignored with no further action required. Note: You may have received these emails even if you're currently on Salesforce, if your account was ever connected to HubSpot in the past. This does not mean you are currently connected to HubSpot.

We see some customers have connectivity issues with Salesforce CRM. In the app it will display "In Trouble" status when it is not connected. All CRM functionality, such as Contact/Account routing, email lookups, and rules logic may not work correctly while this is the status. If you see "In Trouble", or "Disconnected" status, please try reconnecting Salesforce from the appropriate integration user's Salesforce account, which should restore the connection. At this time, we are unsure if the In Trouble status is a result of the Salesforce Core Services outage they are experiencing, but this is something we are considering and are investigating.

We’ve identified several potential root causes and are beginning to roll back the impacted services. We’ll be closely monitoring the system to confirm that these changes produce the expected improvements. At this time, we are still seeing an increase in disconnected customers. We’re actively working to assess the full scope of impact and will communicate directly with affected customers as soon as we have clearer visibility. We’ll continue to share updates as we learn more.

A fix has been deployed, and we are not seeing any additional increase in disconnections. We will continue to monitor this to ensure the issue is resolved, and we may be able to retroactively repair the connection status rather than requiring anyone reconnect. However, if you see your Salesforce connection status currently set to "In Trouble" you can also have the integration user reconnect to Salesforce in the admin center and this too will resolve the problem immediately.

We’ve implemented a retroactive update that ensures tokens remain valid for all tenants that were previously showing an "InTrouble" status. At this time, all tenants are now reflecting as properly connected, aside from any unrelated pre-existing issues (e.g., invalid permissions, invalid users, or previously disconnected accounts). Please note that CRM actions occurring between approximately 13:00 UTC and 20:00 UTC may have failed to create or update as expected. Our retry system is actively working to replay these events, and we expect the majority to be successfully recovered. We’ve identified the underlying root cause as a recent change to credential retrieval that was deployed earlier this morning. This issue was not detected by our internal monitoring systems, and we are incorporating this gap into our ongoing efforts as we finalize a complete fix. We will continue monitoring and provide further updates as needed.

We're investigating reported issues that the Chat Inbox feature of our admin center is not loading for customers. This does not impact customer-facing chat appearance, however it will impact end-user ability to reply. As such, we are treating this as a partial outage to the Chat service. AI-based chats and pre-built chat journeys that require no inbox interaction will not be impacted by this incident. Additionally, this doesn't impact the ability to manage or build a Chat journey.

This incident has been resolved. Internal action was taken to ensure that this specific failure point in the chat service cannot again in the future.

Our team started investigating an issue where Workspaces and licenses are not loading for Admins. The issue appears to be related to our payment processor/billing service, whose own API is erroring. As we access this API for workspace-related data, such as licenses and even product access within the app, this causes the pages to become unresponsive. This only affects the internal Admin Center, Workspaces access, and product-related menus within the app for Admins. It does not impact products or functionality, including booking, routing, and assignment. We will keep this incident updated as soon as we have more updates.

We have observed that our payment processor's API is stable again. Workspaces, licenses, and product-related menus should be available in-app again. Since the issue occurred outside our ecosystem, we are still waiting to hear more details about the root cause from our payment processor team. On our end, we have ongoing efforts to better handle third-party outages, aiming to prevent any interruptions to our workflows and license access in such scenarios.

We are currently investigating an issue that is preventing email notifications from being sent. This means Chili Piper users will likely not receive notifications, such as meeting confirmations and record assignments. This is not impacting Concierge, Concierge routing, the booking/scheduling platform, or ChiliCal's extension. Also, prospects are expected to receive email notifications and calendar events, as the issue is isolated to Chili Piper users' notifications. We will send more updates as soon as our investigation moves forward. If any questions, please reach out to [email protected]

We are now considering this issue resolved, as we're seeing better results. The issue lasted from 17:30 to 19:20 UTC and impacted email notifications such as meeting confirmations emails, record assignment emails, and Reminders for Chili Piper users. The internal service responsible for such notifications had some configuration issues, which were already mitigated. Our teams are already working on lessons learned and how we can avoid this from happening in the future. The issue has not affected the ability to book meetings across products, including our extension, assigning records via Distro, and prospects from receiving meeting invites and calendar event confirmations. If you have any further questions, please reach out to [email protected]

We are currently investigating reports of increased load times affecting legacy links and redirects. The impact we are seeing is on both links and web form snippets. This impacts both standard booking links (meeting types, queues), and Concierge router links. Load times vary from 10-30 seconds on impacted calendar links. Once links load, it appears they do not natively redirect to Fire platform as they are supposed to. As this is impacting only our legacy platform, a potential fix would be to fully migrate services and links to their redirected Fire platform counterparts as they are not impacted. No Fire services appear to be impacted at this time.

It appears the issue may stem from Cross-Origin Resource Sharing (CORS) adjustments that were made. As a result, the legacy Canary platform links are blocked from accessing various resources needed to properly load booking link pages. Our team is working to restore access to these resources now. This does not impact Fire platform or Fire services including Fire scheduling links, and it appears to be limited to a specific "Canary" environment within our legacy platform.

We've completed the necessary configuration changes and access to resources has been restored for impacted legacy customers. A thorough review of this configuration and what occurred will be completed to ensure it remains stable.

We have been alerted to potential delays while routing records through Distro. As a result, you may notice that Distro is not identifying and routing new records. Customers may also have identified these delays yesterday (12/4) between 11:30 AM and 3:00 PM UTC. This delay was caused by an influx of triggered events that exceeded the maximum load possible. This caused the event processing time to grow exponentially for all customers. Services are expected to recover, identify, and route any delayed records soon, with no action needed on the Chili Piper Admin's side at the moment. This is not impacting Concierge, Concierge routing, the booking / scheduling platform, or ChiliCal's extension. We will keep monitoring the results and will soon provide more updates. If you have any questions, please email [email protected]

This incident has been resolved.

Our team has received reports and has found evidence that in many cases the Google calendar service is not creating Events on the calendar. This appears to be a wide-spread issue, but does not appear to impact all meetings booked. We are investigating the root cause of this issue now and will provide recommendations once we discover more about the circumstances of this issue.

Our team identified that the issue was related to a deployment of our integration services at approximately 15:32 EST (20:32 UTC) for an incident span of approximately 2 hours. We rolled back the deployment and this resolved the issue, and we saw error rates drop to zero. It is likely that any meetings booked during this time may have been impacted where no Google calendar event was created in its place. We strongly recommend manually creating these events if they're identified in the meantime rather than waiting on a replay action (if this is possible from our end) as this will ensure that all parties have the most up-to-date calendar appearance, and ensures prospects receive meeting invites. Apologies for the disruption this may have caused.

We discovered this morning that Outlook invites and events were also impacted by the same service failure, extending the scope to both Google and Outlook customers. We are working on recovering all failed events, but don’t have a specific timeline for this action. Our hope is to get this finalized today. It’s likely that, while we are waiting on recovery, other meetings may get booked into those open calendar slots. This would result in overbooked meetings once the recovery finalizes, but this is the only way we can accommodate both parties who booked meetings during that time slot. One of the remaining events would ultimately need to be rescheduled manually. We are investigating our processes that led to the release of this bug and will improve to ensure more thorough testing is performed before release. Apologies for the inconvenience this has caused.

Our team started investigating an issue where Workspaces and licenses are not loading for Admins. As a side effect, some products may appear inaccessible within the app. This only affects the internal Admin Center and product-related menus within the app. It does not impact products or functionality, including booking or routing. We will keep this page updated once we have more updates.

We are continuing to investigate this issue.

The issue appears to be related to our payment processor, whose own API is erroring. As we access this API for workspace-related data, such as licenses and even product access within the app, this causes the pages to become unresponsive. We have already escalated this to their team with high priority, who have already acknowledged the issue. We will send more updates as available.

We have observed that our payment processor's API is stable again. Workspaces, licenses, and product-related menus should be available in-app again. Since the issue occurred outside our ecosystem, we are still waiting to hear more details about the root cause from our payment processor team. On our end, we have already started coordinating efforts to better handle third-party outages, aiming to prevent any interruptions to our workflows and license access in such occurrences.

The service responsible for processing Salesforce actions is currently experiencing degraded performance, which can cause delays on the following related Salesforce actions: Create Event, Create or Update Record, Update Owner, Add to Campaign, and Update Field. You may see your Salesforce actions as "Pending" or "In progress". The booking/scheduling process and apps are not affected, meaning meetings should be booked normally. Our teams are currently working to mitigate delays, and we will keep you informed once we have more updates.

Our teams reported that the consumption should be back to normal. You should notice previously delayed Salesforce actions being marked as Successful, and new ones should be processed within our standards. Over the past few days, the system reached a critical load level that exposed a performance bottleneck in our Salesforce data processing. After thorough analysis, we identified and resolved the issue, restoring normal operation and improving overall throughput. We are continuously improving our systems and infrastructure to process customer data more efficiently and reliably. As data volumes increase, we regularly adjust configurations and upgrade underlying tools and libraries to maintain stable performance. If you still notice any unexpected behavior, please report it to [email protected] for further analysis.

Microsoft services, including calendars, Azure, and hosted office apps are all experiencing degradation due to a global Microsoft issue. As a result, this may impact the ability to access or load some Chili Piper pages and services as well. All legacy customers will likely experience some slowness or page loading issues. Most Outlook/Office-based customers will see some performance issues as well. We are monitoring from our end, however for status updates please check official Microsoft service channels such as : https://azure.status.microsoft/en-us/status https://status.cloud.microsoft/m365

We are seeing services recover, so are marking this "Resolved". For more in-depth information about the incident and its current status please check the following status updates provided by Microsoft: https://azure.status.microsoft/en-us/status https://status.cloud.microsoft/m365

We have been alerted to potential issues routing records through Distro. In most cases these do recover and route after a delay, but in many instances the routes don't appear to be processed at all. As such, we are treating this as a critical incident within the Distro service. This is not impacting Concierge, Concierge routing, or the booking / scheduling platform. Our team will continue to monitor this and work with our engineers to get to the root cause as soon as possible.

Our team applied improvements to our Distro service, and we’re seeing better results. Still, records that should’ve been routed via Distro while the service had degraded performance may take some time to be processed. It was also identified that this affected only one of our environments, so we don’t expect all Distro users to be affected by this. We will keep monitoring the results and will soon provide more updates and a root cause. If you have any questions, please email [email protected]

Distro is back to an operational state. The issue appears to have been caused by an influx of triggered events which exceeded the maximum load possible. This caused the event processing time to grow exponentially for all customers. We are correcting for this in several ways including improving current services to accommodate the increased load, and also improving the strategy for how events are processed. While some of these improvements are ongoing, the incident is considered resolved.

Our team has begun investigating an issue where any admin attempting to access workspaces or admin center internally will have the app crash and show "Something went wrong. Please, reload the page". While reloading the page does seem to work occasionally, the page will eventually crash again and has other performance issues loading. This only impacts the internal admin center and is not impacting products or functionality including booking or routing.

The issue appears to have been related to our payment processor, whose own API had an outage. As we access this API for various admin center and workspace related data such as licenses and even product access, this caused the pages to become unresponsive and crash. We have observed the related API self-healing, and access to workspace and admin center have both been restored, however as this service is outside of our ecosystem we are monitoring now to confirm that everything is back to being stable. We are going to examine how we can better handle (gracefully) third-party billing and payment outages such as this in order to prevent interruption to workflows, while also maintaining product license access in the future.

We have had reports that Hubspot is no longer functional for all customers due to a new scope being requested on the authorization. Our teams are investigating the source, but the issue has caused all Hubspot API connections to drop. This will cause issues routing with Concierge, or properly updating Hubspot records. We are investigating a potential solution, or temporary workaround, that can be provided until we identify the root cause.

A fix was deployed and all connections to Hubspot are working again. We have confirmed that customers are able to route and update Hubspot records properly. It was noted that not all Hubspot CRM customers became impacted by this issue immediately. We began receiving reports as early as 3pm EDT(7pm UTC) and so you may see issues in routing for at least this amount of time. During that time, customers may not have been able to route / book a call. If you find that your Chili Piper account is still not properly connected to Hubspot, or if you continue to experience issues with routing and are a Hubspot customer, please try browsing to your integration settings within the Chili Piper admin panel and reconnecting Hubspot. The root cause appears to be related to a permission scope that Hubspot used to provide "as requested" now requires explicit permissions for it to work. Since we didn't authenticate this permission in the past, all authorizations were failing. We have deployed a fix which essentially removes the need for this scope. Our team will explore if this scope is going to be needed in the future and will take appropriate measures if this is the case. We will also continue monitoring to ensure that all instances are back up and running properly with this provided hotfix. Please feel free to reach out to [email protected] for further information.

We are in the process of moving to a new billing / payment processor this morning. As a result, you may have temporarily lost the ability to login for a few seconds up to a few minutes, however this should not impact any workflows, routing, or bookings. If you notice the inability to login, please wait 5-10 minutes and try again. If you continue to experience issues, feel free to reach out to [email protected] so that our Customer Love team can take a closer look for you! Apologies in advance for the inconvenience this may have caused.

Chili Piper experienced a full platform outage related to services dependent on Kafka. These services were unable to send or receive data during this time which essentially caused all products to go offline and lose access. The outage was detected immediately by our internal alerting system and our on-call Site Reliability Engineering team began working to correct the issue. We raised the escalation with the Confluent Cloud team through all available channels. Approximately 1 hour and 50 minutes after the incident began, Confluent confirmed that their internal teams were actively engaged in resolving the issue with additional escalation within Confluent prioritized at this time. To mitigate the issue, we created a new subscription through their system in order to restore access. Service was fully restored at 19:12 UTC (3:02pm EDT). No existing data was lost, but no new data was processed during this time which means no routes or meetings could be booked through Chili Piper.

## **Action Taken** * Escalated to Confluent and engaged with their engineering team throughout the incident. * Internally began preparation of a fallback Kafka cluster independent of Confluent, to be used if needed. * After recovery, performed thorough verification of platform functionality and monitored stability. * Initiated and coordinated an investigation with Confluent. * Requested and received a detailed postmortem from Confluent outlining root cause and contributing factors. ## **Follow-up Steps** * Evaluate feasibility and timeline for decoupling from Confluent to ensure greater control and resilience. * Set up monitoring and alerting specifically for Kafka cluster reachability to detect future issues earlier. * Review and improve vendor migration validation processes. * Review Confluent root cause analysis and postmortem