Historia awarii Chameleonio

We identified a backed-up queue that caused some requests to our backend to be rejected (with a 500/503 error code) -- the impact was that during the time period from 6:10 eastern to 10am eastern only approximately 92% of calls to chmln.identify were successful

OAuth authentication was not working for approximately 3pm est Sunday 2026-03-14 to 8:30 am est Tuesday 2026-03-16

We're looking into an issue where new magic links / OAuth / SSO are not working as expected

The issue has been identified and a fix is being implemented. **Note: If you're already logged in, you will continue to stay logged in and we will likely resolve this before you are automatically logged out**

This incident has been resolved.

Chameleon is not and was not not affected by the recently disclosed MongoDB vulnerability CVE-2025-14847 (aka. MongoBleed). It allows attackers to read arbitrary data from the database's heap memory. Chameleon's MongoDB instances are not publicly accessible and are also hosted with MongoDB Atlas which means they were patched before disclosure. We will, however, keep up to date on any developments from this incident

Chameleon **was not part of** the security incident reported by Mixpanel today (https://mixpanel.com/blog/sms-security-incident/). We will, however, keep up to date on any developments from this incident

We're seeing higher than normal error rates when editing within Chameleon. Our engineers have identified the cause and are working on a fix

Error rates are back down to normal levels

This incident has been resolved.

As of approximately 6:45 AM ET, Cloudflare reported an incident affecting many sites and vendors across the internet (Check https://www.cloudflarestatus.com/ for any direct updates). While Chameleon does not directly rely on Cloudflare, some services we depend on do. Consequently, you may experience the following: Copilot: OpenAI powers this feature and is currently impacted. Integrations: Data from integrations (Mixpanel, Amplitude, HubSpot, etc.) may be delayed until the incident is resolved. Recommendation: If your own product is experiencing downstream issues, we encourage you to use Chameleon to publish a downtime banner to alert your users. Please reach out if you need assistance setting this up.

According Cloudflare and others, this incident is resolved

During the AWS outage, Chameleon remained online and was not directly affected by the ongoing issues. All Chameleon Experiences were published on schedule, and new experiences went live as expected. Many customers also used Chameleon to notify *their customers* via in-product banners linking to their own status pages. During this period, we ingested and queued a large volume of data — typical for EU and US Eastern morning hours. However, today our auto-scaling was unable to bring additional resources online due to the AWS and Heroku issues. As a result, we are currently processing a backlog of data and background jobs. The current processing delay is approximately 30 minutes (Experience metrics are delayed by this amount, but microsurvey responses are not). This backlog will decrease quickly once auto-scaling resumes normal operation.

Updating to have a "Minor" impact

Our current application / background servers are running as-is but our ability to scale our cluster is impacted. We will need additional resources provisioned within the next few hours to bring the backlog back into a manageable length. We will continue to monitor and implement any fixes that we can along the way and will plan to send another update within the next 2-3 hours

Currently the Segment page (https://app.chameleon.io/segments) is only loading for a small percentage of requests due to upstream impact at Clickhouse => https://status.clickhouse.com

According to AWS "AWS services which rely on EC2 instance launches such as [Heroku] are working through their backlog of EC2 instance launches successfully and we anticipate full recovery of the backlog over the next two hours" <= for Chameleon this means out data backlog should start processing more quickly automatically over the same time frame

The backlog of events has been processed and any new data coming in will be processed in near real time. TL;DR; Aside from the data backlog (peaked at 2 hours delay), and a ~10% error rate for loading Segments of users (in the Chameleon dashboard; no impact for your end-users), it was a relatively normal day at Chameleon in terms of the "rate of experiences delivered" for the baseline. Chameleon delivered approx. 200k downtime notices I hope today was not too stressful for you / your business and as always feel free to reach out with questions

Privacy update (not an incident) We’ve recently added a small number of trusted vendors to support key workflows across our platform, including: - Anthropic – AI functionality for workflow automations - Fathom – Call recording and transcription - Linear – Product issue and roadmap management - Synder – Financial system integrations - Mutiny – Website content personalization - Chili Piper – Meeting scheduling - Unify GTM – Go-to-market orchestration We’ve also streamlined our list by removing subprocessors no longer in use. These changes reflect our commitment to using modern, secure, and effective tools that help us improve our processes and deliver the best product and service to our customers. You can view the full, current list of subprocessors at our new subprocessors page: 👉 https://www.chameleon.io/security/subprocessors If you have any questions, feel free to reach out to us at [email protected]. Thanks, The Chameleon Team

We've identified instability in one of our internal APIs, which is causing intermittent 500 errors when editing and displaying Experiences. We've identified the root cause, and our team is actively working on stabilizing our services.

Solved: We've fixed the instability issues by scaling our service capacity.

We've identified an issue stemming from this Heroku incident https://status.heroku.com/incidents/2822 (upstream of Chameleon) which is causing us to have limited access to the heroku platform. We use Heroku for auto-scaling, console access, and deployment. This incident is currently affecting the availability of the Chameleon Dashboard, Builder, and our ability to run End-user Experiences (Tours, Launchers, Microsurveys).

The issue stems directly back to the Heroku outage (https://status.heroku.com/incidents/2822). Our current deployment of servers is running and handling the traffic to an acceptable level but we're unable to make any changes to the deployment, the number of servers (via autoscaling). We're seeing more total percentage of successful requests to Chameleon and that is also trending in the right direction. Will update again when we have new information

We continue to track the Heroku outage but continue to see intermittent issues with loading Chameleon Dashboard and generally connecting to Chameleon backend applications

We've now seen consistent performance and response times from our Heroku instances for the past 3 hours and are making the incident as resolved. We will monitor the system as a whole now that everything is catching back up and the jobs/work to be done in the background queues. Thank you for your patience

At Apr 16, 2025 12:01pm UTC, Chameleon deployed a code update that caused some Launchers to be shown to users not Segmented to them. Chameleon added an event listener to check for changes to the URL rules, but in case of a new match, this listener accounted only for URL rules, and didn't check Segmentation. The change was rolled back when discovered, at Apr 16, 2025 7:07pm UTC.

Chameleon continues to experience extremely high load and is taking steps to increase capacity above normal limits to address this. The current impact is that some requests to Chameleon are queued for a long time and/or timing out. In the UI this looks like page not loading, auto-saving not working, and for end-users experiences may not display. This currently appears to be happening for a < 10% of traffic but we'll continue to monitor as we work to resolve the issue

Our new capacity has been online for several hours now and is performing as expected. We will keep this incident open for another 24 hours in case of any new developments or issues. Thanks for out patience

We are currently investigating slow downs and 503 errors on profile requests.

We are continuing to monitor system performance as response times and error rates have improved over the past hour. No new end-user-facing timeouts have been observed recently, but we are still actively investigating to ensure stability.

This incident has been resolved, and all systems are operating as expected.

We are currently investigating the cause of intermittent slowness and timeouts resulting in occasional Application errors

We are continuing to investigate this issue.

We have identified the following issues: 1) Experiences sometimes not being delivered to end-users. 2) Experiences sometimes not being able to be opened/edited in Chameleon Builder. We're actively monitoring and investigating the root cause.

This incident has been resolved and systems are back to normal.

We are currently investigating the cause of intermittent slowness and timeouts resulting in occasional Application errors

This incident was kept open for monitoring and has been resolved.

# **Chameleon Incident Report - Feb 10, 2025** ## **Summary** ‌Dashboard pages did not load or were slow, and end-user experiences did not show in a timely manner. This was caused by elevated traffic that resulted in high latency and errors primarily affecting database writes to the `profiles` collection in MongoDB. The issue cascaded into other collections and impacted the overall system health / performance. ## **Main Impacts** * Dashboard availability was reduced leading to errors on pages loading or slowness in loading data * End-user Experiences failed to show, or were delayed, in cases where these depended on updated user attributes \(because Chameleon was slow or not accepting updates to user profiles via the client side\) ## **Root Causes** * Unexpectedly high database “write” traffic exceeded the capacity of our MongoDB instance, leading to operations being queued beyond expected thresholds  * Our utilization of database capacity was higher than best practices in certain metrics \(which meant a smaller buffer before we exceeded capacity\) * Write locks were held longer than expected, causing significant queuing. * The disk was overutilized due to concurrent read and write operations, further increasing response times. * Insufficient RAM allocation to MongoDB exacerbated performance bottlenecks. ## **Resolution** This is what we did to resolve the incident as fast as possible: * Upgraded MongoDB instance RAM and disk capacity. * Monitored database performance post-upgrade to ensure stability. * Lowered concurrency of background processing across certain operations that, if left to run in parallel, would cause a flood or contending write operations. * Identified operations that may not need to be run live \(this will be incorporated into “Future Prevention” below\) ## **Future prevention** This is what we are doing and intend to do, to prevent this type \(similar or related\) incident in future: **Improving our capacity to handle growing volume of database traffic** * Evaluate further MongoDB scaling options to handle peak loads \(short-term\) * Improve our auto-scaling rules and capacity management, e.g. conduct additional load testing to anticipate future demands \(short-term\) * Improve database monitoring to detect and mitigate similar issues \(e.g. we’ve already added “total write time” as a new metric\) before they escalate; we found that our monitoring did not indicate exactly where the problem lay which delayed finding a resolution * Optimize “write” operations to reduce contention and queue buildup, especially those that are not needed to be live and can run in the background with batching, for example when updating company attributes data * Explore adding rate limiting to our Segment integration to prevent batch flooding of data * Accelerate our plans to upgrade our infrastructure to a more modern and state-of-the-art database system \(ClickHouse\) to supplement MongoDB in certain cases **Improving our processes and communication around issues** Partly due to the infrequent nature of incidents like this, we uncovered some material gaps in our approach in communicating these effectively, and accordingly will undertake significant improvements to this, including: * Clearer notices in-product, in our help center, and via email when an incident is occurring * Proactively suggesting to new customers \(and all existing customers\) to join the subscription list for incidents and security updates \(we already have this available via our dashboard and status page but want to increase visibility of it\) * Adding some expected timelines for regular updates to our Status page and completion of the post-mortem report _\(this document\)_ * A proactive email when an incident is resolved \(seems like “resolution” of an incident does not trigger an email update to subscribers from our Status tool, unlike for other incident categories\) and also to send out the post-mortem report * Formalizing support communication \(as we did in this incident\) including a “status ticket” in Intercom where we attach all messages about the incident, and a note in our support bot * Clearer ownership of tasks \(and accountability for these\) during an incident response

We are seeing elevated error rates, and various systems being slow or not loading. At the moment this includes the Chameleon dashboard, identifying users, and sending data to Chameleon. We know this is related to the performance of our primary database and will have an update again shortly.

Response times and error rates are returning to normal and will continue to monitor. At this time, we know that a large increase in latency when writing to the database caused the latency of essentially all other operations to increase. We will leave this incident open until we're sure that the changes have a chance to full stabilize, the background jobs have processed, and the workload returns to normal levels/latencies.

This incident has been resolved.

Post mortem available [here](https://status.chameleon.io/incidents/04c5y9b9d4xd)

Intermittent slowness and timeouts resulting in occasional Application errors

This incident is still ongoing, we will have more updates within the hour. The main impact is that requests that typically take 30-50ms are varying from 30ms to 3s, with some timing out and never completing, this is causing some Chameleon experiences to not start when they otherwise would

We're still actively monitoring this incident and will report any new information we have

This incident is has now been partially resolved, some data ingestion delay will be processed over the next few hours as more capacity is brought online to assist in working through a backlog of events and updates from integrations

This incident is over but will remain open while the final data is processed

Available [here](https://status.chameleon.io/incidents/04c5y9b9d4xd)

We are currently investigating the cause of intermittent slowness and timeouts resulting in occasional Application errors

This incident is resolved

Available [here](https://status.chameleon.io/incidents/04c5y9b9d4xd)

We are currently investigating issues with data collection and log in.

We've addressed the data collection and log in issues customers were experiencing. We are still monitoring this issue while we understand the root cause.

This incident is resolved (as of 4:30pm pacific). The main impact was that certain API actions were resulting in errors. These errors were most obvious for calls to track events where the result of the API call was a 500 error. In addition, some data was delayed in reaching the intended user because of delays in processing the incoming data. An incident like this will have an investigation followed by implementing the recommended updates. As always, you may reach out to us for further information, if needed.

Some data ingestion is experiencing higher than normal latency in being added to Chameleon's analytics charts and aggregate totals. The queues are on track to catch back up soon

This incident has been resolved.