Historia awarii Pianoio

We are currently experiencing a minor service disruption affecting user logins to the Piano dashboards, caused by an ongoing outage on Cloudflare's side. Our team is actively monitoring the situation. We apologize for any inconvenience this may cause and appreciate your patience. Further updates will be provided as soon as more information becomes available. For reference: https://www.cloudflarestatus.com/incidents/v3yl7jqmqj51

The incident has been mitigated, along with our provider's resolution. We will continue to monitor the situation.

This incident has been resolved.

Since approximately 11:00 PM (UTC) on June 8, 2026, there have been intermittent delays loading files into Snowflake. The average load duration is currently about 10 minutes, compared to the usual under 2 minutes. The issue has been identified and a ticket has been opened with Snowflake.

The incident has been resolved and is no longer affecting Piano Analytics. We apologise for the inconvenience caused.

**Incident explanation:** The latency was caused by an issue on our cloud data platform provider's side. An internal infrastructure configuration change introduced resource contention that affected workloads with high ingestion concurrency, leading to elevated response times and intermittent timeouts when loading files. The change has been reverted by the provider, and ingestion performance has fully returned to its normal baseline. **Actions performed:** Opened and tracked an incident with our data platform provider and provided detailed monitoring evidence to support their investigation. Deployed mitigations on our streaming ingestion stack (increased service scaling and improved handling of stalled ingestion calls) to limit customer impact during the incident. Temporarily tuned ingestion timeout and retry settings to keep file loading as stable as possible while the root cause was being addressed. Continuously monitored ingestion latency and error rates until performance returned to normal. Confirmed full recovery, then reverted the temporary configuration to restore standard behaviour and avoid masking any future issues. **Actions to come:** Continued close monitoring of ingestion latency over the coming days to confirm lasting stability. Keep the temporary mitigation patterns configurable in the code so they can be re-applied quickly should a similar provider-side degradation occur.

We are currently experiencing an issue with the Content Crawler, which is failing to crawl content. Our team is actively investigating the root cause. We will provide further updates as more information becomes available.

We are continuing to investigate this issue.

The issue has been identified and a fix is being implemented.

A fix has been implemented and all Content pipeline services have been restored. Some processing delays may persist while the crawling backlog clears. We will continue to monitor and confirm full resolution.

This incident has been resolved.

RCA report for the incident is available at [**this link**](https://docs.piano.io/wp-content/uploads/2026/06/2026-05-29-RCA-Content-Crawler-Service-Disruption-1.pdf).

We identified an issue in our systems starting around 10:20 UTC that is currently degrading CXSense platform responsiveness over the /public/user/id endpoint. As a result, some users may experience failed requests, while others may experience longer response times and timeouts. Our engineering team is actively investigating and working on mitigation efforts. We will continue to provide updates as more information becomes available.

A fix has been implemented and we are currently monitoring the results. At this time, API performance has returned to normal operation levels and the platform is behaving as expected again. We will continue monitoring closely to ensure stability.

This incident has been resolved.

RCA report for the incident is available at [**this link**](https://docs.piano.io/wp-content/uploads/2026/06/2026-05-28-RCA-Audience-API-Failures-happening-in-Europe-environment.pdf).

On 20 May 2026, starting at approximately 09:58 UTC, we identified an issue affecting Piano Activation in the US environment, impacting the Dashboard, Piano ID authentication, and checkout flows. A fix has been applied and the platform is now operating normally. We are continuing to monitor the situation and further details will follow. We apologise for any inconvenience caused.

The incident has been resolved. All services are operating normally. Between approximately 09:58 and 10:11 UTC, users in the US region may have experienced issues accessing the Piano Dashboard, signing in via Piano ID, or completing checkout flows. This was caused by a routine configuration update to our US environment that introduced a conflict and disrupted traffic routing to the affected services. Our team identified the issue, rolled back the change, and confirmed full recovery shortly after. To prevent this from happening again, we are reviewing our release process and will validate the change in a controlled manner before re-applying it. We apologize for the inconvenience and thank you for your patience.

We are currently experiencing degraded performance across the Publisher Dashboard in the AP environment, with data taking up to 3 hours longer than usual to reflect recent changes. This may affect any section of the Dashboard — including user search, reporting, and other data views. Our engineering team has identified the underlying cause and is actively working on a resolution. We will provide updates as the situation progresses.

This incident has been resolved.

We are currently investigating issues affecting the execution of experiences in our European region for segmentation that relies on custom field rules. Impacted users may notice that experiences are not triggering as expected, not reaching the desired outcome. Our engineering team is actively investigating the issue.

The issue has been identified and fixed; we're currently monitoring the experience executions.

This incident has been resolved.

An infrastructure update to the EU environment introduced an incompatibility that prevented Composer from retrieving Custom Fields. The issue was limited to the EU region and did not affect other environments. The update introduced a behavioural change in the network routing layer where a configuration setting that had previously been relied upon was no longer honoured under the new setup. This incompatibility was not detected during pre-production testing, as the test environments did not fully replicate the conditions needed to surface the issue. The root cause was identified and the configuration gap was addressed promptly. Piano is taking steps to improve parity between pre-production and production environments and to strengthen the validation process ahead of future infrastructure changes to prevent similar issues.

Between 07:42 and 07:52 AM UTC, Piano experienced elevated error rates affecting the Publisher API and ID API in the EU environment. The incident occurred during an ingress upgrade from a deprecated version to the current supported version, which caused a routing configuration incompatibility. The issue lasted approximately 10 minutes and has been fully resolved. Publisher Dashboard, Checkout, and access controls in Composer operated normally throughout the incident and were not affected.

We are currently experiencing a production incident on Piano Analytics products. Please excuse us for any inconvenience caused.

The issue has been resolved and all Piano Analytics tools and interfaces are fully available. We are currently catching up the delay in real-time data retrieval.

Composer is experiencing a service incident affecting C1X segmentation in both EU and US regions. Impact includes high error rates, degraded segment lookup, and degraded widget performance. The issue started at approximately 22:00 UTC on May 7 and continues with periods of instability. Our team is actively investigating the issue.

A fix has been implemented and we are monitoring the results.

The incident has been resolved.

We are currently experiencing an issue with Composer in our sandbox environment; the Composer module and experiences fail to load. Our team has identified the root cause and is actively working to resolve the issue. We will provide updates as more information becomes available.

The team has deployed a fix, and the issue has been resolved. Experiences are executing, and the Composer module is working normally. We will continue monitoring the remediation.

We are continuing to monitor for any further issues.

This incident has been resolved.

Between 06:48 and 07:05 UTC, Piano ID API experienced elevated error rates in the US cluster. Requests requiring user authentication, token verification, and user profile retrieval were affected. Other Piano services including Composer and VX subscription management were not impacted. Service has been fully restored and all systems are operating normally. We apologize for any disruption caused during this incident.

This incident has been resolved.

We have found an issue with our priority rules for API interaction on server/edge in Composer. The dev team is currently investigating. Clients may find their list order being changed or unsorted without any user interaction. We will continue updating this incident once we identify the root cause.

Our team has restored the priority rules to their pre-incident state while we continue to investigate the root cause. We advise customers to avoid making changes to priority rules until the investigation is complete. We will provide further updates as more information becomes available.

The issue has been identified and mitigated; we will continue monitoring the remediation.

This incident has been resolved.

RCA report for the incident is available at [**this link**](https://docs.piano.io/wp-content/uploads/2026/04/2025-05-06-RCA-Experience-Priority-Data-Incident.pdf).

Composer is experiencing a latency incident affecting executions in the US region starting at 10 AM UTC. Impact includes timeouts and degraded performance.

The incident has been resolved. We will continue monitoring the latency on the backend servers.

We are continuing to monitor for any further issues.

This incident has been resolved.

We are currently investigating an issue affecting the publisher dashboard where users are unable to interact with the experience version dropdown and other buttons in Composer.

The issue has been identified and a fix is being implemented.

This incident has been resolved. The problem was caused by a compatibility issue between two system components that were running different versions. We have updated one of the components to ensure they work together properly, which has restored normal functionality. Some users may need to clear their browser cache to fully resolve the issue and restore normal functionality. We apologize for any disruption this issue may have caused.

Composer is experiencing a service incident affecting Audience segments and Propensity Models in the EU region. Impact includes high error rates, degraded segment lookup, and degraded widget performance.

The issue has been identified and a fix is being implemented.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

Composer is experiencing a service incident affecting Audience segments and Propensity Models in the EU region. Impact includes high error rates, degraded segment lookup, and degraded widget performance.

A fix has been implemented and we are monitoring the results.

The service disruption caused by a temporary load balancing issue has been fully resolved. We are deploying additional API server nodes to better handle future traffic spikes and prevent similar incidents.

Mandrill team has notified Piano about an issue impacting our account. Our team is currently coordinating efforts with the Mandrill account. The emails are queued but not getting delivered at the moment. We will continue to provide updates via this thread. Please monitor our status page for latest information.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

Following an infrastructure update on April 3, a timezone-related bug was introduced, causing anomalies in Site Licensing, Subscription Grace Periods, and Scheduled Upgrades for clients in the Production-AP region. The root cause has been identified and a fix has been deployed. Remediation of the affected entities is in progress. A detailed Root Cause Analysis report will be shared once the incident is fully resolved, covering the scope of impact, timeline, and preventive measures.

The incident has been resolved. A Root Cause Analysis report is currently in progress and will be shared once completed.

RCA report for the incident can be found at [**this link**](https://docs.piano.io/wp-content/uploads/2026/04/2026-04-06-Activation-Timezone-Configuration-Error-AP-region.pdf).

We've received some complaints that end users are unable to pass the CAPTCHA verification. Our development team is currently looking into the issue.

We have identified the issue, and our team is actively working on a fix. We'll keep you updated on our progress.

We have successfully implemented a fix for the CAPTCHA issue. We're actively monitoring the system.

The incident has been resolved.