Pianoio outage history

Piano identified an issue affecting access to Piano Insight for some users. The certificate causing the errors has been renewed, and the fix has been verified internally. We are monitoring to confirm the issue is fully resolved and will share a further update shortly.

We are continuing to monitor for any further issues.

This incident has been resolved.

We are experiencing an issue with report generation. Our team is aware and actively working to resolve it.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

Piano is experiencing an issue affecting login and registration for users in the EU region. The login window may fail to open, or may be unresponsive. We have identified the cause and our team is deploying a fix. We will share an update as soon as further information is available.

Piano has identified that this issue affects login and registration across all regions, not only the EU as initially reported. Users in all regions may experience the same symptoms — the login window failing to open, or becoming unresponsive. Our team continues to deploy the fix. We will share an update as soon as further information is available. Affected components: Registration + Login - US - id.piano.io Registration + Login - Australia - id-au.piano.io Registration + Login - Asia - id-ap.piano.io Registration + Login - EU - id.piano.io

We are continuing to work on a fix for this issue.

Piano can confirm this issue in fact only affects the US and EU regions, as well as Sandbox environment. AP and AU are not affected. We apologize for any confusion caused by the earlier update. Sandbox does not have a dedicated status page component since it's not part of production infrastructure, this update is provided for visibility. Our team continues to deploy the fix. We will share an update as soon as further information is available.

Piano has implemented a fix across all affected regions and environments, including Sandbox. Login and registration functionality has been restored and is operating normally. We will continue to monitor the situation and will provide a further update should any issues arise.

This incident has been resolved.

RCA for the incident can be found at [**this link**](https://rca.piano.io/wp-content/uploads/2026/07/2026-07-27-RCA-Identity-Management-login-issues.pdf).

Following recent scheduled database maintenance in the AP region, we have identified a data gap affecting the Subscription Log and Subscription Insights sections of the Publisher Dashboard. Some clients may notice that recent purchases are not yet reflected in the Subscription Log report, and some measurements in Subscription Insights may appear temporarily inconsistent. Our team is regenerating the affected data and expects this to be resolved within the next few days. We will share an update as soon as we have more information.

A fix has been implemented and we are monitoring the results.

We have restored the affected data for Subscription Insights, and measurements there should now be displaying correctly. The Subscription Log report has not yet been fully recovered, and some recent purchases may still be missing from this view. Our team continues to regenerate the remaining data. We will share an update as soon as we have more information.

The data gap affecting the Subscription Log has now been fully resolved, and all recent purchases in the AP region are correctly reflected in the report. Combined with the earlier fix to Subscription Insights, both affected areas of the Publisher Dashboard have been restored to normal. This incident is now resolved. Thank you for your patience while we completed the data regeneration.

We are currently investigating an issue affecting the delivery of paywalls and experiences for publishers in Asia and Australia. Some requests are failing to complete successfully. Our engineering team is actively working to identify the root cause and restore normal service. We will share an update as soon as we have more information.

The issue has been identified and a fix is being implemented.

A fix has been implemented and we are monitoring the results.

An internal infrastructure issue affecting a backend system dependency in our Tokyo data center caused authentication failures that cascaded into request failures for the Audience API. Because Composer traffic for Asia (AP) and Australia (AU) routes through this instance, both regions saw failures of Composer requests. This incident has been resolved. Our engineering team identified the cause and restored the affected system; service has returned to normal in all regions. We apologize for any inconvenience this may have caused.

We are observing data processing delays on the User profile backend caused by data improvement efforts for managing the cx prefix and the composer application ID. If you are using the cxense's external profile update API (https://api.cxense.com/profile/user/external/update) you will observe 1-2 hours of delays when reading the same profile using (https://api.cxense.com/profile/user/external/update). Our team is now reconciling the data back into a single, correct view, and this reconciliation is the source of the ongoing delay. No data has been lost, all events are being processed, currently with an approximate delay of 1–2 hours. We apologize for the inconvenience and will share an update as the backlog clear

Data processing delays continue on the User Profiles backend, with current delays of 1–2 hours. We have identified that a recent data change led to a significant increase in data volume for a subset of customers, which is the primary driver of the backlog. Our team is carefully increasing processing throughput while closely monitoring overall system stability to avoid introducing further issues. We are optimistic about stabilizing the situation soon and will share an update as soon as we have more information.

Data processing on the Piano Audience User Profiles backend has returned to near real-time and the backlog is cleared. We are setting this incident to Monitoring while our team completes residual cleanup from the earlier data change. We will share a further update once the incident is fully resolved.

The incident has been resolved after monitoring for more than 24 hours. We are seeing the processing of the data in real-time

We are observing data processing delays on the User profile backen caused by data improvement efforts for managing the cx prefix and the composer application ID. If you are using the cxense's external profile update API (https://api.cxense.com/profile/user/external/update) you will observe 1-2 hours of delays when reading the same profile using (https://api.cxense.com/profile/user/external/update). Our team is now reconciling the data back into a single, correct view, and this reconciliation is the source of the ongoing delay. No data has been lost, all events are being processed, currently with an approximate delay of 1–2 hours. We apologize for the inconvenience and will share an update as the backlog clear

We are continuing to work on a fix for this issue.

We are currently experiencing a minor service disruption affecting user logins to the Piano dashboards, caused by an ongoing outage on Cloudflare's side. Our team is actively monitoring the situation. We apologize for any inconvenience this may cause and appreciate your patience. Further updates will be provided as soon as more information becomes available. For reference: https://www.cloudflarestatus.com/incidents/v3yl7jqmqj51

The incident has been mitigated, along with our provider's resolution. We will continue to monitor the situation.

This incident has been resolved.

Since approximately 11:00 PM (UTC) on June 8, 2026, there have been intermittent delays loading files into Snowflake. The average load duration is currently about 10 minutes, compared to the usual under 2 minutes. The issue has been identified and a ticket has been opened with Snowflake.

The incident has been resolved and is no longer affecting Piano Analytics. We apologise for the inconvenience caused.

**Incident explanation:** The latency was caused by an issue on our cloud data platform provider's side. An internal infrastructure configuration change introduced resource contention that affected workloads with high ingestion concurrency, leading to elevated response times and intermittent timeouts when loading files. The change has been reverted by the provider, and ingestion performance has fully returned to its normal baseline. **Actions performed:** Opened and tracked an incident with our data platform provider and provided detailed monitoring evidence to support their investigation. Deployed mitigations on our streaming ingestion stack (increased service scaling and improved handling of stalled ingestion calls) to limit customer impact during the incident. Temporarily tuned ingestion timeout and retry settings to keep file loading as stable as possible while the root cause was being addressed. Continuously monitored ingestion latency and error rates until performance returned to normal. Confirmed full recovery, then reverted the temporary configuration to restore standard behaviour and avoid masking any future issues. **Actions to come:** Continued close monitoring of ingestion latency over the coming days to confirm lasting stability. Keep the temporary mitigation patterns configurable in the code so they can be re-applied quickly should a similar provider-side degradation occur.

We are currently experiencing an issue with the Content Crawler, which is failing to crawl content. Our team is actively investigating the root cause. We will provide further updates as more information becomes available.

We are continuing to investigate this issue.

The issue has been identified and a fix is being implemented.

A fix has been implemented and all Content pipeline services have been restored. Some processing delays may persist while the crawling backlog clears. We will continue to monitor and confirm full resolution.

This incident has been resolved.

RCA report for the incident is available at [**this link**](https://docs.piano.io/wp-content/uploads/2026/06/2026-05-29-RCA-Content-Crawler-Service-Disruption-1.pdf).

We identified an issue in our systems starting around 10:20 UTC that is currently degrading CXSense platform responsiveness over the /public/user/id endpoint. As a result, some users may experience failed requests, while others may experience longer response times and timeouts. Our engineering team is actively investigating and working on mitigation efforts. We will continue to provide updates as more information becomes available.

A fix has been implemented and we are currently monitoring the results. At this time, API performance has returned to normal operation levels and the platform is behaving as expected again. We will continue monitoring closely to ensure stability.

This incident has been resolved.

RCA report for the incident is available at [**this link**](https://docs.piano.io/wp-content/uploads/2026/06/2026-05-28-RCA-Audience-API-Failures-happening-in-Europe-environment.pdf).

On 20 May 2026, starting at approximately 09:58 UTC, we identified an issue affecting Piano Activation in the US environment, impacting the Dashboard, Piano ID authentication, and checkout flows. A fix has been applied and the platform is now operating normally. We are continuing to monitor the situation and further details will follow. We apologise for any inconvenience caused.

The incident has been resolved. All services are operating normally. Between approximately 09:58 and 10:11 UTC, users in the US region may have experienced issues accessing the Piano Dashboard, signing in via Piano ID, or completing checkout flows. This was caused by a routine configuration update to our US environment that introduced a conflict and disrupted traffic routing to the affected services. Our team identified the issue, rolled back the change, and confirmed full recovery shortly after. To prevent this from happening again, we are reviewing our release process and will validate the change in a controlled manner before re-applying it. We apologize for the inconvenience and thank you for your patience.

We are currently experiencing degraded performance across the Publisher Dashboard in the AP environment, with data taking up to 3 hours longer than usual to reflect recent changes. This may affect any section of the Dashboard — including user search, reporting, and other data views. Our engineering team has identified the underlying cause and is actively working on a resolution. We will provide updates as the situation progresses.

This incident has been resolved.

We are currently investigating issues affecting the execution of experiences in our European region for segmentation that relies on custom field rules. Impacted users may notice that experiences are not triggering as expected, not reaching the desired outcome. Our engineering team is actively investigating the issue.

The issue has been identified and fixed; we're currently monitoring the experience executions.

This incident has been resolved.

An infrastructure update to the EU environment introduced an incompatibility that prevented Composer from retrieving Custom Fields. The issue was limited to the EU region and did not affect other environments. The update introduced a behavioural change in the network routing layer where a configuration setting that had previously been relied upon was no longer honoured under the new setup. This incompatibility was not detected during pre-production testing, as the test environments did not fully replicate the conditions needed to surface the issue. The root cause was identified and the configuration gap was addressed promptly. Piano is taking steps to improve parity between pre-production and production environments and to strengthen the validation process ahead of future infrastructure changes to prevent similar issues.

Between 07:42 and 07:52 AM UTC, Piano experienced elevated error rates affecting the Publisher API and ID API in the EU environment. The incident occurred during an ingress upgrade from a deprecated version to the current supported version, which caused a routing configuration incompatibility. The issue lasted approximately 10 minutes and has been fully resolved. Publisher Dashboard, Checkout, and access controls in Composer operated normally throughout the incident and were not affected.

We are currently experiencing a production incident on Piano Analytics products. Please excuse us for any inconvenience caused.

The issue has been resolved and all Piano Analytics tools and interfaces are fully available. We are currently catching up the delay in real-time data retrieval.

Composer is experiencing a service incident affecting C1X segmentation in both EU and US regions. Impact includes high error rates, degraded segment lookup, and degraded widget performance. The issue started at approximately 22:00 UTC on May 7 and continues with periods of instability. Our team is actively investigating the issue.

A fix has been implemented and we are monitoring the results.

The incident has been resolved.

We are currently experiencing an issue with Composer in our sandbox environment; the Composer module and experiences fail to load. Our team has identified the root cause and is actively working to resolve the issue. We will provide updates as more information becomes available.

The team has deployed a fix, and the issue has been resolved. Experiences are executing, and the Composer module is working normally. We will continue monitoring the remediation.

We are continuing to monitor for any further issues.

This incident has been resolved.

Between 06:48 and 07:05 UTC, Piano ID API experienced elevated error rates in the US cluster. Requests requiring user authentication, token verification, and user profile retrieval were affected. Other Piano services including Composer and VX subscription management were not impacted. Service has been fully restored and all systems are operating normally. We apologize for any disruption caused during this incident.

This incident has been resolved.

We have found an issue with our priority rules for API interaction on server/edge in Composer. The dev team is currently investigating. Clients may find their list order being changed or unsorted without any user interaction. We will continue updating this incident once we identify the root cause.

Our team has restored the priority rules to their pre-incident state while we continue to investigate the root cause. We advise customers to avoid making changes to priority rules until the investigation is complete. We will provide further updates as more information becomes available.

The issue has been identified and mitigated; we will continue monitoring the remediation.

This incident has been resolved.

RCA report for the incident is available at [**this link**](https://docs.piano.io/wp-content/uploads/2026/04/2025-05-06-RCA-Experience-Priority-Data-Incident.pdf).