Nacelle outage history

Duration: 3/10 from ~9:00 am - 10:00 am PT (~1 hour) Issue: Starting at ~9:00 am PT on 3/10, some merchant builds failed due to 500 responses when fetching navigation data from the Storefront API. The navigation data service had encountered scaling issues that resulted in a lack of enough replicas being available for the volume of requests. Solution: By ~10:00 am PT, existing self-healing services had allowed navigation data service to stabilize and handle all incoming traffic. As a long-term solution, the number of minimum replicas enabled for the navigation data service has been doubled, and additional monitoring alerts have been implemented for this service.

Nacelle experienced an outage to its Storefront API that impacted all merchants between 3:26 PM PT and 3:48 PM PT. This outage is currently under further investigation.

Between 6:26 PM ET and 7:21 PM ET, Nacelle experienced a spike of new requests that required Nacelle’s DevOps team to scale additional Kubernetes pods. While these new pods were scaling up, the API experienced degraded performance (e.g., long latencies). The root cause of the issue was how Nacelle's custom schema fetching works across new additions to our GraphQL interface. To ensure stability over Black Friday / Cyber Monday, the Nacelle team has removed features still in development from the Storefront API schema service.

We're currently investigating an outage related to collection reindexing. As we continue to investigate, we recommend _not_ reindexing to ensure missing collection data does not impact your storefront.

We've narrowed this issue down to a handful of spaces. Further investigation is ongoing. We've downgraded the impact of this incident based on these findings.

This incident has been resolved.

We're currently experiencing intermittent failures for Shopify webhook product/update events. This is only impacting merchants who are using the 2023-01 version of the Shopify API. We're currently investigating and will provide updates as soon as possible. As we work to remedy this issue, please use manual space reindexes to solve any data consistency problems.

This issue is now resolved. Webhook updates are now coming through as expected.

We are investigating an issue impacting Contentful webhooks. Users looking to index data from Contentful should manual re-index within the Nacelle dashboard.

This incident has been resolved.

We're currently investigating an issue with V1 data ingestion. More information will be shared as soon as possible.

The issue has been identified and a fix is being implemented.

This incident has been resolved.

From 5:40pm to 10:25pm PDT Nacelle was experiencing issues related to Storefront API “content” queries on the v2 platform. This could have had an effect on builds during this window and/or client side content calls.