Störungsverlauf von Shelfio

Between 23 Jun 12:00 UTC and 25 Jun 19:00 UTC, we experienced intermittent errors in core resource fetching. This caused a small percentage of requests to fail, which in some cases resulted in users seeing a white screen instead of the app. The incident affected a small number of customers globally. Due to a configuration issue, impact may have been higher for some customers than others. The issue has been resolved, and service has returned to normal. We apologize for the disruption.

Some customer Self-Service Portal homepages were showing other customers' homepage content. Other pages (search, article, product pages) remained functional. Workaround: avoid the homepage. Append a language code, or add a query character such as ?, to force a fresh render.

Root cause narrowed to CDN cache contamination triggered by a configuration rollback. The contamination affected homepage edge cache only.

Self-Service Portal homepages are now returning correct content after cache purges. We are monitoring edge cache consistency and user reports for recurrence.

Duration: ~2h55m. Cause: CDN cache contamination of homepage caches. Impact limited to US and EU Self-service portal homepages. All systems normal.

We are currently investigating this issue.

We are continuing to investigate this issue.

The issue has been identified and a fix is being implemented.

A fix has been implemented and we are monitoring the results.

We're still observing occasional timeouts in Agent Assist searches for certain real-time suggestions.

Reopened. Our earlier resolution was premature. Our cloud infrastructure provider continues to report widespread degradation in their US‑East region (about 80 services affected, several in a brownout state). This is causing intermittent errors, elevated latency, and timeouts across multiple Shelf services in the US region, Answer Assist and Search, and integrations. EU and CA regions remain healthy. We are working with the provider and have applied mitigations to reduce impact. Next update within 30 minutes.

Update - This remains a brownout, not a full outage. Most requests in the US region are succeeding, but ongoing upstream degradation is causing intermittent errors and latency. Authentication, Viewing Content & Search, Self‑Service Portals, Emails & Notifications, and Making Changes to Content are largely available; Integrations and Insights & Analytics are the most affected with higher delays/timeouts. We continue to mitigate and monitor.

Update - Error rates are trending down. Over the last few minutes the US region is averaging about 0.4% 5xx responses. Core paths (Authentication, Viewing Content & Search, Self‑Service, and content edits) are each ≤0.5%, while Integrations and Insights & Analytics fluctuate slightly higher but remain under 1%. We'll keep mitigations in place and continue to monitor.

In the last three hours we observed a single brief spike in 5xx responses around 20:45 UTC in the US region; since then, 5xx has held at 0% across monitored endpoints. For transparency, we'll keep the incident open in Monitoring with no components marked as affected through the end of the day while we continue to observe.

This incident has been resolved.

We are investigating reports of sign‑in failures in our US region. The first errors began at approximately 06:48 UTC. EU and CA regions remain healthy. We'll share more as we learn. Next update within 30 minutes.

We have identified that the sign‑in failures correlate with a regional incident at our cloud infrastructure provider in their US‑East region. Customers served from our US region may see "incorrect password" messages or slow/failed requests. Our EU and CA regions remain healthy.

The issue persists. Our cloud infrastructure provider continues to experience a region-wide incident in their US‑East region. We are actively working with the provider to reduce impact.

US region services have returned to normal operation as of 09:22 UTC. This incident stemmed from a region‑wide issue at our cloud infrastructure provider and caused intermittent errors and latency across multiple US services. Impact window: 06:48-09:22 UTC (2 hours 34 minutes). No data was lost. EU and CA regions were not affected. Thank you for your patience.

This incident has been resolved.

We are currently investigating increased error rates affecting search and content discovery in our US region. Some users may experience slower response times or intermittent errors when searching for content, viewing announcements, or submitting feedback. Other platform capabilities, including content creation and editing, remain fully operational.

We have identified and addressed the underlying issue affecting search and content discovery performance in the US region. Our engineering team has implemented corrective measures and is actively monitoring system behavior. During peak impact, approximately 20-40% of search and content discovery requests experienced errors. We continue to observe system performance to ensure full stability.

This incident has been resolved. All search, announcement, and feedback services in the US region have returned to normal operation. System monitoring confirms stable performance with no residual errors. We apologize for any disruption this may have caused.

On October 17, 2025, between 15:22 and 16:03 UTC, users in our US region experienced intermittent errors when searching for content, viewing announcements, and submitting feedback. At the peak of the incident, approximately 40% of requests to these services returned errors, while other platform capabilities remained fully operational. This event occurred during an ongoing maintenance process to upgrade our search relevance engine - work aimed at delivering more accurate, higher‑quality results to users. We ran this upgrade path in the background for several days at minimal load to avoid disruption. Near the end of the week, a configuration error caused the process to place more load on the system than intended. Coupled with a spike in search activity, this created temporary capacity pressure and intermittent errors in the US region. The outage was partial: error rates varied over the window and peaked around 40%, with the majority of requests continuing to succeed. We've already applied lessons from a prior search incident and rolled out the upgraded search infrastructure in the EU and CA regions; the US region is scheduled next. We'll continue with this upgrade, and will better communicate maintenance windows in advance, align change timing to lower‑traffic periods, and add redundancy specific to search operations to further protect performance during these necessary maintenance activities.

Starting at 16:30 UTC, an infrastructure issue caused elevated 5xx errors for application and API requests. We have identified the cause and are rolling out a fix. Impact affects a subset of customers across multiple regions.

We are seeing recovery in the CA and EU regions, and a fix is being rolled out to the remaining regions. A subset of customers outside those regions may still be impacted.

We are continuing to work on a fix for this issue.

Search has recovered across regions. Feedback and CPW remain degraded for a subset of customers while the fix completes rollout. Next update within 30 minutes.

We’ve deployed a fix and are seeing recovery for application and API requests across regions. If you encountered errors, please try again or refresh your browser.

All services have recovered. Error rates and latency are back to normal across all regions. We’ll continue to monitor closely for the next hour. If you still encounter issues, please refresh your browser and try again.

This incident has been resolved.

Starting at 14:38 UTC, we observed a spike in SSO authentication failures. Affected users may be unable to sign in and see "access_denied (Unauthorized)" after returning from their IdP. We are actively investigating with our identity provider. Next update within 30 minutes.

We have deployed a fix to our authentication flow at 16:15 UTC. Users should begin to see successful sign-ins resume across all regions. Please retry logging in; starting a new browser session (or incognito) may help clear a failed attempt. We’re closely monitoring error rates and will provide another update within 30 minutes.

SSO sign-ins have been restored. Between 14:38-16:27 UTC a configuration regression in our authentication layer caused the authorization code exchange to fail, resulting in "access_denied (Unauthorized)" after IdP login. We corrected the configuration and refreshed services; metrics show recovery across all regions.

We are investigating an issue where users could experience intermittent errors and delays when searching or viewing content; content creation and edits remain available.

We have identified a performance issue and are working to restore normal search and content viewing.

Mitigation is in place and performance is improving for search and content viewing; we will provide another update by 18:56 UTC.

The issue affecting search and content viewing has been resolved and service is operating normally.

Between 18:26 - 18:56 UTC in the US region, approximately 20% of search and content viewing requests failed outright. The remaining successful requests experienced elevated latency, with p99 around 17 seconds. Content creation and edits were not affected.

We're currently experiencing an issue with a third-party service provider that may impact some users' experience. Our team is actively working with the provider to resolve the situation as quickly as possible.

We are continuing to investigate this issue.

We have identified that the issue is limited to gem listing updates. Specifically, some gems may continue to appear in their previous location after being moved or archived. Our team is actively working on a fix to ensure updates are reflected correctly. We will share another update as soon as we have progress to report. Thank you for your understanding.

Core issue is isolated, we are working on the fix

Our Team is actively working on a resolution. We will provide further updates as soon as more information becomes available. Thank you for your patience.

We have begun applying fixes to address the issue. While progress has been made, the issue is not yet fully resolved. Our team continues to work toward a complete resolution, and we will share updates as they become available.

Some functionalities have been successfully restored. We are continuing to work on bringing full service back online and will provide updates as progress continues.

The issue has been fully resolved and all functionalities are now restored. Thank you for your patience throughout this incident.

The issue has been identified and a fix is being implemented.

This incident has been resolved.

On 27 June 2025, between 14:40 UTC and 14:51 UTC, some users in the US region were unable to retrieve search results after selecting a library. Other areas of the platform continued to operate normally.

We are investigating reports of users being unable to view Gem content.

We have identified the root cause related to a deployment configuration and are deploying a correction.

The configuration issue has been corrected, and Gem content access via KMS is restored.

Between approximately 15:37 UTC and 16:03 UTC on April 29, 2025, some users experienced an issue where content pages \('Gems'\) would not load when accessed directly from the main Shelf dashboard or search results. This temporary issue stemmed from the premature activation of a configuration change during preparations for our planned user interface update, scheduled for April 30th. **Impact & Workaround:** During this time, content remained accessible via the Gem Preview feature. Other platform areas like Agent Assist, Self-Service Portals, and Content Intelligence were unaffected. **Resolution:** The issue was identified and fully resolved by 16:03 UTC. All systems are now operating normally. We apologize for any inconvenience caused.

We are currently experiencing a disruption to our Content Intelligence service in the US region that began at 17:20 UTC. Users may be unable to view analytics dashboards and charts. All data remains secure, and any changes made during this period will be properly processed once service is fully restored. Our engineering team has identified the root cause as an underlying infrastructure issue with our service provider and is actively implementing mitigation measures, including routing operations to redundant systems. We are in direct communication with AWS support engineers via a priority incident channel and collaboratively working toward resolution. This issue affects only the US region; EU and CA regions remain fully operational. We will provide updates as the situation progresses.

The Content Intelligence service in the US region has been fully restored as of 20:00 UTC. Our engineering team successfully implemented a failover solution using a recent system backup taken moments before the disruption. All user data is intact and operations performed during the outage have been properly stored. Some users may experience brief periods of reduced performance as our systems complete the scaling process. We continue to monitor the service closely during this stabilization period and appreciate your patience throughout this event.

The disruption to Content Intelligence service in the US region has been fully resolved. Our engineering team successfully restored all services to their normal state. The situation has been closely monitored, and performance metrics have stabilized without further issues. We appreciate your patience and understanding during this time.

We are experiencing a service disruption affecting our search-related functionality across multiple features including Gems search, Feedback list, and Announcement list. Our engineering team has been notified and is actively working to restore normal operations. While basic platform functionality remains available, search capabilities are currently limited. We will provide updates as the situation develops.

Our engineering team has identified the source of the disruption affecting search functionality. We are actively implementing necessary adjustments to restore service. Search capabilities remain limited across affected features while we work on the resolution. We will provide another update once service is restored.

We have successfully restored search functionality for Announcements, Feedback lists, and CPW tasks. Gems search remains temporarily affected while we complete the final recovery steps. Our engineering team continues working on full service restoration. We will provide another update once all features are back online.

Search functionality has been restored across all major features including Gems search. The system is now operating normally and we continue to monitor for stability. We appreciate your patience during this disruption.

## What Happened? On October 24, 2024, customers reported difficulties using Shelf's search functionality. Our monitoring systems detected performance degradation in the search service, affecting users' ability to find and access their content. The issue was first identified through our automated monitoring systems and confirmed by customer reports. ## Why Did It Happen? The search service experienced capacity constraints due to an unexpected pattern in request handling. When users performed searches, our system is designed to make a single attempt to retrieve results. However, a recent configuration change inadvertently allowed multiple retry attempts for each search request. This meant that a single user search could generate several duplicate requests, creating unnecessary load on the system. Under normal conditions, our search infrastructure efficiently handles the typical volume of search requests. In this case, the combination of increased search traffic and the multiplicative effect of retry attempts exceeded the system's designed capacity. This overload caused the search service to respond slowly or fail to respond at all. ## Impact Users in the US region experienced difficulties performing searches across the platform for approximately one hour. While other platform features remained functional, any operations requiring search capabilities were affected. Users could access their content directly through navigation, but could not search for specific items or use search-dependent features. The EU and Canada regions were not impacted by this incident. The exact number of affected users during this period cannot be precisely determined, but the issue was limited to users actively attempting to use search functionality in the US region between 19:28 UTC and 20:21 UTC. ## Solution Our team implemented an immediate fix by adjusting the request handling configuration to prevent excessive retries. We also increased the search service's capacity to ensure stable operation. After implementing these changes, we conducted comprehensive testing across all affected components and monitored the system for an extended period to confirm the solution's effectiveness.

We are aware of an issue where a recent update to Google Chrome is causing errors within the user interface of our platform. Our team is actively investigating the matter to identify the root cause and implement a fix. We will provide further updates as soon as more information is available. Thank you for your patience and understanding.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

We have identified an issue with the task publishing process and are actively working to resolve it.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

The issue has been identified and a fix is being implemented.

We are continuing to work on a fix for this issue.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

The issue has been identified and a fix is being implemented.

We are continuing to work on a fix for this issue.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

The issue has been identified and a fix is being implemented.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

We are currently experiencing an unexpected outage with our search functionality that may affect your ability to locate content swiftly across our platform. Our engineering team has been notified and is actively working on diagnosing and resolving the issue as promptly as possible. We apologize for any inconvenience this may cause and appreciate your patience. Please stay tuned to this status page for real-time updates and estimated resolution times as we work to restore full service.

We're pleased to report that the outage affecting our search capabilities has been resolved. Our team addressed a performance issue with our search cluster under heavy load. We've monitored the service and confirmed it is functioning normally. Thank you for your patience during this time. Happy searching!