51 Searchspring incidents · luty 2022 — official updates, affected components, duration and resolution details.
Obecnie badamy tę kwestię.
Widzimy przerywany problem w niektórych regionach, gdzie strona PLP render dostaje błąd z powodu błędu składni JSX
Ten incydent został rozwiązany.
Automatyczne tłumaczenie oficjalnej aktualizacji incydentu.
We are seeing a high indexing queue in multiple regions
We are curently investigating the problem
The issue has been identified, we are currently working on fixing the issue
A fix has been implemented and we are monitoring the results.
We are continuing to monitor for any further issues.
This incident has been resolved.
We experienced elevated error rates and intermittent performance degradation affecting search traffic due to a combination of infrastructure strain and unexpected automated traffic patterns. The issue primarily impacted search functionality and resulted in multiple brief spikes in server errors.
Systems are currently stable. Error rates have returned to normal levels, and search performance is operating within expected thresholds. We are continuing to monitor closely and further analyze traffic patterns to prevent recurrence.
This issue has been resolved. All systems are operating normally, and error rates and performance have returned to expected levels.
Were seeing an increased error rate on search api requests in the Australia region.
Operations have returned to normal, we are continuing to monitor the situation.
This incident has been resolved.
A subset of BigCommerce customers experienced product listing pages failing to load results due to a template change on the BigCommerce platform (we estimate). The change caused Searchspring’s Snap integration to misinterpret the “Search Results” page as a category, applying an invalid background filter that returned no products. The issue was intermittent and did not affect Searchspring’s core search service. Functionality was restored within minutes, likely following a rollback of the update by BigCommerce.
We have identified an incident that our Search API is currently down, post AWS outage. The team is currently investigating, and identified a potential problem.
We have isolated the problem to be fully related to AWS outages. Indexing, Search functionality and also management console are impacted.
In order to protect our system and make sure we get services back as quickly as possible once AWS outage is recovered, we have to put our Management Console into maintenance mode. Users are blocked from making any changes as of now. Thanks for your understanding during this difficult time!
Updating impact: we have confirmed the impact is confined to the same region (us-east) as AWS has identified. Traffic on us-west and other regions are still fine.
Part of the services are brought back to normal level. We are still waiting on another AWS-dependent vendor to come back to normal before we take Management Console back)
Search API is now back to normal level - we are monitoring
We are monitoring the results for Search, Autocomplete, while waiting for AWS fix to propagate
We continue to observe some instability on AWS side as well as our vendor who's hosted on AWS. Team is still working on restoring the services, while monitoring the restoring effort of AWS.
Search and Autocomplete is being recovered. Team is working on a fix to bring Indexing and Management back.
Search API/Autocomplete API are fully recovered. Engineering team is still working on fixes to bring Indexing service back.
Our engineering team has taken Management Console out of maintenance mode. We are still working on bringing indexing service back.
Management Console is back to Operational
Reporting in SMC is currently experiencing issues due to the same ongoing AWS outage. We are working on bringing it back.
We are seeing Search/Autocomplete fully recovered. SMC is mostly recovered, experiencing intermittent issue with reporting (not being able to render reports properly). Our Engineering team is still actively working on restoring our indexing services.
Our Engineering team has brought Indexing Service back. We are still seeing reporting panel fail to render, we are working with our backend team and vendor to bring it back. We will be updating the individual service status in the next couple hours as services stabilize.
SMC and reporting panel is fully operational. Search, Indexing, and autocomplete fully back to normal state. We will be monitoring in the next 24 hours before we close this incident.
Systems are back to normal. Team are doing clean ups, expected to mark the incident to "Resolved" by the end of today EST.
We have the incident fully resolved, all systems are stabilized.
AWS is currently having a major outage, this is impacting our indexing services https://health.aws.amazon.com/health/status
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We are currently experiencing a high volume of 503 Service Unavailable errors impacting the search experience across a small subset of sites. This may result in degraded performance or temporary unavailability of search functionality for some users. Our team has identified the root cause and are working to resolve this issue. We will provide another update in 30 minutes or sooner.
A fix has been implemented, and we are currently monitoring system performance to ensure the issue has been fully resolved.
The issue has been fully resolved, and all affected services are now operating as expected.
We are currently investigating an indexing issue for customers on the new Shopify connector. Our teams are actively working on a resolution and will provide an update within the next 30 minutes. We appreciate your patience.
The issue has been identified and our teams are working on implementing a fix. We’ll provide more updates as soon as we have additional information. We will provide another update in 30 minutes or less.
A fix has been implemented and we are monitoring the results.
We are continuing to monitor for any further issues.
This incident has been resolved.
We are currently investigating an issue preventing users from accessing the Searchspring Management Console (SMC). Our teams are actively working on a resolution and will provide an update within the next 30 minutes. We appreciate your patience.
We are continuing to investigate this issue.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We are currently experiencing an issue impacting a subset of Shopify customers, where products may be missing from collections. Our team has identified the issue and our teams are working on resolving the issue as quickly as possible. We will provide another update in 30 minutes or less.
Our team is still working to resolve the issue for all impacted customers, and we appreciate your patience. We will provide another update in 30 minutes or less.
Work continues on the issue and we will update you again in another 30 minutes.
We continue to work on a fix to resolve the issue. We will update you again in approximately 30 minutes.
We’ve identified the issue and implemented a workaround while we await a technical response from a third-party provider. We’ll keep you informed and provide an update as soon as we have more information.
We are still waiting on a technical response from a third-party provider. While we await their response Live Indexing is disable, however normal indexing functionality was restored with our previous status. Our next update will be once we've received a response from the third-party provider.
We are continuing to work on a fix for this issue.
Shopify has acknowledged the issue on their end is working towards a solution. They have not yet provided us with a timeline. Once they have implemented a fix, we will be able to evaluate if Live Indexing can be re-enabled for customers who were previously running it.
Shopify has begun making changes to resolve the API issue which started last week. As a result of those changes, we are currently experiencing issues with data collection from their platform. Our team is currently investigating the root cause of the issue. We will provide another update within the next hour.
We are currently awaiting an update from Shopify regarding API changes that they've made. Search is operational, however we are not receiving properly formatted Collection data from Shopify. Additionally, product data may be stale. We will provide additional information as it becomes available to us.
It appears that Shopify has rolled back the undocumented API changes that they made overnight, as we are now receiving data and collections have been restored. As we have not yet received a confirmation or update from Shopify, we are continuing to monitor the situation. Once we are confident that additional changes aren't being made by Shopify, we will mark this issue as resolved.
We have been monitoring the data that we are receiving from Shopify and are now satisfied that the undocumented API changes they made have been reverted. For customers who we had to temporarily disable live indexing, this functionality has been fully restored on your sites. We appreciate your patience during this time, and we’ll continue to keep a close eye on the situation to ensure consistent performance.
We are currently investigating an issue that is impacting the data structure of some BigCommerce customers. We believe we have found the source of the issue already and are currently working on resolving it.
We identified the issue as being a change in the default delimiter for handing BigCommerce data feeds. This impacted our ability to read the data feed structure for BigCommerce customers. We have corrected the issue and are re-indexing customer's who encountered the error, once the index is complete the problem should be resolved.
The source of the problem has been addressed and all customers who were impacted have either already been re-indexed, or are in the process of indexing. As those indexes complete, your site will function as expected.
We have detected an issue impacting a subset of Shopify clients, where more products than expected are being excluded. This is due to a recent change on Shopify’s end. Our engineering team is actively investigating the issue and working towards a resolution. We will provide an update within the next 30 minutes.
The incident has been resolved and all affected sites are back to normal product exclusion numbers.
We are currently investigating an issue causing a slowdown in indexing. You can expect an update in 30 minutes or less once we have more information.
We have identitied and resolved the issue causing the degraded state of our indexes. The indexing queue is returning to normal.
We are currently seeing 504 errors across all autocomplete and search requests, preventing search results from being returned. Our team is actively investigating and will provide an update in 30 minutes or sooner as more information becomes available.
We have identified and resolved the issue affecting our systems. Recovery is currently in progress. Our team will continue to monitor the situation closely until the systems have fully recovered. We will provide an update within the next 30 minutes or sooner with more information.
Systems have recovered and are back to operational in all areas. We are closing out this incident thank you for your patience during this time.
We are currently investigating a degraded state of indexing where some sites are not completing successfully. We will update you here in 30 minutes
We have identified and resolved the issue. Indexing has returned to a normal state.
We are currently investigating an issue affecting the AU region. An issue was identified with the clusters in this region, causing search calls to return errors. We have successfully diverted traffic to a failover cluster, and search functionality has been restored. Our team is continuing to investigate ongoing indexing issues in the AU region.
We are continuing to investigate this issue.
We have identified the issue and are in the process of taking corrective actions to resolve it. We will post another update in approximately 30 minutes.
Indexing for the AU region is currently paused as we implement corrective measures. The systems are in the process of recovering, and we are working diligently to restore full functionality. We will post an update in approximately 60 minutes.
Indexing remains paused for the AU region as we continue to work on implementing corrective measures. We will post an update in approximately 60 minutes
All previously impacted services have now recovered and the issue has been cleared. Our team is still monitoring the situation as we work to migrate all traffic back to it's original regions.
We are currently experiencing an elevated number of errors related to our search API. This will impact some traffic in the North America region. We are currently investigating and will provide regular updates in 30 minutes intervals or sooner.
We have identified and resolved the source of the issue. We are now monitoring the situation the ensure normal performance continues.
The issue has now been resolved.
Our Searchspring Management Console (SMC) is experiencing an issue due to a 3rd party vendor we use. They have identified an outage cause and are working to resolve the issue. Searchspring users may see missing logos, buttons, and other assets. We will continue to monitor their status page and update you here as soon as we know more.
The issue with the 3rd party vendor has been resolved and the SMC is back and fully operational.
This incident has been resolved.
We are currently experiencing intermittent slowdowns in our bulk indexing process. While sites are still completing, you may notice that it is taking longer than usual. Our engineering team is actively investigating the issue to identify the root cause. We are prioritizing this matter and will provide updates as soon as we have more information.
The engineering team has advised that we are still working on the delays in bulk indexing. Slow down and delays are still being experienced intermittently by some clients. We will update you as soon as we have more information to offer.
The intermittent slow-downs have now been resolved.