Retention Science 장애 기록

Due to changes in the Shopify API we are unable to import any data with the Retention Science/Ecommerce Pro application created for Shopify customers. We are actively working on a solution and will update the status page with more info when it is available. In the meantime, manually exporting users from Shopify to a file and importing into accounts is the most viable workaround. (users, orders, items and other data will be uploaded as soon as the application is restored.) For manual imports, please see: https://help.retentionscience.com/hc/en-us/articles/115008016767-How-to-upload-lists For manually importing data that is not just users, please reach out to support for next steps.

We are continuing to work on a fix for the Shopify import issue. We are working on early versions of the new API calls that were migrated from REST endpoints to GraphQL endpoints, and things seem to be going well so far. It will take some time to finish replacing all endpoints, fix the automated tests, and do thorough manual testing.

We are continuing to work on the broken Shopify imports issue. We have rough replacement code written for all affected API endpoints, but we are in the manual testing phase, which may take some time. Our current estimate is a production deployment by mid next week. Thank you for your patience, and please stay tuned for more updates.

Good news--we've deployed an Alpha version of the fixed code and are currently testing. Will update further once the production testing is completed.

Great news! We have deployed a change that will work for most clients. We are pushing out the change carefully, monitoring for issues. Because we now have a path forward, we are downgrading this status to a "partial outage". Clients who have a very large amount of data may have to wait a bit longer for a fix, as there are some Shopify API limitations for which we may need to find a workaround.

Excellent news-- we've finished rolling out Shopify imports for all clients. We're closing this status as fully operational! We hope you have a wonderful Thanksgiving!

This incident has been resolved.

We are currently investigating, but it seems one of our databases has run out of temporary disk space to unload large tables to our machine learning algorithm. Not all sites are affected, and it seems to primarily be an issue for larger sites (many millions of users). We are looking to remediate this, but also we're trying to find out why this suddenly started happening even though we haven't changed much on the database server side. We will update here with more findings as we have them.

We are continuing to investigate. The issue seems to have started on April 14th when we applied a AWS-required MySQL 5.7 => 8 upgrade to our Subscription service database. This has apparently caused some unforeseen performance issues when running multiple sites' machine learning jobs. We will be upgrading the database instance size in order to sidestep the space issue temporarily. Our hypothesis is that this will buy us some time and (hopefully) allow our big jobs to continue running. Meanwhile, we will be investigating how to make the disk usage more efficient, or resolve the issue overall.

The issue has been identified and a fix is being implemented.

We found a missing configuration that did not carry over from the old version of the database to the new. We added this configuration on Friday around 6pm Pacific, which has fixed the problem with our big data operations on our Subscription service database for all clients. We monitored over the weekend, and there were no further errors. This issue has been resolved. We have taken steps to make sure all of our databases have this configuration and will be monitoring any similar issue with other databases going forward.

Date Started: March 27th Date Resolved: April 11th Cause of Incident: A Migration/Upgrade to MySql v8 per Amazon AWS requirements occured on March 26th. When the migration happened there was a small bug in the code/query around NULL date values. This caused rows in our AI predictions output to go missing. Any row that had a date or timestamp or a 0 or null or date outside the normal range would cause that row to be removed entirely from the output. This was something the new version of sql was sensitive to that the old version of sql was not. How we will prevent this from happening in future: A Database migration is not a frequent occurrence, something that happens once every 4 years so so. We will do a better job in QAing by involving the Client Success team to QA a few of their largest clients to make sure no anomalies are occurring with the stages and engagement.

Large amount of requests from one client caused an 11pm - 1:42am Pacific downtime (almost 3 hours) in various APIs and parts of our site. We have reduced concurrency on some services so we are less likely to see this happen again.

Downtime for members.retentionscience.com - users were unable to access page. Incident has been resolved.

We are currently investigating this issue for immediate repair

This incident has been resolved.

We have identified an issue with our send pipeline causing extremely low throughput, delaying most sends. Only customer accounts using our proprietary SMTP (accounts migrated off of Sendgrid) have been affected. Our team is actively working to restore system performance.

There is a widespread AWS outage affecting several components of Cortex. There may be issues loading pages, setting up emails, and sending campaigns during this time. We are monitoring the issue closely and will notify you when systems are back online.

This incidence has been resolved. All systems are back online and email sends will resume. Previously scheduled sends should be verified and rescheduled if needed.

Campaign performance metrics are temporarily missing for some accounts - our team is aware of the issue and will update with a resolution soon. Email sends are not impacted.

A patch has been put into place and this incident is now resolved. It may take up to 24 hours for the system to update and reflect stats correctly.

We have identified an intermittent issue caused by our click tracking service resulting in some click throughs erroring out and some clicks not properly getting tracked. Sends are NOT impacted. Our team is pushing an update shortly.

This incident has now been resolved.

We are experiencing an issue that is preventing emails from going out for list sizes greater than 5000 recipients. We have identified the issue and are currently working on a solution.

This issue has now been resolved. All backed up emails will begin sending and all remaining emails will go out as planned.

Although there is no longer an issue with emails sending, the outage with our infrastructure provider is causing delays in analytics for sends since this morning. Sends prior to today should reflect accurate reporting, and once the outage is resolved we will backfill reports to reflect data accurately. JavaScript tracking of onsite events is also impacted and thus page views/carts/checkouts are currently missing and will be backfilled once the outage is resolved. Open and click tracking are working as expected. We expect a full recovery to take a few hours and will continue to monitor and update as we learn more.

AWS is experiencing an outage that is impacting much of its North America presence: https://techcrunch.com/2020/11/25/amazon-web-services-outage-takes-a-portion-of-the-internet-down-with-it/ This began impacting ReSci clients as of about 6AM PST. We are seeing delays and incorrect statistics / counts in our analytics dashboards as well as a delay in some morning sends. Our main Cortex dashboard, SMS, data science APIs, events APIs, template tools, and other APIs remain operational as expected. Here is an update on the affected components. Promo Blasts: Operational As of 1PM PST, all promo blasts that were delayed from this morning (from ~6AM PST) have been sent out. Realistic Test Sends: Degraded performance A small percentage of test-sends are timing out. This is resulting from the system delays we experienced this morning. Click tracking: Operational Open Tracking: Degraded performance Some “open” events are missing from analytics and filtering from approximately ~6AM PST until now. JS Events: Degraded performance Some events are missing from analytics and filtering from approximately ~6AM PST until now. Segment sync to FB and Google Degraded performance We are experiencing issues with segments being automatically synced to Google and Facebook.

This incident has been resolved.

We have identified an issue with our infrastructure provider since approximately 5:30AM PST that is causing delays in analytics and causing a subset of sends to experience delays. We are carefully monitoring and scaling to address the immediate issue.

This incident has been resolved.

When attempting to add or update an image in the template builder tool, the user is brought to the File Manager portion of the builder which throws an error. Attempting to retry results in the same error message being displayed. This does not affect email sends or reporting - only the ability to upload images when creating templates.

This incident has been resolved

We are experiencing an outage with our webhooks for the shopify integration specifically for SMS. Shopify webhooks for email are currently functioning as expected. We are working on a fix. This outage only impacts segmentation capabilities based on shopify attributes. It doesn't impact deliverability of automated or one time campaign sends or capturing leads through modals or any other list building flow.

This incident has been resolved.

We are investigating a network connectivity issue that is causing a delay in the send pipeline. Promo blasts scheduled are being processed but may experience delay until this issue has been resolved. We are actively working through this and will update as soon as we can.

This incident has been resolved. The scope of this issue affects Promo blasts that were scheduled between 4-6:30AM PST. Any blasts that may have been delayed have now been queued for sending.

We've identified an issue causing degraded performance for some Cortex filters. This is causing Cortex sends and promo blasts to be delayed for a subset of clients. We are actively working with our database provider to resolve this as soon as possible. Analytics pages may also experience delayed metrics. Our metrics are regenerating and will be fully updated by end of day.

This incident has been resolved.

We've identified an issue causing degraded performance for some Cortex filters. This is causing Cortex sends and promo blasts to be delayed for a subset of clients. We are actively working with our database provider to resolve this as soon as possible. Analytics pages may also experience delayed metrics. Our metrics are regenerating and will be fully updated by end of day.

This incident has been resolved.

Some pages within Cortex are intermittently unavailable, but fortunately sends are not affected. We are investigating the issue and expect to have a fix in place soon.

This issue is now resolved.

We are experiencing intermittent issues with Cortex. Access to Cortex may be limited during this time. We have identified the source issue and expect a fix to be in place momentarily.

This incident has been resolved and full access to Cortex has been restored