39 Sift incidents · October 2018 — official updates, affected components, duration and resolution details.
Sift services are impacted due to an issue experienced by our cloud provider. We apologize for any inconvenience.
The issue has been identified and a fix is being implemented.
We are continuing to work on a fix for this issue.
All Sift services are operational. Thank you for your patience.
Sift Console is experiencing is an outage due to an outage experienced by our data infrastructure provider. We apologize for any inconvenience. https://status.snowflake.com/incidents/x150l0cr8khv
The outage affecting our data cloud infrastructure provider has been resolved, and the Sift console is now fully operational. We appreciate your patience during this incident.
Sift has identified an issue with our API services, beginning at 8:33 am PT. We are actively investigating the issue and will follow up as we learn more.
This incident has been resolved and all services are operational.
Workflow Review Queues were not populated for certain entity types from January 22nd 7:21PM PST - January 23rd 8:51AM PST. This issue has been resolved and Review Queues are being populated again. There was no impact to Score API or Workflow decisions
Sift API service experienced a partial disruption from 3:58pm PST - 4:50pm PST intermittently. Issue has been resolved and all Sift services are back to normal. We apologize for this disruption.
From 5:44 - 5:55pm PT, Sift experienced elevated failure rates in Events API requests as well as issues with Console access. The issues were identified and resolved as of 5:55pm PT.
We are currently investigating the issue and working on bringing back Sift API endpoints as soon as possible.
We have identified a DB upgrade as the root cause of the incident and we are working on restoring it as soon as possible.
We are continuing to work on a fix for this issue.
We have restored the DB and our API endpoints have recovered. We are closely monitoring our services for the full recovery.
This incident has been resolved. We have fully restored our services.
Sift is experiencing a technical issue with its console. We are actively working on investigating and fixing the issue. We are still processing data and returning scores/workflow decisions, but latency may increase.
Our systems are back to normal, but we are continuing to monitor.
This incident has been resolved.
Sift is experiencing increased latency for all endpoints. This is leading to increased error rates. We are actively working on investigating and fixing issue. We are still processing data and returning scores/workflows for events that do not error.
The issue has been resolved. Latency and error rates are back to normal. We are continuing to monitor our systems.
Sift is experiencing a technical issue with its console. We are actively working on investigating and fixing the issue. We are still processing data and returning scores/workflow decisions.
We are continuing to investigate this issue.
Sift Console issue has been resolved and fully operational. Thank you for your patience as we worked through resolving this issue.
Sift has identified an issue with our indexing service as of 7:30am Pacific. We are actively working on restoring the service. This only impacts console indexing - Explore and search, but does not impact any data ingestion or APIs. All score and decision pipelines are unaffected. Links to specific users in the Sift console & Review Queues are unaffected.
Sift is continuing to work on recovering our indexing service.
We are continuing to work on a fix for this issue.
We have resolved the console indexing service issue and now processing the backlog of events since the incident started. We are closely monitoring the service to ensure there are no further incidents.
We have resolved the console indexing service issue and now processing the backlog of events since the incident started. We are closely monitoring the service to ensure there are no further incidents.
We are continuing to monitor for any further issues.
At this time the issue with Console Indexing service has been resolved. At Sift we value product excellence. Our customers should expect nothing less. We will work hard to ensure this and similar incidents cannot be repeated. Thanks for fighting fraud with Sift Science.
Sift has identified an issue which is causing an increase in response latency, mobile event errors, and Async API events beginning at roughly ~8:47AM PST. We are actively investigating the issue and will follow up as we learn more.
We have identified the root cause for this issue and the issue has been resolved. We are monitoring the services closely.
Between 8:06am PDT and 8:40am PDT, Sift was making infrastructure upgrades which caused approximately 60% of API calls to Sift to fail. The issues have been identified and resolved. We are continuing to monitor our systems.
Sift is currently investigating reports of our console being inaccessible. We will update as we learn more.
Sift APIs are now also affected. We are still investigating the issue, but we suspect it is due to issues with our underlying cloud provider.
Our underlying cloud provider is recovering from a networking issue. Sift services are recovering, and we are continuing to monitor until we confirm a full recovery.
The incident has been resolved and Sift services have been fully restored as of 10:09am PST. Our APIs were affected from 9:47 to 10:09am Pacific.
From 3:32-4:16pm Pacific time, Sift experienced increased API failure rates across all Sift APIs. The issue has been identified and resolved. We are continuing to monitor our systems.
We have identified an issue causing increased latency and error rates across our scoring pathways. A fix has been identified and put in place and we continue to monitor system status.
We are continuing to monitor for any further issues.
This incident has been resolved.
Starting at approximately 1:57pm Pacific Sift has experienced increased latencies and failure rates against synchronous and asynchronous scores and workflows. We have identified the issue and are working to resolve it.
As of 2:27pm Pacific, fix has been put in place and error rates and latencies are fully recovered for all scoring pathways. We will continue to monitor.
This incident has been resolved.
Starting at approximately 1pm Pacific today, Sift has identified issues processing score and workflow requests. The issue has been identified and we are working on resolving. No data has been lost.
Synchronous Scores and Workflows are now fully operational. Event ingestion for non synchronous calls is delayed by about 25 minutes.
All scoring is back to normal.
Between 9:29am-9:57am Pacific today, Sift experienced issues processing score and workflow requests. The issue has been resolved and our systems recovered. No data has been lost. We are now processing the backlog of asynchronous requests, we expect this to finish at 10:50am Pacific.
From 1:41pm-2:10pm Pacific, Sift saw an increased failure rate in our synchronous workflow requests and synchronous scoring. The issue has been identified and resolved as of 2:25pm. Additionally, we are experiencing a delay in ingesting asynchronous events that we received during the impacted time period. We are draining the backlog. No data was lost, and the Score API was unaffected.