Ravelin outage history

Revolut issued cards are failing 3DS authentications. There appears to be an issue on the Revolut platform. You can track the status of Revolut here: [https://www.revolut.com/system-status/](https://www.revolut.com/system-status/ "https://www.revolut.com/system-status/") Cards from other issuers are being processed normally. We will continue to monitor the situation carefully. We are sorry for any inconvenience.

The source of impacted 3DS authentications is an issue on the Google Cloud Platform in region Europe West 2. [https://status.cloud.google.com/incidents/Xs2XTHzAVC9YnafnPyzj](https://status.cloud.google.com/incidents/Xs2XTHzAVC9YnafnPyzj "https://status.cloud.google.com/incidents/Xs2XTHzAVC9YnafnPyzj") We have noticed the following Issuers' ACS have been impacted. • 3DS_LOA_ACS_RELT_020301_01127 (Revolut) • 3DS_LOA_ACS_OPES_020301_00996 (Openway Europe S.A) We will continue to monitor the situation.

We are no longer seeing a high level of failed authentications for Revolut cards. However the Revolut status page and Google Cloud Platform are still showing the incident as ongoing. We are continuing to monitor the platform carefully. We are sorry for any inconvenience.

The incident has been now been resolved. We are sorry for any inconvenience.

We are aware that some users are receiving errors when accessing the Ravelin dashboard. We are currently investigating the cause and will post updates here as we know more.

We rolled back the changes which were causing the dashboard errors, and saw the errors stop at approximately 10:45 UTC. We are continuing to investigate the root cause of the issue.

The dashboard is operational again, errors have stopped. We are continuing to monitor for any further dashboard errors and continuing to investigate the root cause.

The previous incident has been now been resolved. We are sorry for any inconvenience.

We are experiencing increased error rates from our 3DS API. US based traffic is being rerouted to our EU cluster for processing, so you may experience elevated response times. We are currently in the process of repairing our US cluster. We are investigating to discover the cause. Please check back here for further updates.

``` Our US cluster has recovered and is processing traffic normally again. Google have identified the issued described below: Virtual Private Cloud (VPC) - VM to VM communication - VM to VM communication over internal IPs In the following locations: - asia-east1 (active) - us-central1 (active) This impacted the connectivity of our components running the us-central1 region. We shall continue to monitor the situation. ```

Google have now posted details of the on-going incident here: [https://status.cloud.google.com/incidents/J5ia5t9p3g9Q5Wi7r8Ev](https://status.cloud.google.com/incidents/J5ia5t9p3g9Q5Wi7r8Ev "https://status.cloud.google.com/incidents/J5ia5t9p3g9Q5Wi7r8Ev") We will continue to monitor the 3DS API. We are sorry for any inconvenience.

The previous incident has been now been resolved. We are sorry for any inconvenience.

We are seeing a small number of errors periodically due to an issue in our aggregations service. We are currently investigating.

We have identified the issue and have mitigations in place such that the api should no longer be returning errors. We are continuing to monitor and address the root cause.

We have stopped returning 500s. We are monitoring the situation.

The incident has been resolved.

An issue regarding datalist configuration caused an increase in errors returned from the PSP API. This has now been resolved and we are monitoring.

API performance has returned to expected performance levels

Between 14:46 and 15:47 (UTC), some clients experienced 500 errors when accessing the Explore Customers and Orders features in the Dashboard. This was caused by a code deployment that inadvertently removed functionality required for filters in Explore. We have rolled back the problematic deployment and confirmed that the Explore feature is now functioning normally. All services have been restored to their previous stable state.

The dashboard is now fully operational.

An issue with a database update caused a subset of requests to return a 500 error.

This is now resolved

We're seeing elevated scoring response times at the moment. We are investigating as a priority

Response times have returned to normal. We are continuing to investigate and are monitoring the situation.

Our system is fully operational

We are seeing some elevated latency on the API. We have identified the issue and are looking to remedy it ASAP.

Response times have returned to normal levels.

The issue is resolved

We are experiencing a partial API outage. We are investigating and will provide updates when we have more information.

It appears GCP, where Ravelin is hosted, is experiencing an outage. This is also affecting the Ravelin dashboard and the PCI and 3DS API. We are continuing to investigate.

This appears to be a widespread issue with GCP, our cloud provider. We are continuing to look into it.

We are seeing the system start to recover, but are still seeing elevated response times. We expect this to continue as we retry failed requests. We are continuing to monitor and investigate.

Our system appears to have recovered. Error responses have stopped and response times have returned to normal. We are continuing to monitor the situation.

Our system is fully operational.

We are seeing elevated latency on the core Ravelin API. We have identified the issue and you should see latency coming down. We apologise for the issue.

Response times have recovered and we are monitoring the situation.

Latency has returned to normal levels.

One of our server nodes in our Kubernetes cluster was running with high CPU for all the pods provisioned on that node. We are unsure the cause at the moment (could be a noisy neighbour, could be hardware issues, etc) but we have removed that node which solved the issue.

Our API was generating 500 errors for about six minutes due to a delay in configuration between services

We are seeing an increase in error responses from our API. We are currently investigating.

We have identified the source of the issue and have taken steps to reduce the errors. Errors rates have now dropped. We are continuing to monitor the situation.

The API outage has been resolved, and all services are now fully operational.

We are now seeing issues accessing the Ravelin dashboard. We are also seeing a slight increase in API response times. We are not sure if this is related to the previous issue but are investigating.

We have made some changes which we believe have addressed the issues. We are continuing to monitor the situation.

We have not seen any further issues since our previous update. The dashboard and API are both operating normally.

We are experiencing errors on the fraud API. The issue has been identified and is being fixed now.

Things have recovered after a roll back of a deploy. We are monitoring the situation but expect no more errors at this point.

This is now resolved.

We're investigating an issue with connections to Ravelin's dashboard.

We seem to be getting some connections to the dashboard but still investigating the issue.

The issue has been identified as an increased load on the database due to an error case in a recent deploy. This has been mitigated and we are monitoring the issue

The incident is now resolved

We are currently investigating an issue with the thresholds displayed in the threshold configuration section of the dashboard. This was due to a recent config migration. We do not believe this is having any effect on the thresholds used for scoring. Please do not make any threshold changes via the dashboard based on the current values. We are currently investigating and working on a fix.

We have identified the problem and are deploying a fix.

Our fix has been deployed and the correct thresholds should now be displayed in the dashboard. We are continuing to monitor the situation. When rolling out the threshold config change we did also see a temporary issue with scoring requests between 08:20 and 08:25 UTC, when a small number of requests failed with a 500 response code. We will also be investigating this as part of this incident.

The issue has been resolved. We will continue to investigate the cause for this issue and provide an update on any further findings.

We are currently investigating issues loading the rules page on the dashboard.

We are continuing to monitor the situation but the affected component has recovered

This incident is resolved. It was an issue with queries not progressing within Bigquery. Bigquery has since recovered

We are seeing elevated errors on the API. We are investigating

This has settled down. It was caused by a short spike in latency on our primary database, we are continuing to investigate.

This incident has resolved. We are continuing to monitor and investigate isolated cases of increased latency.

We're seeing timeouts loading data from our primary database to view in the dashboard. We've scaled this up a little and are monitoring the situation. Although the same database is used for scoring that has not been effected so far

We continue to monitor the timeouts related to loading data from our primary database. Our primary and secondary databases have been swapped. That caused a spike in response times but those are now improving.

We're continuing to see intermittent performance issues with our main database. We've now seen it affect our main API as well as the dashboard. The overall impact remains small. We've scaled up the database server capacity and are monitoring the situation

This was resolved late last week

Searching in the dashboard will occasionally timeout or be slower than expected. We are currently investigating.

We are aware of a slow down in our search database. We are attempting to scale further nodes and investigate root cause.

We are still investigating and managing node scaling on our search database. We will update you further as things progress.

We have done some mitigation that will improve search latency for the dashboard but data will be delayed. We are continuing to work on the core database issue.

Mitigation is still ongoing. Search times have improved and index latency is improving. Not recovered yet but going in the right direction.

Index time has gone back to nominal latency. We are currently draining queues of backfilled data. Expect to have search data up to date shortly

We've fixed the elastic search database cluster issues, so we are no longer experiencing any errors or timeouts. However the cluster is not running completely up to date yet so there will be stale data returned on searches. We're running through the queues to fix that, and expect them to finish in ~2hrs by 5pm GMT

All data has now been updated. Searching within the dashboard should be up to date and running as normal. Thanks for your patience