Störungsverlauf von Ravelin

We are seeing a small number of errors periodically due to an issue in our aggregations service. We are currently investigating.

We have identified the issue and have mitigations in place such that the api should no longer be returning errors. We are continuing to monitor and address the root cause.

We have stopped returning 500s. We are monitoring the situation.

The incident has been resolved.

An issue regarding datalist configuration caused an increase in errors returned from the PSP API. This has now been resolved and we are monitoring.

API performance has returned to expected performance levels

Between 14:46 and 15:47 (UTC), some clients experienced 500 errors when accessing the Explore Customers and Orders features in the Dashboard. This was caused by a code deployment that inadvertently removed functionality required for filters in Explore. We have rolled back the problematic deployment and confirmed that the Explore feature is now functioning normally. All services have been restored to their previous stable state.

The dashboard is now fully operational.

This is now resolved

An issue with a database update caused a subset of requests to return a 500 error.

We're seeing elevated scoring response times at the moment. We are investigating as a priority

Response times have returned to normal. We are continuing to investigate and are monitoring the situation.

Our system is fully operational

We are seeing some elevated latency on the API. We have identified the issue and are looking to remedy it ASAP.

Response times have returned to normal levels.

The issue is resolved

We are experiencing a partial API outage. We are investigating and will provide updates when we have more information.

It appears GCP, where Ravelin is hosted, is experiencing an outage. This is also affecting the Ravelin dashboard and the PCI and 3DS API. We are continuing to investigate.

This appears to be a widespread issue with GCP, our cloud provider. We are continuing to look into it.

We are seeing the system start to recover, but are still seeing elevated response times. We expect this to continue as we retry failed requests. We are continuing to monitor and investigate.

Our system appears to have recovered. Error responses have stopped and response times have returned to normal. We are continuing to monitor the situation.

Our system is fully operational.

We are seeing elevated latency on the core Ravelin API. We have identified the issue and you should see latency coming down. We apologise for the issue.

Response times have recovered and we are monitoring the situation.

Latency has returned to normal levels.

One of our server nodes in our Kubernetes cluster was running with high CPU for all the pods provisioned on that node. We are unsure the cause at the moment (could be a noisy neighbour, could be hardware issues, etc) but we have removed that node which solved the issue.

Our API was generating 500 errors for about six minutes due to a delay in configuration between services

We are seeing an increase in error responses from our API. We are currently investigating.

We have identified the source of the issue and have taken steps to reduce the errors. Errors rates have now dropped. We are continuing to monitor the situation.

The API outage has been resolved, and all services are now fully operational.

We are now seeing issues accessing the Ravelin dashboard. We are also seeing a slight increase in API response times. We are not sure if this is related to the previous issue but are investigating.

We have made some changes which we believe have addressed the issues. We are continuing to monitor the situation.

We have not seen any further issues since our previous update. The dashboard and API are both operating normally.

We are experiencing errors on the fraud API. The issue has been identified and is being fixed now.

Things have recovered after a roll back of a deploy. We are monitoring the situation but expect no more errors at this point.

This is now resolved.

We're investigating an issue with connections to Ravelin's dashboard.

We seem to be getting some connections to the dashboard but still investigating the issue.

The issue has been identified as an increased load on the database due to an error case in a recent deploy. This has been mitigated and we are monitoring the issue

The incident is now resolved

We are currently investigating an issue with the thresholds displayed in the threshold configuration section of the dashboard. This was due to a recent config migration. We do not believe this is having any effect on the thresholds used for scoring. Please do not make any threshold changes via the dashboard based on the current values. We are currently investigating and working on a fix.

We have identified the problem and are deploying a fix.

Our fix has been deployed and the correct thresholds should now be displayed in the dashboard. We are continuing to monitor the situation. When rolling out the threshold config change we did also see a temporary issue with scoring requests between 08:20 and 08:25 UTC, when a small number of requests failed with a 500 response code. We will also be investigating this as part of this incident.

The issue has been resolved. We will continue to investigate the cause for this issue and provide an update on any further findings.

We are currently investigating issues loading the rules page on the dashboard.

We are continuing to monitor the situation but the affected component has recovered

This incident is resolved. It was an issue with queries not progressing within Bigquery. Bigquery has since recovered

We are seeing elevated errors on the API. We are investigating

This has settled down. It was caused by a short spike in latency on our primary database, we are continuing to investigate.

This incident has resolved. We are continuing to monitor and investigate isolated cases of increased latency.

We're seeing timeouts loading data from our primary database to view in the dashboard. We've scaled this up a little and are monitoring the situation. Although the same database is used for scoring that has not been effected so far

We continue to monitor the timeouts related to loading data from our primary database. Our primary and secondary databases have been swapped. That caused a spike in response times but those are now improving.

We're continuing to see intermittent performance issues with our main database. We've now seen it affect our main API as well as the dashboard. The overall impact remains small. We've scaled up the database server capacity and are monitoring the situation

This was resolved late last week

Searching in the dashboard will occasionally timeout or be slower than expected. We are currently investigating.

We are aware of a slow down in our search database. We are attempting to scale further nodes and investigate root cause.

We are still investigating and managing node scaling on our search database. We will update you further as things progress.

We have done some mitigation that will improve search latency for the dashboard but data will be delayed. We are continuing to work on the core database issue.

Mitigation is still ongoing. Search times have improved and index latency is improving. Not recovered yet but going in the right direction.

Index time has gone back to nominal latency. We are currently draining queues of backfilled data. Expect to have search data up to date shortly

We've fixed the elastic search database cluster issues, so we are no longer experiencing any errors or timeouts. However the cluster is not running completely up to date yet so there will be stale data returned on searches. We're running through the queues to fix that, and expect them to finish in ~2hrs by 5pm GMT

All data has now been updated. Searching within the dashboard should be up to date and running as normal. Thanks for your patience

We’ve seen gradually increasing latency over the last 24h that may result in increased timeouts. The latency has returned to baseline but we’re investigating what the issue was in case it recurs.

We've identified the cause of the problem as a bug in logging that resulted in ever-increasing log message sizes, eventually causing slowdowns. We've identified the problem components and rolled out versions without the logging issue. Latency has returned to baseline and we do not believe it will increase again.

We are currently experiencing an outage on our pci.ravelin.com API. We are actively investigating the issue. We will provide an update as soon as we have more information.

Our PCI API is operational again. We are monitoring the situation and continuing to investigate the root cause.

Our PCI API is operating normally again. We are continuing to investigate the root cause of the issue and are preparing a full post-mortem analysis.

We are seeing increased latency on our API. We are investigating.

We believe the latency issues are now resolved. We are continuing to monitor situation.

All our metrics have returned to baseline so we now consider the issue resolved