Historia awarii Cognito

On May 21, 2024 beginning at 11:20 AM PDT until 2:12 PM, 80% of calls to Cognito's /identity_searches endpoint were erroneously returning an empty set of results in the "identity_records" field. The error was due to a code change that introduced a race condition, which caused the endpoint to return a response before the results were available. Once we identified the issue the code change was reversed and normal operation was restored within 10 minutes.

We are implementing a fix for a bad deploy that is impacting some identity searches

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

An upstream data provider is currently experiencing issues. They are aware of the problem and are working to remediate. This is not likely to impact your requests, but could increase 95p response times.

Our upstream provider has confirmed resolution of the issue, we apologize for the degraded performance.

This issue has been resolved.

ID document processing has an increased error rate. The issue has been found and a workaround is in progress. Any ID documents that cannot be processed are still stored in your account and can be overridden at any time, and users are able to complete all steps in their session. This does not affect Lightning, Liveness, or the Risk Engine.

A fix for the underlying issue is still in progress.

Error rates have resolved after a fix was implemented. We apologize for the disruption. As a followup to this event, we will be adding additional redundancy to all affected systems.

Our upstream hosting provider, AWS `us-west-2`, is having intermittent connectivity problems. We are monitoring the situation.

Per our hosting provider: 8:01 AM PST We have identified the root cause of the Internet connectivity to the US-WEST-1 Region and have taken steps to restore connectivity. We have seen some improvement to Internet connectivity in the last few minutes but continue to work towards full recovery.

Systems have been running normally for the last 10 minutes. We have not yet received official confirmation from our hosting provider that the issue has been resolved, so we will be monitoring in the meantime.

Our upstream provider has confirmed resolution of the issue, we apologize for the intermittent errors.

For a period of 6 minutes there was an increased error rate due to a slow database migration. The issue has been resolved.

AWS is currently having internet connectivity issues with its US-West-2 region which is impacting response times for some customers. We are monitoring the situation. You can also receive updates on the Amazon Web Services status page: https://status.aws.amazon.com

AWS have identified a fix and are working towards a resolution.

All requests have stabilized at their normal speeds. We are monitoring and will update this when we receive the final confirmation from AWS.

AWS have confirmed that their networking issues have been resolved, thank you for your patience.

There was approximately a 5m spike in response time latency for some customers. We initiated a failover of the unresponsive data sources and all systems have returned to normal. We are continuing to monitor the situation.

We have confirmed systems have been stable and everything has returned to normal.

We are investigating an issue with one of our upstream data providers that is causing searches to fail for a subset of customers.

We are continuing to investigate this issue.

The upstream provider has issued a fix and we are monitoring for any additional instability. All identity searches are functioning normally.

We're seeing an increase in response times on certain data providers causing searches to be delayed

Response times have recovered. We are continuing to monitor for stability.

This incident has been resolved.

Identity search performance is currently degraded due to an upstream provider outage. We have failed over to secondaries while we work with the provider to restore full service.

The upstream provider has deployed a fix and services have been restored to normal.

We are currently investigating elevated response times on identity searches

The issue has been fixed and we are monitoring the situation.

The incident has been resolved.

We are executing a server failover due to elevated response times

The failover was successfully executed and service levels have returned to normal.

We're investigating issues with a data provider that is causing requests to fail. Initiating fallback procedures, service should be restored momentarily.

Fallback procedures have been completed successfully and systems should be nominal. We are working with the upstream data provider to determine the cause of the issue.

We have restored redundancy and all systems are go! We apologize for the inconvenience. Please let us know if you continue to encounter any issues.

We are investigating an issue with reduced match rate on date of birth and SSN.

We are in touch with data providers to rectify the issue and have put a temporary fix in place that may result in marginally lower hit rates. We will update this incident as we near a resolution.

We have worked with our data providers to address the issue. We are monitoring for regressions, but match quality appears to have returned to normal.

This issue has been resolved.

A small subset of our data is responding slowly and we are working to resolve the situation.

The part of our data provider infrastructure, applicable to a subset of our customers, that was responsible for slow or failing requests appears to have recovered. We are actively monitoring the situation and looking further into the triggering event. For customers that were experiencing failed requests, those should now be resuming successfully.

All components have been stable since the monitoring phase and are continuing to operate normally.

For a period of about 30 to 45 minutes, a portion of all live API requests were incorrectly rejected due to an application server that became stuck in a misconfigured state. Several other servers remained operational so most requests were able to be serviced. A fix for the underlying issue was deployed and the incident was resolved.

We are investigating an increased error rate with an upstream provider. We are in contact with them and are working towards a solution.

The upstream provider has discovered the issue and they are working to remedy it as quickly as possible.

Services are currently operating normally. We are monitoring the situation to ensure that no regressions occur.

We are continuing to monitor for any further issues.

The issues have been fully resolved, we apologize for any inconvenience this has caused. We have been in the process of adding additional redundancy for this particular provider and will continue to work to do so to ensure this does not happen again in the future.

We are investigating increased instances of errors occurring when creating identity searches and delayed queues.

Automated routine maintenance by Heroku on a Redis instance is causing unexpected errors. We are working with Heroku engineers to remedy the situation as quickly as possible.

We are continuing to work with Heroku engineers to remedy the issue.

Searches have returned to normal and we are monitoring to ensure that everything is stable.

Systems have remained stable for over an hour. We apologize for this inconvenience and we will work with Heroku to ensure that it does not happen again.

Cognito's sandbox API is currently experiencing some network connectivity issues and we are working to restore services. The live API is unaffected.

The incident has been resolved.