Alloy 장애 기록

Our API integration tests have encountered an increase in errors. We are currently investigating. Stay tuned for updates.

We are currently investigating a spike in 500 errors affecting dashboard loading for some users . Our team is actively working to identify the root cause and mitigate the issue. We also observed a brief increase in API errors during this period, but API performance is now operating as expected.

We are seeing recovery in dashboard performance with error rates returning toward normal levels. We are closely monitoring the platform to ensure stability is fully restored.

After further montioring, we have not observed any new errors. All systems are operating normally.

Our API integration tests have encountered an increase in errors. We are currently investigating. Stay tuned for updates.

Our API integration tests have encountered an increase in errors. We are currently investigating. Stay tuned for updates.

Service has returned to normal operation as of 3:36 PM ET, and our monitoring confirms that API performance is stable. We will continue to monitor closely to ensure ongoing reliability. We apologize for any disruption this may have caused and appreciate your patience while we worked to resolve the issue.

Starting at 9:17 AM ET, the homepage analytics dashboards were not loading as expected. All other dashboard pages and services remained fully operational. As of 11:47 AM ET, the system has recovered and is fully operational.

We're currently investigating a period of latency and intermittent API unavailability that occurred between 11 and 11:15 ET.

We believe the cause of the latency is resolved.

Our API integration tests have encountered an increase in errors. We are currently investigating. Stay tuned for updates.

The impact is limited to a subset of test suite endpoints used by the dashboard, our production apis were not impacted.

The issue has been resolved.

We recently identified and resolved an issue affecting some plugins initialized through the Alloy SDK. For a subset of SDK Plugins, applications initializing the SDK may have experienced unexpected behavior, which could present as UI instability or premature close events. We mitigated the issue by rolling back a recent SDK change. Following the rollback, normal plugin behavior was restored. If you continue to experience any issues, please contact Alloy Support.

Between March 4 at 11:00 PM ET and March 5 at 10:07 AM ET, we experienced an elevated rate of memory-related errors impacting evaluations running through JQ input attributes. As a result, some evaluations returned a null value for the input attribute instead of the correct value, potentially interrupting decisioning in the Workflow. Impacted applications would need to be re-run. Only workflows using input attributes with JQ were impacted; other Workflow logic was not impacted. The issue has been identified and resolved, and all systems are now operating normally.

Our API integration tests have encountered an increase in errors. We are currently investigating. Stay tuned for updates.

API functionality has been fully restored. We are continuing to investigate the impact and look into root causes.

We are currently experiencing delays in realtime webhook processing. The root cause has been identified and we are applying mitigation measures. We are now monitoring recovery as the processing queue returns to normal levels. Webhook events continue to be accepted; however, processing and delivery responses may be delayed until the existing backlog has been fully cleared.

Realtime webhook processing has been fully restored. All new webhook events are now completing in realtime as expected. During the incident, webhook events generated between 2:24 PM - 4:38 PM ET and 5:42 - 5:45 ET were not successfully processed and were permanently lost. Webhooks generated outside of this time window were not impacted. This issue affects applications that are stuck at a Journey step dependent on an Alloy webhook response to proceed (for example, action nodes), as well as applications that completed with a Manual Review outcome but are not reflecting back in core after review. Applications submitted during the affected window that are currently blocked due to a missing webhook dependency will need to be replayed. We recommend resubmitting those applications via API. Re-running an application will leverage cached data by default and will not trigger new calls to external data vendors. A thorough postmortem will be completed a Root Cause Analysis (RCA) can be shared with customers - please email support@alloy.com if you would like to receive the RCA. We are also investing in further improvements to the reliability and resiliency of our webhook queueing mechanism in the future to prevent event loss in the future.

Realtime webhook processing has been fully restored, and has maintained stability after monitoring. All new webhook events are now completing in realtime as expected. To further strengthen system stability, the engineering team has deployed an additional mitigation to prevent webhook queue backlogging under similar spike conditions. A thorough postmortem will be completed a Root Cause Analysis (RCA) can be shared with customers - please email support@alloy.com if you would like to receive the RCA. We are also investing in further improvements to the reliability and resiliency of our webhook queueing mechanism in the future to prevent event loss in the future. Please note, action is required to ensure applications process: During the incident, webhook events generated between 2:24 PM - 4:38 PM ET and 5:42 - 5:45 ET were not successfully processed and were permanently lost. Webhooks generated outside of this time window were not impacted. This issue affects applications that are stuck at a Journey step dependent on an Alloy webhook response to proceed (for example, action nodes), as well as applications that completed with a Manual Review outcome but are not reflecting back in core after review. Applications submitted during the affected window that are currently blocked due to a missing webhook dependency will need to be replayed. We recommend resubmitting those applications via API. Re-running an application will leverage cached data by default and will not trigger new calls to external data vendors.

We are currently investigating reports of elevated response times that affect both the Alloy Dashboard and API. Some customers may experience slower load times for the dashboard and longer response times for the API. Our engineering team is actively investigating.

The root cause has been identified. Response times for the Dashboard and API have returned to normal levels since 3:46 PM ET.

Between 11:20 AM and 12:04 PM ET, some workflows returned a 500 error when attempting to save a new version. Existing workflow versions and decisioning on existing workflow versions were not impacted. Once the root cause was identified as a previous code change, the issue was resolved at 12:04 PM ET when the affected change was rolled back, and all functionality related to workflow version saving was fully restored.

We are currently investigating an issue where Journey Analytics pages, Investigation detail pages, Fraud Attack Radar, and some Evaluation detail pages may not be loading data as expected. The issue is intermittent, and not all pages are impacted. Decisioning is not impacted. Journey Applications and Evaluations are completing successfully.

Our team continues to investigate the issue. At this time, there is no new information to share, but we are actively working to identify the root cause and will share updates as soon as possible.

We are still investigating this issue and will provide another update as soon as more information is available.

A fix has been implemented and all systems are operational.

We are currently investigating an issue where Journey Analytics pages, Investigation detail pages, Fraud Attack Radar, and some Evaluation detail pages may not be loading data as expected. The issue is intermittent, and not all pages are impacted. Decisioning is not impacted. Journey Applications and Evaluations are completing successfully.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

Our monitoring has detected that calls to the OpenCorporates data integration are failing since Dec 2nd 4:52 PM ET. Requests are returning a 403 “Rate limit exceeded” error. We’re currently investigating and working with urgency with the OpenCorporates team to troubleshoot.

The OpenCorporates team has assisted in resolving the issue, and the OpenCorporates data service is now fully operational. Any applications that previously failed due to this incident can now be re-run.

A widespread Cloudflare outage is impacting multiple services, including Iovation and Middesk. We are monitoring the situation and investigating possible mitigation.

Our monitoring has detected that the services Middesk, Iovation, Iovation Registration, and Zenwork Compliancely TIN Match are seeing full downtime due to a CloudFlare outage. We have reached out to the services for more information, and we are awaiting resolution from CloudFlare. We will periodically update the status page as we learn more. More information about the CloudFlare outage can be found here: https://www.cloudflarestatus.com/incidents/8gmgl950y3h7

Alloy is observing error rates have decreased across all impacted services as of 9:36 AM ET. CloudFlare has confirmed a fix has been implemented as of 9:42 AM ET. More information about the CloudFlare outage can be found here: https://www.cloudflarestatus.com/incidents/8gmgl950y3h7

Our monitoring has detected that calls to Experian are failing

Our monitoring confirms that processing has returned to normal. We have not observed any further errors since 12:20 PM ET. Any applications that failed during this period can now be re-run successfully.

Our API integration tests have encountered an increase in errors. We are currently investigating. Stay tuned for updates.

We have identified an issue with our cloud provider and are waiting for additional information

Our cloud provider confirmed there were service outages and have since released a fix. We are monitoring as systems recover.

We are investigating ripple effects of the cloud outage on our outgoing webhooks and journeys processing.

We are still working on resolving issues related to Webhooks processing. Customers may experience issues with Journey Application processing in this time

Clients on /evaluations are not impacted. Clients implemented on Journeys may experience delayed journey API response and delayed webhooks that will self resolve. The is no data loss and no need for retries at this time

We are still addressing the issues after the AWS outage. Alloy Engineering team is working on resolving the ongoing latency issues with Journeys and Webhooks. Any vendors that depend on AWS may also be experiencing degraded performance at the moment.

We are still addressing the issues following the recent AWS outage. Customers may experience issues in the following areas: - API: We are investigating unavailability in the API. Requests may return 5XX errors. - Journeys and Webhooks: Ongoing latency as the Alloy Engineering team continues to work on resolution. - Logging into Alloy and Dashboard: Latency may occur while logging in, navigating the dashboard, or submitting reviews. Some users may intermittently see gateway timeout errors. - Data Vendor: Services that depend on AWS may also experience degraded performance, causing applications to fail and result in partial results. We appreciate your patience as our team continues to work toward full restoration of service.

We are still addressing the issues following the recent AWS outage - you can follow their status at https://health.aws.amazon.com/health/status. Customers may experience issues in the following areas: - API: We are investigating intermittent unavailability in the API. Requests may return 5XX errors. - Journeys and Webhooks: Ongoing latency as the Alloy Engineering team continues to work on resolution. If there were any Journey Applications in a pending state at 11:29 AM ET they will not complete successfully, and any associated asynchronous tasks have been lost. If a Journey Application was run during this time and is not in a terminal state (e.g., Approved or Denied) - such as those waiting on step-ups or webhook actions - it will remain in an incomplete state. API calls for these applications will need to be re-run. - Logging into Alloy and Dashboard: Intermittent may occur while logging in, navigating the dashboard, or submitting reviews. Some users may intermittently see gateway timeout errors or blank screens. - Third party integrations: Services that depend on AWS may also experience degraded performance, causing applications to fail and result in partial results. Our engineering team is actively working to restore all services to full capacity as quickly as possible. We’ll continue to provide updates as more information becomes available

We are still investigating the issue. AWS has implemented additional mitigation steps and is observing some recovery, but operations remain degraded and have not been fully restored. Alloy systems have not yet stabilized, and we are working closely with AWS support to restore full functionality.

AWS continues to investigate ongoing EC2 and networking issues. Alloy services remain impacted. Our team remains in contact with AWS and will share updates every 30 minutes or as more substantial information becomes available.

At this point, we unfortunately have no substantial update to share. We continue to investigate this with AWS at the highest priority. Thank you for the ongoing patience. We will keep you updated every 30 minutes or as more substantial information becomes available.

AWS has reported progress in EC2 recovery, but Alloy’s full recovery remains dependent on AWS systems stabilizing. At this time, we are observing the following: - Webhooks: Backlog processing has begun, but a large queue remains. Real-time webhook delivery continues to experience delays. - Journeys: There continue to be intermittent Journeys failures. Some Journey Applications are failing to write to S3 - those will need to be retried once the incident is resolved. - Customer Dashboard: Intermittent latency may occur when loading or navigating the dashboard. - API: Intermittent latency persists for some requests. Our team continues to closely monitor AWS recovery and system performance. We’ll provide further updates as additional progress is made as close to 30 minute intervals as possible.

Network connectivity issues have been resolved. The Alloy Engineering team has rebuilt internal queues at 2:40 PM ET, and we are now seeing internal tests are now passing. - Webhooks: Backlog is still clearing. Realtime webhooks will continue to experience significant delays until the backlogged queue clears. - Dashboard: Has been restored to full functionality. - Journeys: Recovery is in progress. Journey Applications are starting to progress. - API: API latencies have resolved. We are continuing to monitor system performance as services come back online. We will be doing a root cause analysis into the full scope of impact. Further updates will be shared as recovery progresses.

We are continuing to monitor for any further issues.

Our monitoring has detected an increase in automated test failures, and our engineering team is currently investigating.

We are continuing to investigate connectivity issues within our cloud services. Customers may experience API timeouts while the issue persists. The dashboard remains fully operational. Webhooks are not impacted by the connectivity issues, but are still experiencing real-time processing delays as we continue clearing the backlog. We’ll provide further updates as more information becomes available.

We are continuing to investigate connectivity issues within our cloud services. Customers may also experience Dashboard slowness while the issue persists. We’ll provide further updates as more information becomes available.

At this point, we again unfortunately have no substantial update to share. We continue to investigate this with AWS at the highest priority.

AWS continues to experience residual latency. All components are currently seeing elevated latency. Our engineering team is recycling service pods, and we are beginning to see signs of recovery. We are also investigating additional areas to further mitigate the impact. We will continue to closely monitor performance and share updates as they become available.

We believe we have identified a third-party component that is throttling requests as part of recovery efforts to mitigate the ongoing impact from the AWS outage. This throttling is contributing to elevated network latency across Alloy services. We are optimistic that we can mitigate some of this impact with changes on our side and are actively working on those adjustments now. We will post another update at 8 PM ET or once substantial updates become available.

Following a best-practice recommendation from our third-party vendor, we are deploying a change to our staging environment for validation and testing. We’ll proceed to production once we confirm the fix is effective. Next update to come after we complete the staging release and confirm results - we’ll aim for an update in about 30 minutes. Additionally, the webhook backlog continues to clear. Real-time processing of webhooks remains delayed until the queue is fully processed.

We’ve completed a release to production after confirming that all errors were resolved in staging. The production environment is now showing signs of recovery. We are actively monitoring progress. Additionally, we are still waiting for the webhook backlog to clear. Real-time processing of webhooks remains delayed until the queue is fully processed.

We’re pleased to share that normal operations have been restored for all major components. System latencies and performance metrics have returned to expected levels. All internal tests are passing, and we are no longer seeing errors related to the third-party component. Our teams work closely with AWS and the third-party vendor to conduct a comprehensive post mortem of the incident. We will share a Root Cause Analysis report once this review is complete. Please email support@alloy.com if you would like the RCA once it’s available. Please note that real-time processing of Webhooks is still delayed. Until our queue clears there will be delayed Webhooks. The queue is at about 60% cleared. We appreciate your patience and understanding throughout this event.

As of 11:44 PM ET, the webhook backlog has been fully processed, and real-time webhook processing has been restored. All major Alloy services are now fully operational, with performance metrics and latency levels remaining stable. Our teams continue to work closely with AWS and the third-party vendor to complete a comprehensive post-mortem. We will share a Root Cause Analysis (RCA) once that review is finalized. If you’d like to receive the RCA once it's available, please email support@alloy.com . We appreciate your patience and understanding throughout this incident.

Our monitoring has detected that calls to ComplyAdvantage are failing, requests are returning a "timeout of 19000ms exceeded" error. The ComplyAdvantage team has confirmed they're aware of the incident and are investigating. We’re awaiting resolution from the ComplyAdvantage team.

Our monitoring confirms that processing has returned to normal. We have not observed any timeout errors since 6:02 PM ET. Any applications that failed during this period can now be re-run successfully.

Our monitoring has detected that calls to Iovation are failing, requests are returning a “Internal Error” response. We’re awaiting resolution from the Iovation team.

TransUnion has provided an update: they've identified an issue that could be intermittently impacting some API calls. Their engineers are actively working to mitigate the issue. We will keep you posted as more information becomes available.

Our monitoring confirms that processing has returned to normal. We have not observed any further errors since 7:01 PM ET. Any applications that failed during this period can now be re-run successfully.

Our monitoring has detected that calls to this service are failing, requests are returning a “Gateway” error with a 5XX response. We’re awaiting resolution from the GBG team.

Our monitoring confirms that processing has returned to normal. We have not observed any timeout errors since 4:46 PM ET. Any applications that failed during this period can now be re-run successfully.