Loyaltylion 장애 기록

We're experiencing elevated response times and error rates in our API/SDK endpoints. We're currently investigating.

We've identified the root cause and applied mitigations; service has recovered. We're continuing to monitor and are working on a permanent fix.

This incident has been resolved.

We're currently experiencing delays in data processing and high error rate in incoming webhooks. Our team is investigating.

We've identified the issue. Webhooks are being accepted again. We're working through the backlog of webhooks now.

The backlog of webhooks has been cleared. This incident has been resolved.

The LoyaltyLion SDK was unavailable for 10 minutes

We are continuing to investigate this issue.

This incident has been resolved.

We're currently experiencing elevated response times on our API endpoints. We have identified the issue and should have a fix deployed in the next hour.

A fix has been implemented and we are now monitoring the results.

This incident has been resolved.

An issue in our caching layer led to elevated error rates and response times. We have identified the issue and stabilised our systems. We're now working through the data processing backlog while monitoring.

Our systems remain stable. We are still working through the data processing backlog whilst continuing to monitor.

A fix has been implemented in our caching layer. We continue to monitor the results.

This incident has been resolved. All our systems are operational and the data backlog has been cleared.

We are currently investigating an issue with delays in data processing, this could cause delays to tier evaluations.

This incident has been resolved.

Our SDK components went offline due to an issue with our CDN provider. We migrated to our backup CDN and full functionality was restored. This only impacted shopper-facing functionality (i.e. claiming rewards on-site), no API or backend requests were impacted

We observed a performance degradation in one of our job queueing subsystems. As part of recovery to normal operations, we - paused processing on customer tier evaluation between 2024-03-02T13:30 UTC and 2024-03-02T19:00 UTC - paused processing on outbound webhook delivery between 2024-03-02T14:30 UTC and 2024-03-03T14:30 UTC - implemented controls on unusual traffic from Shopify Tier assignment and webhook delivery are now functioning as normal. We plan to gradually deliver the webhooks from the paused period over the next few days.

We're investigating increased API and platform latency. The merchant account is unaffected.

We've taken steps to triage and performance is stabilising.

The situation has been resolved, we're continuing to monitor

This incident has been resolved.

Our data processing infrastructure is running behind which is causing delays data synchronisation and events such as email sending, customer search and integrations. We've identified the issue and we're working on it.

We've caught up with the backlog of jobs and our platform is back to normal.

Our background job processing infrastructure is running behind which is causing delays in events such us email sending, customer search and integrations. We've identified the issue and we're working on it.

This incident has been resolved.

Our background job processing infrastructure is running behind which is causing delays in events such us email sending and integrations. We've identified and addressed the problem, and we're working through the backlog of jobs.

We are currently investigating this issue.

We're further increasing capacity to account for additional traffic over the BFCM period

This incident has been resolved.

A misconfigured cache caused elevated response times for our platform and API routes.

We are currently investigating elevated response time in our API and platform.

We've identified the issue and applied a fix. We're currently monitoring to make sure our systems are fully recovered.

This incident has been resolved.

We are currently investigating elevated response time in our API and platform.

Shopper-facing functionality has been restored. We're continuing our investigation in order to restore all services.

Most functionality has been restored. We're still working on a few background services.

All systems are back to normal. We'll continue to monitor and perform a root cause analysis.

We're currently investigating elevated error responses from the LoyaltyLion API

We are continuing to investigate this issue.

The situation has now been resolved

We saw an outage beginning at 23:58 BST on May 1st. Service was partially restored at 00:34 BST May 2nd and normal service levels resumed at 01:38 BST. We're still investigating the root cause of the outage and will update this incident as we learn more.

Between 17:10-17:20 API and Platform endpoints experienced a partial outage. Some API requests may have failed during this period. By 17:20 normal service was restored.

Shopify merchants unable to sign in for ~90 minutes; no shopper-facing functionality was impacted