Historia awarii Voucherify

We observe increased error rate and API response times at voucherify caused by an operational issue of our cloud provider in N. Virginia. We are currently investigating the issue.

A fix has been implemented and we are monitoring the results.

We continue to monitor for any further issues. So far the impact on us1 cluster was very limited: increased API response latency for 50 minutes (mostly p95 and p99 longtail), between 2026-05-07 23:36 and 2026-05-08 00:48 UTC

This incident has been resolved.

dsa1 environment: We've observed a period of elevated errors from both the Voucherify API and Dashboard between 09:38 UTC and 09:46. The situation is now stabilized, we are monitoring the services related to this environment.

This incident has been resolved.

deu4 environment: We've observed a period of elevated errors from both the Voucherify API and Dashboard between 09:36 UTC and 09:40. The situation is now stabilized, we are monitoring the services related to this environment.

This incident has been resolved.

See https://health.aws.amazon.com/health/status?eventID=arn:aws:health:us-east-1::event/MULTIPLE_SERVICES/AWS_MULTIPLE_SERVICES_OPERATIONAL_ISSUE/AWS_MULTIPLE_SERVICES_OPERATIONAL_ISSUE_BA540_514A652BE1A

This incident has been resolved.

The issue has been identified and a fix is being implemented.

We are continuing to work on a fix for this issue.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

The issue has been identified and a fix is being implemented.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

We've been experiencing an elevated rate of API errors over the course of Saturday and Sunday. The issue would crash one of our virtual machines in the US1 cluster once every couple of hours, resulting in a burst of HTTP 500 errors until it would self-heal and stabilize just minutes later, affecting some percent of the incoming traffic. The root cause of the issue is now identified, we already have a temporary fix in place on production, and the ultimate fix is being implemented and expected to be deployed within the next hours.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

We're experiencing an elevated level of API errors and are currently looking into the issue.

The issue has been identified and a fix is being deployed

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

We have experienced an elevated level of API errors from some of our APIs. The problem has been resolved and all systems were back online and fully operational within 20 minutes.

Between 07:10 and 07:26 UTC we've experienced an increased level of API response times and API errors. The issue was related to our cache servers and has already been resolved by now. We are currently monitoring the service.

This incident has been resolved.

We're experiencing an elevated level of API errors and are currently looking into the issue.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

We're experiencing an elevated level of API errors and are currently looking into the issue.

For the past hour the issue has remained open for less than 1% of tenants using the as1 voucherify cluster. We expect to resolve this issue shortly.

This incident has been resolved.

We're experiencing an elevated level of API errors at our eu1 cluster and are currently looking into the issue.

A fix has been implemented and we are monitoring the results.

This incident has been resolved and all operations are back to normal.

We're experiencing an elevated level of API errors and are currently looking into the issue.

A fix has been deployed and we are monitoring the results. So far all API operations back to normal.

This incident has been resolved.

We've experienced an elevated level of API errors that were caused due to resource misconfiguration in our cluster. Update 14:21 UTC: We immediately took action by scaling up the infrastructure and fixing the resources allocation issue. After 16mins the cluster status went back to normal. Within this timeframe, some voucherify clients (using the AS1 cluster) might have received HTTP 500 errors, or timeouts.

We are currently investigating the issue. We have noticed a too long response time for some API methods. This problem occurs only in the AS1 environment, which is hosted in the Singapore AWS region.

The issue has been identified, and we applied the first fix. Now we will spend more time analyzing the root cause to implement the final solution.

There's been an issue with automatic scaling of some of our resources. A fix has been applied and we are monitoring the results.

This incident has been solved. Ultimately, we introduced improved resource allocation on that cluster, and the cluster condition returned to normal.

We are currently investigating the issue. We have noticed a too long response time for some API methods. This problem occurs only in the AS1 environment which is hosted in the Singapore AWS region.

We currently identified a problem with using Voucherify Dashboard related to Connectivity issues with Salesforce API. It seems that network connectivity problems might be related to recent incidents reported by Amazon AWS. Moreover, Heroku's engineers are looking into reports of connectivity problems to Common Runtime apps in the EU Region. Connections to one-off dynos for apps in Common Runtime EU Region and Private Spaces in Dublin are also affected by the network problem. All that results in problems that we can notice in using Voucherify Dashboard. In the meantime, we try to maximize the availability of the Voucherify Dashboard. The biggest problem users will notice in the usage of account management features - billing details, invoices, etc. The API remains unaffected.

We have isolated the issue to a third-party provider. We will update this incident record once the issue has been resolved.

This incident has been resolved. Amazon AWS solved the problem, and we don't see any more problems connecting to Salesforce API.

We are currently investigating the issue. We have noticed too long response time for some API methods. This problem occurs only in the AS1 environment which is hosted in the Singapore AWS region.

We are currently monitoring the platform to see if response times have stabilized.

This incident has been resolved. The SRE team analyzed that topic and planned improvements for the near future. Our optimistic estimation assumes that we will release further improvements in next week.

We want to share more details with our customers and readers on the internet outages that occurred on the 20th of August 2021 and what we are doing to prevent these from happening again. **Incident** August 20th, at 8:12 UTC, our systems detected increased requests latency and total outage of one of our major nodes responsible for processing validation, redemption, and publication API methods. The node manages the customers, orders, vouchers, redemptions entities, and all operations related to them. This incident affected only tenants using the **AS1 cluster \(Singapore, Asia\)**. **Impact on our customers** We saw a big increase in our 50Xs errors. Specifically, the 503 HTTP error indicates that our servers are unavailable. In this case, replicas of one of our services went down and didn’t restore properly after an automatic restart. As a result, **the API hosted on the AS1** cluster ended up in the loop of restarting pods. That caused increased latency and possible request timeouts, which ultimately resulted in reduced availability for part of the API calls. A number of the redemption API requests didn't go through because of this issue. **Source of the problem** One of our customers leveraging an account hosted on the AS1 cluster suddenly started sending a massive count of API calls. Unfortunately, the count of calls was overwhelming the limit allocated to his account, and our rate-limiter implemented in the API gateway didn't act properly. The customer was always invoking the same API method. Unfortunately, for that path used by that customer, we had a bug in the API gateway \(service which, among others, is responsible for controlling authentification and rate-limiting\), resulting in a memory leak affecting the API gateway. At the same time, our auto-scaling mechanism reacted too late and didn't set up additional resources as fast as was required. **Improvements** First of all, we have manually rescaled the resources to improve the responsiveness of our API gateway. The main problem was mitigated after that action. However, it was just a temporary solution, and our primary goal was to solve the memory leak issue as soon as possible. We had released the final improvement within 4 hours after the problem occurrence. We fully solved the problem at 12 pm UTC. In the meantime, we reached out to a customer who was abusing API to notify him about incorrect integration. As an ultimate improvement, we reconfigured the autoscaling mechanism on the AS1 cluster. After a couple of tests, we identified a proper set of parameters that should help us to react in similar cases in the future to keep the API gateway alive for a reasonable time window after recognizing the potential \(memory leak\) issue. **Summary** We understand how critical our infrastructure is for our customers’ businesses, and so we will continue to move towards completely automated systems to deal with this type of incident. Our goal is to minimize disruptions and outages for our customers regardless of the origin of the issue.

We are seeing issues on one of our nodes serving responses to API requests. Our Engineering team has started working on mitigation efforts.

We are continuing to investigate this issue. However, in the last 50 minutes, we haven't noticed any error which impacts API availability.

This incident has been solved. We identified that one of the validation rules due to a bug in a code resulted in an endless loop that was killing individual application nodes. We fixed the problem, and it shouldn't appear anymore. As a next step, we also want to identify how a user added the broken rule to the system and avoid similar cases in the future.