Issues with API availability
This incident was triggered by a routine system maintenance task and escalated due to an edge case in our application caching infrastructure. At 02:50 UTC, while system maintenance was taking place, a database deadlock was caused by multiple background processes attempting to write data simultaneously. Under normal circumstances, this behaviour would self-correct. However, in this instance one of the processes retried repeatedly, and the database became unresponsive at 03:07 UTC. As a result of this, incomplete data were added to the cache, causing up to 100 currency symbols to be temporarily unavailable in live API responses. We identified the issue and immediately began working to resolve it, which required several manual interventions in our infrastructure. Service was restored at 03:32 UTC, 25 minutes after the first production issues were reported. The incident was then marked resolved on our Status Page. \(The HNL currency symbol however remained unavailable in live API responses, until approximately 11:20 UTC when its data contributors were restored\). A fix for this behaviour is being developed, with deployment planned in the next 2 weeks. In the meantime, there remains a small risk that a similar issue could occur again. We are actively working on reducing this risk ahead of the fix being deployed, and in the unlikely event of a repeat occurrence, our 24/7/365 monitoring team are standing by to resolve it without delay.
