39 Opennode incidents · June 2019 — official updates, affected components, duration and resolution details.
Withdrawals are temporarily disabled while we investigate and resolve a platform issue. Our team is working on a fix and will restore functionality as soon as possible.
We are continuing to investigate this issue.
This incident has been resolved.
We are currently investigating this issue.
A fix has been implemented and we are monitoring the results.
We are continuing to monitor for any further issues.
We are continuing to monitor.
This incident has been resolved.
We are currently experiencing an outage with our upstream email sender. As a result, some transactional and notification emails may be delayed or not delivered. Our engineering team is actively working with our provider to restore normal email delivery. This issue does not impact core payment processing or platform functionality. We will provide updates as soon as more information becomes available.
This incident has been resolved.
On May 14, 2024, OpenNode experienced an incident with our production database, resulting in performance degradation and service disruption. The root cause of the issue was identified as a deprecated API loading excessive data, leading to a database overload.
**Summary:** On May 14, 2024, OpenNode experienced an incident with our production database, resulting in performance degradation and service disruption. The root cause of the issue was identified as a deprecated API loading excessive data, leading to a database overload. **Timeline of Events:** * **10:30 PM PST:** Issue Detected * Our monitoring systems alerted us to unusual activity within the production database, indicating potential performance issues. * **11:30 PM PST:** Service restart * A decision was made to restart auxiliary services to alleviate any immediate issues and restore stability to the system. * **12:30 AM PST:** Database Workload Issue Identified and Fix Deployed * Upon further investigation, it was determined that the root cause of the problem was a deprecated API that was loading an excessive amount of data into the database, causing an overload. * A fix was promptly developed and deployed to address the issue and mitigate further impact on system performance. * **1:30 AM PST:** Monitoring Fix and Stability * Additional measures were taken to enhance monitoring systems and ensure the stability of the database environment. * **2:00 AM PST:** Issue Confirmed Fixed * Following the deployment of the fix, monitoring systems indicated a return to normal database workload and performance levels. * Our internal testing confirmed that the issue was resolved, and normal service operations were restored.
Our data processing infrastructure is running behind which is causing inaccuracies in the reporting tools. No data has been lost and the system should be caught up shortly.
All current payments are being processed and the team is working through the backlog of payments.
The system is now fully caught up and all payment notifications were sent out.
Due to unforeseen technical challenges, our scheduled maintenance is extended. We apologize for any inconvenience. Updates will follow as we work to resolve this promptly.
Maintenance has been completed and all systems are fully operational.
We're receiving a higher API error rate on payment requests initiated by merchants/businesses who have auto-conversion turned on. We have notified our exchange partner and are monitoring for resolution.
This incident has been resolved.
We're receiving a higher API error rate on payment requests initiated by merchants/businesses who have auto-conversion turned on. We have notified our exchange partner and are monitoring for resolution. In the meantime, all conversions and outgoing bank transfers are disabled.
Our Exchange Partner has identified a fix and its working on a resolution.
This incident has been resolved.
We have recently been informed that one of our banking partners, Wyre, has made the decision to cease operations over the course of the next month. Clients that are onboarded with this provider and rely on fiat-related activities such as conversions will not be able to generate new payments at this time, unless the automatic conversion functionality is turned off. We have already initiated the process of engaging an alternative banking partner to seamlessly address fiat transaction needs. We anticipate having this new partner integrated into our platform within the next few weeks, thereby ensuring that all fiat operations are available and fully operational without any significant interruption.
Entities and Individuals registered in the United States can now onboard with our new provider through the dashboard. Upon completion, automatic conversion and all fiat related services will be reenabled. If you have questions or encounter issues, please reach out to [email protected] or visit the Intercom widget on the dashboard
We are continuing to monitor for any further issues.
This incident has been resolved.
We're currently investing an issue where incoming payment webhooks are delayed.
A fix has been implemented, we're monitoring the results.
Incoming payments webhook queue is almost clean. Webhooks should start to operate instantly soon.
Queue is now clean. Incoming payment webhooks are now being delivered instantly.
We're receiving a higher API error rate on payment requests initiated by merchants/businesses who have auto-conversion turned on. We have notified our exchange partner and are monitoring for resolution. In the meantime, all conversions and outgoing bank transfers are disabled.
This incident has been resolved and we are monitoring the results. Conversions and bank transfers are now enabled.
This incident has been resolved and the platform is fully operational.
Emails are still being sent and delivered but with significant delays due to an issue our provider.
This incident has been resolved.
Our team is investigating an connectivity incident affecting our platform and will provide updates here.
This issue has been identified and is related to our cloud infrastructure provider. The connectivity issue is partial (it doesn't apply to all regions).
A fix has been implement by our cloud infrastructure provider and we are monitoring the results. Connectivity is coming back up.
This incident has been resolved and the platform is now fully operational.
We're receiving a higher API error rate on payment requests initiated by merchants/businesses who have auto-conversion turned on. We have notified our exchange partner and are monitoring for resolution. In the meantime, all conversions and outgoing bank transfers are disabled.
This incident has been resolved and we are monitoring the results. Conversions and bank transfers are now enabled.
This incident has been resolved and the platform is fully operational.
We currently investigating an issue where Lightning payments are taking longer than expected to update its status. Lightning payment webhooks are also being affected.
The issue has been identified and a fix is being implemented by our engineering team. Paid payments will be updated shortly and webhooks will be fired. Please refer to this page for more updates.
We are continuing to work on a fix for this issue. Please refer to this page for more updates.
Affected lightning payments are now reflecting their updated status and webhooks were sent.
A fix has been deployed. We are monitoring for any further issues.
This incident has been resolved and the platform is fully operational.
We're receiving a higher API error rate on payment requests initiated by merchants/businesses who have auto-conversion turned on. We have notified our exchange partner and are monitoring for resolution. In the meantime, all conversions and outgoing bank transfers are disabled.
A fix has been implemented by our partner and we are monitoring the results.
This incident has been resolved. Conversions and bank transfers are now fully operational.
We're seeing an elevated API error rate on outbound Lightning payments.
We're receiving a higher API error rate on payment requests initiated by merchants/businesses who have auto-conversion turned on. We have notified our exchange partner and are monitoring for resolution. In the meantime, all conversions and outgoing bank transfers are disabled.
A fix has been implemented by our partner and we are monitoring the results.
This incident has been resolved. Conversions and bank transfers are now fully operational.
We're receiving a higher API error rate on payment requests initiated by merchants/businesses who have auto-conversion turned on. We have notified our exchange partner and are monitoring for resolution. In the meantime, all conversions and outgoing bank transfers are disabled.
This incident has been resolved. Conversions and bank transfers are now fully operational.
We're receiving a higher API error rate on payment requests initiated by merchants/businesses who have auto-conversion turned on. We have notified our exchange partner and are monitoring for resolution. In the meantime, all conversions and outgoing bank transfers are disabled.
This incident has been resolved. Conversions and bank transfers are now fully operational.