54 Dwolla incidents · Ιούνιος 2023 — official updates, affected components, duration and resolution details.
Έχουμε εντοπίσει ένα θέμα που προκαλεί σφάλματα HTTP 504 σε τελικά σημεία Sandbox API και την πρόληψη της σύνδεσης ταμπλό Sandbox. Αναγνωρίσαμε το πρόβλημα και αναπτύσσουμε μια λύση. Θα παρέχουμε ενημερώσεις καθώς σημειώνουμε πρόοδο.
Η κατάσταση έχει αναπτυχθεί και το περιστατικό έχει επιλυθεί.
Αυτόματη μετάφραση από την επίσημη ενημέρωση του συμβάντος.
Επί του παρόντος βιώνουμε ένα ζήτημα που επηρεάζει τις Άμεσες Πληρωμές που υποβάλλονται μέσω του API μας. Από περίπου 7:35 AM CT, αυτές οι πληρωμές έχουν αποτύχει με τον κωδικό σφάλματος UR01. Συνεργαζόμαστε με τους εταίρους μας για τον εντοπισμό και την επίλυση του ζητήματος. ACH μεταφορές και άλλες λειτουργίες API δεν επηρεάζονται. Σημείωση: Οι μεταφορές αυτές δεν ολοκληρώθηκαν. Παρακαλώ περιμένετε να υποβάλετε ξανά μέχρι να επιβεβαιώσουμε την επισκευή. Εκτιμούμε την υπομονή σας καθώς εργαζόμαστε για την αποκατάσταση των άμεσων πληρωμών. Περαιτέρω ενημερώσεις θα παρέχονται μόλις είναι διαθέσιμες.
This incident has been resolved. Instant Payments are now processing normally. The last UR01 error was observed at 9:33 AM CT. Transfers that failed with UR01 during the disruption did not complete and may be retried. We appreciate your patience.
Αυτόματη μετάφραση από την επίσημη ενημέρωση του συμβάντος.
Βιώσαμε ένα πρόβλημα που επηρεάζει την παράδοση webhook και τη λειτουργία σύνδεσης ταμπλό στο περιβάλλον Sandbox μεταξύ 12:47pm CT και 4:44pm CT. Το θέμα αυτό έχει πλέον επιλυθεί.
Αυτόματη μετάφραση από την επίσημη ενημέρωση του συμβάντος.
Αυτή τη στιγμή ερευνούμε αυξημένα ποσοστά σφάλματος (HTTP 500) που επηρεάζουν τα αιτήματα προς τους πελάτες, πηγές χρηματοδότησης, και μαζικές πληρωμές API καταληκτικά σημεία. Μερικά αιτήματα για αυτά τα τελικά σημεία μπορεί να αποτύχουν ή να διαλείπουν. Η ομάδα μας ερευνά ενεργά. Θα παρέχουμε μια ενημέρωση μέσα μόλις έχουμε περισσότερες πληροφορίες.
Μια διόρθωση έχει αναπτυχθεί και τα ποσοστά σφαλμάτων επιστρέφουν στο κανονικό σε όλα τα επηρεαζόμενα τελικά σημεία και το ταμπλό. Παρακολουθούμε για να επιβεβαιώσουμε την πλήρη ανάρρωση.
Το θέμα αυτό έχει επιλυθεί.
Αυτόματη μετάφραση από την επίσημη ενημέρωση του συμβάντος.
We have identified an issue causing the "Delivery Status" for Operational Notifications within the Dwolla Dashboard to lag or not update correctly. Please note: Outbound emails are successfully delivering to your end-users in real-time. This is strictly an interface reporting delay within the Dashboard "Notifications" tab. Your end-users are receiving their transactional and account alerts normally, and no action is required on your part. We are working to restore real-time status updates to the dashboard view.
A fix has been implemented to resolve the status sync delay. The "Delivery Status" indicators within the Dwolla Dashboard are beginning to catch up and accurately reflect actual delivery states. We are actively monitoring the dashboard queues until all historical notification statuses are fully backfilled and up to date. As a reminder, actual email delivery to end-users was unaffected during this window.
The issue affecting the dashboard display for Operational Notification statuses has been fully resolved. All delivery statuses within the Dwolla Dashboard are now updating in real-time and accurately reflect historical deliveries. Thank you for your patience.
We experienced a brief service disruption in our Production environment between 3:14 PM and 3:35 PM CT. The issue was related to intermittent resource contention which impacted the processing of balance-related requests. This resulted in elevated error rates for: - Balance Inquiries: Retrieving account or funding source balances (/funding-sources/{id}/balance). System performance stabilized at 3:35 PM CT and error rates have returned to normal. This incident has been resolved.
We have experienced a delay in the morning processing of ACH transactions on the platform. This impacts the webhook delivery and API transfer status updates for ACH credit and debit transactions that are expected to update today. Due to today being a high processing day, we expect this issue to cause impacts with the 9:00 AM CT Same-day ACH debit and credit transactions. These transactions will likely be delayed to the 1PM CT Same-day ACH window. Our support team is working as quickly as possible to recover from this. We will provide an update when the process completes, and we apologize for any inconvenience caused by this disruption.
This incident has been resolved. All transfers statuses have been updated and webhooks are being delivered. Please note: As a result, this impacts the 9:00 AM CT export of Same-day ACH debit and credit transactions. These transactions will now export at 1:00 PM CT.
The FED has experienced system issues this morning causing a delay in Same Day ACH processing. Same-Day ACH debit and credit transactions that were expected to clear at 12:00 PM CT will be delayed. We are monitoring the situation and working with our partners, and will update this status when we have more information about when those transfers will be processed. We apologize for any inconvenience caused by this disruption.
The Federal Reserve experienced system issues this morning, causing a delay in Same-Day ACH processing. Same-Day ACH debit and credit transactions that were expected to clear at 12:00 PM CT will now clear at 5:00 PM CT. We apologize for any inconvenience caused by this disruption.
We are currently investigating reports of intermittent failures and elevated error rates (500 Internal Server Errors) affecting balance checks and balance-sourced transfers within our Production API. Our engineering team is actively working to identify the root cause. We will provide an update as soon as more information is available.
We experienced a service disruption in our Production environment between 11:25 AM and 11:36 AM CT. The issue was related to intermittent resource contention which impacted the processing of balance-related requests. This resulted in elevated error rates for: - Balance Inquiries: Retrieving account or funding source balances. - Transfer Requests: Initiating transfers, particularly those involving balance sources. System performance stabilized at 11:36 AM CT and error rates have returned to normal. We are continuing to monitor our systems to ensure sustained stability while we complete our internal review.
This incident has been resolved.
We experienced a temporary outage that impacted the /customers endpoint. Between 2:22:57 PM and 2:32:38 PM CT, the system was returning HTTP 500 errors due to a brief disruption in service. The issue has been resolved, and the system is stable. We apologize for any inconvenience this may have caused.
We experienced a delay in Same-Day ACH processing. Same-Day ACH debit and credit transactions that were expected to clear at 12:00 PM CT will now clear at 5:00 PM CT. We apologize for any inconvenience caused by this disruption.
We're investigating increased API 500 errors on the customers/{accoundId}/transfers and accounts/{accountId}/transfers endpoints. This affects the API and the Dashboard. We will provide updates as they are available.
A fix has been deployed and the errors have stopped occurring.
We have experienced a delay in the morning processing of ACH transactions on the platform. This impacts the webhook delivery and API transfer status updates for ACH credit and debit transactions that are expected to update today. Our support team is working as quickly as possible to recover from this and avoid downstream impacts with the 9:00 AM CT export of Same-day ACH debit and credit transactions. We will provide an update when the process completes, and we apologize for any inconvenience caused by this disruption.
This incident has been resolved. All transfers statuses have been updated and webhooks have been delivered. There was no impact to the 9:00 AM CT export of Same-day ACH debit and credit transactions. All Same-day transfers have exported.
We have identified an issue impacting sandbox transfer searches via the dashboard and api. The team is working on a fix and will provide updates as they become available. This impacts the sandbox environment only.
A fix has been identified and is being deployed.
The fix is deployed and sandbox transfer search in the dashboard and api are functioning normally.
We experienced a brief disruption with the Sandbox API that impacted funding source creation and retrieval, as well as transfer creation and retrieval for all Sandbox users. The issue occurred between 9:06 AM CT and 9:27 AM CT and has since been resolved.
We are currently investigating an issue with instant payments status updates. The Dwolla platform is currently able to successfully create and process instant payments, but their statuses are not being updated in our system. Clients using instant payments will see these statuses marked as "pending". We are actively investigating and will provide updates as we make progress.
A fix has been applied, and the team is processing the backlog of status updates. We will provide an update when this is completed.
This incident is resolved. All pending instant payment statuses have been updated.
We have experienced a delay in the morning processing of ACH transactions on the platform. This impacts the webhook delivery and API transfer status updates for ACH credit and debit transactions that are expected to update today. Our support team is working as quickly as possible to recover from this. This may have an impact on the 9:00 AM CT export of Same-day ACH debit and credit transactions. We will provide an update as we have more information to share. We apologize for any inconvenience caused by this disruption.
The issue is ongoing due to a disruption with Federal Reserve ACH processing. Please note: As a result, this impacts the 9:00 AM CT export of Same-day ACH debit and credit transactions. The 9:00 AM CT Same-day window will be missed, and these transactions will now export in the 1:00 PM CT file. We will continue to provide updates as more information becomes available.
The issues at the Federal Reserve continue to impact the Dwolla platform. The Dwolla team is monitoring this closely and will initiate internal processes to update ACH credit and debit transactions and send webhooks as soon as possible.
The Dwolla team has initiated the process to update ACH credit and debit transactions and send webhooks. Webhooks should begin arriving at client endpoints in the near future.
All affected ACH transfer statuses from this morning have been updated, and related webhooks are being delivered. Same-day ACH debit and credit transactions originally scheduled to settle at 11:00 AM CT are now expected to settle at 5:00 PM CT. The Dwolla team continues to monitor the systems and will provide an update later this afternoon.
The Federal Reserve is continuing to experience delays. This will delay the webhook delivery and API transfer status updates for ACH credit and debit transactions that are expected to update at 5pm today. The Dwolla team is monitoring this closely and will initiate the process as soon as possible.
ACH transfer statuses that were expected to update at 5pm have all been updated, and related webhooks are being delivered. This incident is resolved.
The Fed experienced a delay processing our Same-Day ACH transactions that should have exported today at 3:00 PM CT. This impacts the webhook delivery and API transfer status updates for Same-Day ACH credit and debit transactions that were expected to process at 5:00 PM CT today. As a result, these transfers are now expected be processed tomorrow morning 02/06/2026. We apologize for the inconvenience.
Dwolla has issued some 1099k forms to end users with the amounts higher than expected. If your business has customers who are receiving a 1099k from Dwolla, they may have received a form 1099k with the wrong amount. The Dwolla team is aware of the issue and is taking steps to correct it. We will provide more information on the steps taken as it becomes available. Note that Dwolla's payment platform and API are not impacted by this issue.
We have sent an email to all affected end users explaining what happened and informing them that corrected Form 1099-K will be issued, sent to them, and filed with the appropriate tax authorities. We are working through the process to send the corrected 1099-K forms to end users, and we will get those sent and filed by the February 2, 2026 deadline. Note that Dwolla's payment platform and API are not impacted by this issue.
We have resolved the issue that caused incorrect amounts. We intend to file the updated 1099K forms with the appropriate tax authorities on Monday morning to avoid any operational impact over the weekend. We will provide an update when this is complete.
Most of the incorrect 1099K forms that were sent to end users were stopped before they were filed with tax agencies. These 1099K forms have been updated and filed. A small number of 1099K forms were filed with tax agencies and require corrections. The team is working through the corrections process. We will provide an update when this is complete.
The team has completed the Form 1099K corrections and filed them with the appropriate tax agencies. This incident is resolved.
We have identified an issue causing a subset of webhooks to remain in a "Queued" or "Pending" state rather than being delivered to API consumer endpoints. Our engineering team has traced the issue to performance degradation within our processing infrastructure that began at approximately 1:30 AM UTC on Saturday, January 3rd. We have taken steps to isolate the affected components and have restored normal processing for new events. Impact: Webhooks generated between Saturday, Jan 3rd and Monday, Jan 5th may experience significant delays or may not have been sent yet. Current webhook traffic is processing normally.
We are currently monitoring system health to ensure continued stability. Our team is now working through the recovery process for webhooks that were delayed or missed during the service interruption. We will provide an update once the backlog has been fully processed and all events have been delivered.
The team is currently processing the backlog of webhooks. We will provide an update when this is complete.
All delayed webhooks and customer notifications have been successfully sent. This incident is resolved.