13 Advicepay incidents · May 2020 — official updates, affected components, duration and resolution details.
An anti-fraud measure that resulted in clients being unable to manually enter bank account information for future-due subscriptions without upfront payments has been resolved.
We’re currently experiencing partial disruption due to a widespread AWS outage affecting several third-party services and integrations. Our core platform remains accessible, but some connected features may be slow or temporarily unavailable. Please also note that response times from our Support team may be delayed as a result of this outage. We’re monitoring the situation closely and will provide updates as AWS restores services. Thank you for your patience.
This incident has been resolved.
At 11:37 AM MDT on 4/19/2023 we received an alert from our uptime monitoring service that the AdvicePay application was unreachable. Functionality was restored as of 11:50 AM MDT. We are investigating the underlying cause.
AWS S3 was having issues with very high latency on the day of the outage. Additionally, AdvicePay was using a default HTTP client without a timeout to request firms' logo images from S3. The combination of those two issues led to a denial of service due to starvation of connections on our servers. AWS has since resolved things on their end. To protect against this issue in the future, AdvicePay has implemented timeouts for all HTTP clients that we use for requesting any resource.
A large number of quarterly requests from our payment processor resulted in slow response times on AdvicePay's production site from 12:00 AM to 12:01 AM.
A series of requests from a third party resulted in slow response times from 9:02-9:04 PM MDT
A series of requests from a third party resulted in slow response times from 1:04-1:05 AM MDT
A series of requests from a third party resulted in slow response times from 11:02-11:03 PM MDT
A series of requests from a third party resulted in slow response times from 1:03-1:04 AM MDT
A series of requests from a third party resulted in slow response times from 10:02-10:04 PM MDT
An unexpected large number of webhook calls from one of AdvicePay's integration partners resulted in slow response times from 3:00AM until 3:02AM while we processed the calls.
An unexpected large number of webhook calls from one of AdvicePay's integration partners resulted in slow response times from 1:00AM until 1:02AM while we processed the calls.
On 7/29/2020, from 8:47 AM until 9:06 AM (MDT) a bad gateway issue prevented users from accessing the AdvicePay production site.
This issue was due to a memory leak when generating invoice PDFs for a large amount of historical data. As a result, we implemented better fault tolerance on the server instances.
A server issue occurred preventing access to the production instance AdvicePay.
AdvicePay’s server instances were upgraded to faster hardware, which uncovered a code issue that resulted in a race condition. The race condition was a low-level issue that caused a server panic.