Störungsverlauf von Subscribe Pro

Background job processing went down on 10/1 or us-east-1 region and 10/4 for eu-west-1 region. Both incidents were due to the same root cause: * A key scheduling service failed. * This was triggered by our core transactional database recycling after a minor version upgrade. * This happened around 3am eastern time on 10/4 in us-east-1 region. * Happened around 3am eastern time on 10/1 in eu-west-1 region. * We should have noticed this, however there was an error in some of our alerting / notification scripts. * We have since updated and thoroughly tested the notification scripts.

We are currently investigating an issue causing an outage of our Merchant App and API.

The issue has been resolved.

This incident was caused because we missed updating a key credential during deployment to our production platform. We will be reviewing our process around how we track such credentials during updates.

We are currently investigating an issue causing degraded performance of our Merchant App and API.

We have implemented a fix and are monitoring. Please contact [email protected] if you continue to experience performance or connectivity issues.

All systems are currently operating normally. We are still monitoring. We will post a postmortem write up once we fully understand the root cause.

We have identified the core issue and a fix is in place. We will follow up with a postmortem.

Currently all payment transactions submitted through the Subscribe Pro Vault are failing. All other APIs and systems are operational. We have identified an issue with an upstream provider causing failing transaction requests. They have published an incident here: https://status.spreedly.com/incidents/1ly5j04m8mjs

Our upstream provider has implemented a fix and transactions appear to be processing successfully now. We are actively monitoring the results.

This incident has been reported resolved by our upstream provider, and we are also no longer seeing any issues, so we are also considering it resolved.

Our Merchant App and API are currently experiencing partial downtime due to a long running database update.

Full service has been restored.

We are currently investigating degraded performance of our Merchant App and API.

A fix has been completed and performance has returned to normal. We will continue to monitor.

This incident has been resolved.

We are currently investigating a partial outage of our Merchant App and API.

The issue causing a partial outage of our Merchant App and API is resolved.

We are currently investigating an outage of our Merchant App UI and API. Currently it looks like an issue with our upstream provider, CloudFlare. We will keep you updated as we learn more.

We are continuing to investigate this issue.

The issue with our upstream provider, Cloudflare, appears to be resolved. We are continuing to monitor.

This incident has been resolved.

We are currently identifying an issue with our merchant and and API.

We identified an issue during our most recent deployment that resulted in traffic to our API and web UI not being processed. The relevant change was reverted and the issue has been resolved.

We are currently investigating an issue causing payment transactions to fail.

We have implemented a stable workaround. The core issue is related to an SSL certificate on core.spreedly.com. We will update when the core issue has been resolved.

A fix has been deployed by Spreedly and the core issue has been resolved. Transactions are processing successfully.

This incident is due to issues at our upstream provider. We will keep you updated as it is investigated and resolved. https://status.platform.sh/

A fix has been applied by our upstream provider. Services are currently up and being monitored.

The issue has been resolved by our upstream provider.

We experienced a brief period of downtime (3-5 minutes) starting around 10:49pm eastern. The issue is related to a Cloudflare outage in Ashburn, VA. https://www.cloudflarestatus.com/ At this point, service has been restored and we are monitoring the issue.

A fix has been implemented and we are monitoring the results.

Network issue was marked resolved by Cloudflare at 4:25am eastern.

We are investigating a partial outage of our Merchant UI and API

A fix is in place and we are monitoring.

This incident has been resolved.

Today our API and web UI were down for close to 15 minutes. The root cause was a long running database update which was necessary as part of an upgrade. When we estimated the length of time the update would need we were off by 1 order of a magnitude. Consequently the update took close to 15 minutes instead of a little over a minute and translated into downtime for the API and UI. These sort of database updates are something we do rarely. Never-the-less, we have identified methods we can use to do more extensive testing in the future, for similar updates. This would allow us to plan alternate, incremental ways to update the database and schedule those at off times.

We are investigating reports that some recurring orders are not being processed as expected.

We have fixed the issue causing some recurring orders not to be processed. We will continue to monitor for any further issues.

This incident has been resolved.

We are currently investigating a 500 Internal Server Error that is preventing access to subscription and customer management pages in our Merchant UI.

A fix has been implemented and we are monitoring. If you continue to experience issues accessing Merchant UI functionality, please contact [email protected]

This incident has been resolved.

We are currently investigating an outage of our platform UI and API.

A fix has been implemented and we are monitoring.

This outage has been resolved. Please contact [email protected] if you experience any further issues accessing our platform web UI or API.

We had an outage this morning that affected all Subscribe Pro customers. The root cause was a disk that filled up on our transactional database system. There were two other systems failures which caused this issue to lead to over an hour of downtime: * Our designated on-call personnel failed to respond as promptly as required. * We were operating with a mis-understanding of the auto-scaling settings around the disk that filled up. We have taken action to address both of these system issues: * We have fine tuned our procedures for on-call personnel and increased our training frequency. * We have adjusted and tested the relevant auto-scaling settings for database disk. We take any down time very seriously and have used this as opportunity to improve our systems and processes so that this type of issue can’t happen again. Thank you.

Issue was determined to be caused by Cloudflare outage - https://www.cloudflarestatus.com/

Cloudflare has implemented a fix. Service appears to be restored. We will monitor and provide an update when the issue is resolved.

Cloudflare has considered this issue resolved. Affected services have resumed normal operation. You can find more information about this issue here: https://www.cloudflarestatus.com/incidents/tx4pgxs6zxdr

We're currently investigating a partial outage of our hosted widget code. The cause seems to be related to a Cloudflare connection issue.

Issue was determined to be caused by Cloudflare outage - https://www.cloudflarestatus.com/ We have temporarily bypassed the issue and hosted code should be available.

A fix has been implemented and we are monitoring the results.

We are continuing to monitor for any further issues.

The root cause was a Cloudflare outage. Cloudflare service has been fully restored. Subscribe Pro hosted code is now fully operational as well. https://www.cloudflarestatus.com/

We are currently investigating a partial outage of our recurring order processing.

We have identified the issue and put a fix into place. We will continue to monitor to ensure everything continues to process as expected. If your accounts' subscriptions were delayed, they should process soon.

The partial outage of recurring order processing has been resolved.

We are currently investigating a partial outage of our platform UI and API.

We experience brief intermittent downtime due to an issue with our logging cluster and application logging queue. Full service has been restored.