Spreedly の障害履歴

1. 2026年9月2日 15:11 UTCinvestigating

   Spreedly はトランザクション API で劣化したパフォーマンスを生じる可能性のある問題を発見しました。 早期通知を可能な限り迅速に通知するよう努めていますが、実際のスコープとインパクトを調べ、詳細が利用可能になったらただちに更新を提供します。 ありがとうございます.

2. 2026年9月2日 15:57 UTCmonitoring

   最近のデータベースメンテナンスに関連する決済処理(特定の購入、キャプチャ、決済方法作成)に影響を及ぼすパフォーマンスラグの解決に取り組んでいます。 修正は進行中であり、当社のチームは積極的にシステムを監視しています。 リアルタイムの更新についてはこちらをご覧ください.

3. 2026年9月2日 21:32 UTCresolved

   この事件は解決しました.

4. 2026年9月9日 21:50 UTCpostmortem

   ## 9月9日、2026 — 関連するエラー率のトランザクション処理に影響する ###ハイレベルサマリー 2026年9月2日、14:35〜15:19 UTCの間では、エンドポイント間でトランザクションリクエストのサブセットに影響を与える高度なエラー率の期間を優先的に経験しました。 問題は、約44分持続し、データベースプラットフォームでインフラストラクチャレベルの問題が発生しました。 私たちのチームは、同じ日に問題を識別し、解決しました。 ### 何が起こっているのか ルーチンスキーマの更新中は、データベースノードの1つが利用できなくなったデータベースプラットフォームに不具合が生じます。 これにより、トランザクションデータの部分の冗長性が一時的に低下し、通常処理の代わりに、システムエラー応答\(HTTP 500\)をタイムアウトまたは受信するリクエストがいくつか表示されます。 当社のチームは、データベースベンダーと直接連携し、根本的な原因を特定し、影響を受けるインフラの完全回復を確認します。 ## 次のステップ データベースベンダーのソフトウェアで既知の問題として根本的な原因を特定しました。 通常の定期メンテナンス中にベンダーによる修正が適用される予定です。 並行して、この問題のクラスを早期に検知し、データベースのメンテナンス作業に関する社内の手順を調整して、同様の事故のリスクをさらに低減します。 -Spreedlyチーム

公式のインシデント更新を自動翻訳しています。

1. 2026年8月14日 16:14 UTCinvestigating

   We are currently investigating an increase in 5xx errors on the Core Transactional API. Some transactions may be failing or returning unexpected error responses during this time. We will provide an update as soon as we have more information.

2. 2026年8月14日 16:34 UTCresolved

   A fix has been implemented and error rates have returned to normal. Some transactions may have failed or returned errors during the affected window. We're continuing to monitor and will follow up with a more detailed post-incident summary.

3. 2026年9月3日 20:55 UTCpostmortem

   ### Service Disruption on Transaction Endpoints ### ‌High Level Summary On August 14, 2026, Spreedly experienced a brief service disruption affecting some transaction-related API endpoints for approximately 20 minutes \(4:06 PM – 4:26 PM ET\). During this window, some requests to endpoints involving transaction lookups, captures, voids, credits, and payment method operations may have received error responses. The issue was identified and resolved quickly by our engineering team. ### What Happened A routine database schema update caused a subset of existing database connections to reference outdated schema information, resulting in intermittent errors on affected requests. Once identified, our team refreshed the affected database connections, which fully resolved the issue. ### Next Steps We are reviewing our schema migration process to add additional safeguards that will prevent this class of issue from recurring, and are improving automated testing coverage for this scenario. We have added auto-retry and recovery mechanisms so that if a connection does reference an outdated schema it will not affect transactions going forward.

1. 2026年7月27日 13:50 UTCinvestigating

   現在、この問題を調査中です.

2. 2026年7月27日 14:00 UTCidentified

   問題が特定され、修正が実装されています.

3. 2026年7月27日 15:00 UTCmonitoring

   コアトランザクションのパフォーマンスに影響を及ぼす可能性のある問題の早期発見。 問題が解決するようになりましたが、当社のチームは継続的な安定性を確保するためにシステムを積極的に監視しています。 調査をフルスコープにし、衝撃が完成したら最終更新を行います。 引き続き、StatusPageを最新のアップデートでフォローしてください.

4. 2026年7月27日 21:41 UTCresolved

   徹底的に監視し、すべてのシステムが安定化し、期待どおりの機能を確認した後、このインシデントは解決と見なされます。 さらなる顧客への影響は期待できません。 事故および残留衝撃の原因に関する調査を終了しました。 大変申し訳ございません.

5. 2026年9月3日 20:57 UTCpostmortem

   ###ハイレベルサマリー 2026年7月27日、購入、承認、Webhooksなどの取引リクエストのサブセットで、慎重に経験した断続的なエラー。 ### 何が起こっているのか データベースの移行は、保存された支払い方法を管理する内部サービスのスキーマを更新しました。 これは、階段キャッシュを参照する既存の接続上のエラーを引き起こしました。 変更をロールバックし、それらの接続をサイクルし、問題を完全に解決しました。 ## 次のステップ スキーマのマイグレーションプロセスを見直し、この問題のこのクラスが再発を防止する追加の保護策を追加し、このシナリオの自動テストカバレッジを改善しています。 接続が古いスキーマを参照していなければ、自動再試行と回復メカニズムが追加されました。トランザクションは前方には影響しません.

公式のインシデント更新を自動翻訳しています。

1. 2026年7月11日 14:33 UTCinvestigating

   特定の API リクエストに影響を及ぼす高いエラー率を調査しています。 一部のリストと検索エンドポイントは、root 原因を特定するために動作する間、断続的な 500 応答を返す可能性があります.

2. 2026年7月11日 15:38 UTCresolved

   特定のリストと検索エンドポイントに対する断続的な500応答を引き起こしている問題を特定し、解決しました。 リクエストが早期終了しないように設定更新が展開され、サービスの監視を継続しています.

公式のインシデント更新を自動翻訳しています。

1. 2026年7月8日 15:22 UTCinvestigating

   サイバーソース・クレジット取引において、業績が低下する可能性がある問題が明らかになっています。 この早期通知を可能な限り迅速に通知するよう努めていますが、実際のスコープと影響を調査しています。 詳細については、随時更新を行います.

2. 2026年7月8日 16:08 UTCidentified

   弊社では、15:05と15:05の間に発生したトランスエント内部のDNS解像度エラーを特定し、Spreedlyサービス全体で500個のエラーを選択しました。 今後も、インパクトを十分に調査し、追加情報を提供してまいります.

3. 2026年7月8日 16:54 UTCresolved

   徹底的に監視し、すべてのシステムが安定化し、期待どおりの機能を確認した後、このインシデントは解決と見なされます。 さらなる顧客への影響は期待できません。 事故および残留衝撃の原因に関する調査を終了しました。 大変申し訳ございません.

公式のインシデント更新を自動翻訳しています。

1. 2026年6月17日 14:55 UTCinvestigating

   Spreedly は、非同期トランザクションでパフォーマンスを劣化させる可能性のある問題を発見しました。 早期通知を可能な限り迅速に通知するよう努めていますが、実際のスコープとインパクトを調べ、詳細が利用可能になったらただちに更新を提供します。 ありがとうございます.

2. 2026年6月17日 15:00 UTCmonitoring

   非同期取引の修正と安定化を行いました。 私たちのチームは、追加の影響がないことを確実にするために、状況を密接に監視しています。 引き続き、StatusPageを最新のアップデートでフォローしてください.

3. 2026年6月17日 16:03 UTCmonitoring

   さらなる問題や追加の影響については、密接に監視しています。 引き続き、StatusPage を最新のアップデートでフォローしてください.

4. 2026年6月17日 21:42 UTCresolved

   監視期間を過ぎると、問題が解決され、システムが正常に動作していることが確認されています。 引き続き、問題が発生した場合は、サポートにお任せください.

5. 2026年6月24日 20:10 UTCpostmortem

   2026年6月24日 — 非同期トランザクション処理の中断 **ハイレベルサマリー** 2026年6月17日には、約10:02 AMから10:41 AM UTC、Spreedlyは非同期ゲートウェイ取引と3DS認証フローに影響を与える混乱を経験しました。 39分程度のウィンドウでは、確認された結果なしで、約1,634のトランザクションが保留状態に残り、その期間中の総取引量の1%未満を表しています。 **何が起こっているか** コード展開では、新規展開と以前に実行したアプリケーションインスタンス間での互換性の問題が導入されました。 ロールバックプロセス中に、いくつかのリクエストは混合されたコードバージョンにルーティングされ、非同期トランザクションステータスの更新を処理するときに失敗を引き起こしました。 Affectedトランザクションタイプには、非同期ゲートウェイトランザクション\(ゲートウェイ固有の3DSおよびその他の非同期検証、承認、購入\)と3DSグローバル認証が含まれます。 このウィンドウで影響を受けるフローに入ったトランザクションは、未定の結果を反映しているかもしれません。ゲートウェイのトランザクション状態は、Spreedlyレポートとは異なることがあります。 **次のステップ** 6 月 17 日に 10:02-10:41 AM UTC ウィンドウからトランザクションをレビューすることをお勧めします。 検証は安全に取得することができます。 正規化は、再試行する前にゲートウェイで無効にする必要があります。 購入は重複の危険性を持ち、再試行する前に返金をおすすめします。 影響を受けた取引の特定や適切な是正経路の決定が必要な場合は、Spreedly サポートにご連絡ください。 導入の安全性向上と自動ロールバックの検出により、問題のこのクラスを防止する取り組みを積極的に行っています。 -Spreedlyチーム

公式のインシデント更新を自動翻訳しています。

1. 2026年5月8日 18:00 UTCmonitoring

   A fix has been implemented and we are monitoring results.

2. 2026年5月8日 18:05 UTCresolved

   This incident has been resolved.

1. 2026年3月19日 15:13 UTCidentified

   We are currently investigating performance of the Dodgeball service.

2. 2026年3月19日 15:30 UTCidentified

   We are working to resolve an issue with delayed processing of requests to the Dodgeball services.

3. 2026年3月19日 15:55 UTCmonitoring

   We’ve implemented a fix and stabilized the Dodgeball component that was experiencing a partial outage. We are continuing to monitor

4. 2026年3月19日 16:07 UTCmonitoring

   After closely monitoring and confirming that all systems are stabilized and functioning as expected, this incident is considered resolved. No further customer impact is expected. We apologize for any inconvenience or disruption.

5. 2026年3月19日 16:09 UTCresolved

   This incident has been resolved.

1. 2026年3月16日 17:45 UTCinvestigating

   We are currently investigating elevated errors on API endpoints

2. 2026年3月16日 17:51 UTCmonitoring

   A fix has been implemented and we are monitoring to ensure services are operating correctly

3. 2026年3月16日 18:07 UTCresolved

   After closely monitoring and confirming that all systems are stabilized and functioning as expected, this incident is considered resolved. No further customer impact is expected. We are completing our investigation concerning the causes of the incident and any residual impact. We apologize for any inconvenience or disruption.

4. 2026年3月19日 19:07 UTCpostmortem

   ### Mar 18th 2026 — _Decryption Errors on API endpoints_ ### Summary Between 5:40 PM and 5:43 PM UTC on March 16, 2026, we experienced a brief issue during a key rotation that caused a small subset of API requests to fail due to decryption errors. These errors occurred within our system before any external processing took place. The issue was quickly identified, and traffic was immediately redirected to a stable environment, restoring normal service. ### What Happened At 5:36 PM UTC, we routed traffic to a newly deployed cluster of our encryption service running with the latest security keys. Shortly after, we observed decryption errors affecting a small subset of records, which resulted in some requests returning 500 error responses. We promptly redirected traffic back to the existing cluster, restoring normal operation. Further investigation determined that these records were still associated with a key in the process of rotation, and we updated them as a follow-up action to prevent the issue from recurring. ### Next Steps We have enhanced our process to ensure that a key’s usage is fully verified before it is removed from the encryption service. -The Spreedly Team

1. 2026年2月26日 16:45 UTCinvestigating

   We are currently investigating an issue affecting customers using Dodgeball. The production database backing Dodgeball has reached critically low available storage, which is expected to result in elevated error rates. Customers may begin experiencing errors within the next 10 minutes as the system’s ability to process requests degrades. Our engineering team is actively investigating and working to restore normal capacity as quickly as possible. We will provide further updates as more information becomes available.

2. 2026年2月26日 16:59 UTCmonitoring

   The engineering team has successfully resolved the storage issue affecting the Dodgeball production PostgreSQL instance. Capacity has been restored and the system is now operating normally. At this time, we are not observing any elevated error rates for customers using Dodgeball. We will continue to monitor the system closely to ensure stability. Further updates will be provided if necessary.

3. 2026年2月26日 18:44 UTCresolved

   The storage issue affecting the Dodgeball production instance has been fully resolved. Capacity has been restored and systems are operating normally. We are no longer observing elevated error rates for customers using Dodgeball. Thank you for your patience while we worked to resolve this issue.

1. 2026年2月25日 19:18 UTCinvestigating

   We are currently investigating performance of the Dodgeball service.

2. 2026年2月25日 19:48 UTCidentified

   We are working to resolve an issue with processing of Checkpoints that is causing latency and errors for requests to the Dodgeball services.

3. 2026年2月25日 20:35 UTCmonitoring

   Service has resumed and we are monitoring for further impact. We apologize for any inconvenience this has caused

4. 2026年2月25日 21:52 UTCresolved

   After closely monitoring and confirming that all systems are stabilized and functioning as expected, this incident is considered resolved. No further customer impact is expected. We are completing our investigation concerning the causes of the incident and any residual impact. We apologize for any inconvenience or disruption.

1. 2026年2月5日 15:56 UTCinvestigating

   We are currently investigating an issue where credit transactions referencing Capture transactions are failing in production. Some customers may see errors when attempting to issue credits against Capture transactions. Capture transactions themselves are processing successfully. Our engineering team is actively investigating this behavior to determine the cause and confirm whether this is a regression or an unintended change. We will provide updates as soon as more information is available.

2. 2026年2月5日 16:25 UTCmonitoring

   We’ve deployed a revert to address the issue where credit transactions referencing Capture transactions were failing. We’re actively monitoring to ensure credit requests referencing Captures are processing as expected. At this time, we’re not seeing further errors related to this behavior.

3. 2026年2月5日 17:01 UTCresolved

   After monitoring the system post-deploy, we’re no longer seeing errors related to credits referencing Capture transactions, and processing has returned to normal. Thank you for your patience.

4. 2026年2月10日 17:22 UTCpostmortem

   ### February 10, 2026 — Credits Referencing Capture Transactions Failing ### ‌High Level Summary Between approximately 1:56 PM UTC and 4:15 PM UTC on February 5, 2026, credit transactions where the reference transaction was a capture failed with the error "The specified `reference_token` corresponds to a transaction which cannot be used as a reference." ### What Happened At approximately 1:56 PM UTC on February 5, 2026, Spreedly rolled out a change to support credits on the CyberSource Rest gateway. This included a change in the way we validated reference transaction types used in credits. Specifically, there was an unintended side-effect which caused captures to no longer be treated as a valid reference type for credits on gateways other than CyberSource Rest. Once this was discovered, the code was reverted at 4:01PM UTC. By 4:15 PM UTC, credit requests against captures were fully functional. ### Next Steps * Any credit attempts that failed with this error are safe to retry * We are making improvements to our code review processes and regression test suite to ensure we catch similar errors before they make it into production * We have identified a gap in our observability surrounding validation errors that we will be prioritizing ‌ -The Spreedly Team

1. 2026年2月4日 18:15 UTCinvestigating

   Spreedly has detected an issue that is affecting multiple customers using Payflow Pro where some transactions fail with the error: “Invalid Request. Malicious string found in XML document”. While we are providing an early notification in an effort to alert you as quickly as possible, we are still investigating the actual scope and impact and will provide an update as soon as more details are available. Thank you for your patience.

2. 2026年2月4日 19:52 UTCmonitoring

   We are no longer seeing errors at this time, and the issue appears to have subsided. We are working with Payflow Pro to identify the root cause. Our team continues to monitor the situation closely to ensure stability. Please continue to follow our StatusPage for the latest updates.

3. 2026年2月4日 19:56 UTCresolved

   This incident has been resolved.

1. 2026年1月27日 19:59 UTCinvestigating

   Spreedly has detected an issue related to a dodgeball component experiencing a partial outage. While we are providing an early notification in an effort to alert you as quickly as possible, we are still investigating the actual scope and impact and will provide an update as soon as more details are available. Thank you for your patience.

2. 2026年1月27日 20:12 UTCresolved

   We’ve implemented a fix and stabilized the Dodgeball component that was experiencing a partial outage. After closely monitoring and confirming that all systems are stabilized and functioning as expected, this incident is considered resolved. No further customer impact is expected. We apologize for any inconvenience or disruption.

1. 2026年1月27日 19:08 UTCinvestigating

   Spreedly has detected an issue potentially resulting in a high number of pending 3DS2 Global transactions. While we are providing an early notification in an effort to alert you as quickly as possible, we are still investigating the actual scope and impact and will provide an update as soon as more details are available. Thank you for your patience.

2. 2026年1月27日 19:25 UTCmonitoring

   We’ve implemented a fix and stabilized the 3DS2 Global transactions. Our team is monitoring the situation closely to ensure there are no additional impacts. Please continue to follow our StatusPage for the latest update.

3. 2026年1月27日 19:29 UTCresolved

   After closely monitoring and confirming that all systems are stabilized and functioning as expected, this incident is considered resolved. No further customer impact is expected. We are completing our investigation concerning the causes of the incident and any residual impact. We apologize for any inconvenience or disruption.

4. 2026年2月3日 15:38 UTCpostmortem

   ### February 3rd, 2026 — 3DS2 Global Transactions Pending ‌ Between 5:40 pm UTC and 7:24 PM UTC on January 27, 2026 the latest version of the Spreedly iFrame \(`iframe-v1`\) was configured to use version 1.180 which contained a bug related to the handling of device fingerprinting for `Sca::Authentications` in an end user's browser. As a result there was an elevated number of `Sca::Authentications` stuck in a pending flow with a `required_action` of `device_fingerprint`. These transactions were held in a pending state and never completed. Our team identified the issue and updated the version of iFrame \(`iframe-v1`\) to utilize `1.179`. ### What Happened At 5:40 pm UTC on January 27th, 2026 a new version of iFrame was released \(1.180\) and updated the version to be used in the `iframe-v1.min.js` asset to point to `1.180`. This change was meant to improve the error handling for the loading of 3rd party critical SDKs however it caused the SDK to enter a permanently error-ed state when attempting to perform device fingerprinting due to a race condition with loading the SDKs. At 7:20 pm UTC on January 27th, 2026 the change was rolled back and by 7:24 pm UTC, 2026 the `iframe-v1.min.js` asset globally was pointing to version `1.179`. ### Next Steps We have identified a gap in our observability and alerting within the iFrame service related to the managing of 3DS lifecycle flows that we will be prioritizing. -The Spreedly Team

1. 2026年1月9日 18:46 UTCinvestigating

   We are currently investigating this issue.

2. 2026年1月9日 18:52 UTCinvestigating

   We are continuing to investigate this issue.

3. 2026年1月9日 19:07 UTCmonitoring

   A fix has been deployed and we are seeing systems recover. We will continue to monitor to ensure stability and will provide further updates as needed.

4. 2026年1月9日 19:10 UTCresolved

   After a period of monitoring, we have confirmed that the issue has been resolved and systems are operating normally. If you continue to experience any issues, please reach out to Support.

1. 2025年12月29日 16:50 UTCresolved

   Between 7:00 PM EST on Saturday, December 27th, and 1:00 AM EST on Sunday, December 28th, we identified a period of higher-than-normal latency caused by a pre-planned customer event that resulted in a significant traffic spike. Although we did not observe any service failures or timeouts, our systems experienced degraded service due to the increased latency. Systems recovered quickly once peak traffic subsided and are now operating normally. We expect performance to remain stable moving forward.

1. 2025年12月16日 21:56 UTCresolved

   Between 3:59 and 4:03 EST, customers may have experienced degraded performance when processing transactions that required 3D Secure (3DS) authentication via our Global 3DS service. Performance has since returned to normal, and no further impact has been observed. If you have any questions about potential impact to your transactions during this time window, please reach out to Spreedly Support for assistance.

2. 2025年12月18日 21:26 UTCpostmortem

   # _December 18, 2025 — Delayed Processing of Some 3DS Global Transactions_ ‌ _A targeted update intended for Gateway-Specific 3DS processing impacted a subset of 3DS Global transactions. This caused a small number of transactions to remain in a "Pending" state rather than progressing to completion._ ### What Happened Between 08:59 PM and 09:03 PM UTC on December 16, 2025, a deployment intended to update Gateway-Specific 3DS logic inadvertently affected the processing of 3DS Global transactions. As a result, 52 transactions failed to reach a final state and remained "pending." During this 4-minute window, customers attempting a 3DS-authenticated operation would have encountered an interruption, preventing their transactions from completing. Our team identified the impact immediately and rolled back the change to restore normal service. ### Next Steps _We have successfully identified and corrected the underlying logic error within the change. We are also enhancing our automated deployment tests to better catch cross-module dependencies of this nature. We sincerely apologize for any disruption this caused to your operations._ ‌ — The Spreedly Team

1. 2025年12月9日 15:58 UTCinvestigating

   We are investigating an issue where roughly 30–40% of transactions are not appearing in Spreedly reporting since 12/03. Primary datastores and secondary services show no data loss, indicating the issue lies within the reporting ingestion process. Please reach out to Spreedly Support if you need assistance.

2. 2025年12月10日 6:53 UTCinvestigating

   Our team continues to investigate the root cause of the missing transaction data appearing in Spreedly reporting. All primary datastores and transactional processing are functioning normally, and there is no indication of data loss at the source. The issue appears to be isolated to the reporting ingestion process.

3. 2025年12月10日 14:42 UTCmonitoring

   We have identified and fixed the issue that was causing some transaction data to be missing from Spreedly reporting. Our team is now working on backfilling the missing reporting data to ensure completeness and accuracy. All primary processing systems continue to operate normally, and there is no impact to transaction processing.

4. 2025年12月10日 20:04 UTCresolved

   Our team has completed the backfill process, and all missing transaction data has now been fully restored in Spreedly reporting. Transaction reporting is up to date and functioning as expected. All primary processing systems continued to operate normally throughout this incident, and no transaction processing impact occurred.

5. 2026年1月2日 21:44 UTCpostmortem

   ### _Postmortem: January 2, 2026_ — _Transaction Reporting Degraded_ ### ‌High Level Summary Select data related to transactions did not make it into our downstream systems, causing inaccurate reports in [app.spreedly.com](http://app.spreedly.com) ### What Happened After completing system updates on December 3, 2025, approximately 30-40% of data related to transactions did not make it into our reporting tools due to a latent bug in one of our supporting applications. This caused inaccurate reports in [app.spreedly.com](http://app.spreedly.com). This did not impact any primary system functionality and no data was lost. ### Next Steps * Our team has since identified and fixed the affected application. The missing data was backfilled and subsequent data continues to flow normally * We are adding additional monitoring in order to more quickly flag discrepancies * We are continuing to make updates to simplify our data pipeline that will prevent similar issues in the future -The Spreedly Team

1. 2025年10月20日 15:41 UTCidentified

   The major AWS outage on October 20, 2025, has been largely resolved for most affected services, though some residual issues may persist. Spreedly’s core platform remains stable and fully available. However, due to downstream impact from the AWS outage, some of Spreedly’s indexing endpoints and callback processing are experiencing delays. This is related to dependencies on third-party systems that were affected by the outage and are still undergoing recovery. As a result, you may continue to see intermittent latency or delayed responses for API requests involving these systems. We are actively monitoring the situation as AWS and its downstream services continue recovery efforts. For ongoing updates, please refer to the AWS status page here

2. 2025年10月20日 20:22 UTCresolved

   After closely monitoring and confirming that all systems are stabilized and functioning as expected, this incident is considered resolved. No further customer impact is expected.

3. 2025年10月23日 18:27 UTCpostmortem

   ### Postmortem: October 24, 2025 — _AWS outage_ Monday's AWS outage had downstream impact on some 3rd-party systems utilized by Spreedly's platform. Namely, our indexing endpoints and callbacks lagged behind due to the outage and recovery steps. ### What Happened Spreedly's 3DS global service and legacy smart routing product both faced a period of failures due to 3rd-party systems relying on AWS. ### Next Steps Spreedly will improve its response to our alerts platform, and are actively working on multi-region support slated for 2026, which should mitigate the impact of similar events going forward. -The Spreedly Team