Spreedly 中断历史

1. 2026年9月2日 UTC 15:11investigating

   Spreedly检测出一个可能导致交易性API性能下降的问题. 我们正在提供早期通知,以尽快通知你们,与此同时,我们仍在调查实际的范围和影响,并将在获得更多细节后立即提供最新情况。 谢谢你的耐心.

2. 2026年9月2日 UTC 15:57monitoring

   我们正在努力解决与近期数据库维护相关的一些支付处理(具体来说是购买、捕获和支付方法的创建)方面的业绩滞后问题。 我们的团队正在积极监测这个系统。 请回到这里 实时更新.

3. 2026年9月2日 UTC 21:32resolved

   这一事件已经得到解决.

4. 2026年9月9日 UTC 21:50postmortem

   9月9日 高级摘要 2026年9月2日,在协调世界时14:35至15:19之间,斯普雷德利经历了一段错误率高涨的时期,影响到一个子集跨端点的交易请求. 这个问题持续了大约44分钟,是我国数据库平台的基础设施问题造成的。 我们的小组同一天确定并解决了这个问题。 发生了什么事? 在例行更新计划期间,我们数据库平台的一个缺陷导致我们数据库的一个节点无法使用。 这暂时减少了我们部分交易数据的冗余,导致一些请求要么超时,要么收到系统出错响应\(HTTP 500\)而不是正常处理. 我们的小组同我们的数据库供应商直接合作,查明根源,确认受影响基础设施的全面恢复。 下一个步骤 我们在数据库供应商的软件中 找出了潜在的原因 计划在正常预定的维修期间使用供应商提供的固定设备。 与此同时,我们正在增加额外监测,以更早地发现这一类问题,并调整围绕数据库维护工作的内部程序,以进一步减少类似事件发生的风险。 - 间谍队

自动翻译自官方事件更新。

1. 2026年8月14日 UTC 16:14investigating

   We are currently investigating an increase in 5xx errors on the Core Transactional API. Some transactions may be failing or returning unexpected error responses during this time. We will provide an update as soon as we have more information.

2. 2026年8月14日 UTC 16:34resolved

   A fix has been implemented and error rates have returned to normal. Some transactions may have failed or returned errors during the affected window. We're continuing to monitor and will follow up with a more detailed post-incident summary.

3. 2026年9月3日 UTC 20:55postmortem

   ### Service Disruption on Transaction Endpoints ### ‌High Level Summary On August 14, 2026, Spreedly experienced a brief service disruption affecting some transaction-related API endpoints for approximately 20 minutes \(4:06 PM – 4:26 PM ET\). During this window, some requests to endpoints involving transaction lookups, captures, voids, credits, and payment method operations may have received error responses. The issue was identified and resolved quickly by our engineering team. ### What Happened A routine database schema update caused a subset of existing database connections to reference outdated schema information, resulting in intermittent errors on affected requests. Once identified, our team refreshed the affected database connections, which fully resolved the issue. ### Next Steps We are reviewing our schema migration process to add additional safeguards that will prevent this class of issue from recurring, and are improving automated testing coverage for this scenario. We have added auto-retry and recovery mechanisms so that if a connection does reference an outdated schema it will not affect transactions going forward.

1. 2026年7月27日 UTC 13:50investigating

   我们目前正在调查这一问题.

2. 2026年7月27日 UTC 14:00identified

   这一问题已经确定,一个解决办法正在实施之中.

3. 2026年7月27日 UTC 15:00monitoring

   迅速发现一个可能影响核心交易业绩的问题。 虽然这个问题现在似乎已经解决,但我们的小组正在积极监测制度,以确保持续稳定。 一旦完成对全部范围和影响的调查,我们将提供最后最新情况。 请继续沿用我们的状态Page以获取最新消息.

4. 2026年7月27日 UTC 21:41resolved

   在密切监测和确认所有系统都按预期稳定和运作之后,认为这一事件已得到解决。 预计不会对客户产生进一步的影响。 我们正在完成对事件起因和任何剩余影响的调查。 我们为任何不便或干扰道歉.

5. 2026年9月3日 UTC 20:57postmortem

   * 高级别摘要 2026年7月27日,斯普雷德利在包括购买,授权,和webhooks在内的一组交易请求上相继出错. 发生了什么事? 数据库迁移更新了管理存储支付方法的内部服务的计划。 这导致对现有连接引用一个 stale 缓存时出现错误 。 我们扭转了变化,循环了这些联系,充分解决了这一问题。 下一个步骤 我们正在审查我们的移徙计划,以增加额外的保障措施,防止这种类型的问题再次发生,并正在扩大这种情形的自动化测试范围。 我们增加了自动重试和回收机制,以便如果连接提到过时的计划,不会影响今后的交易.

自动翻译自官方事件更新。

1. 2026年7月11日 UTC 14:33investigating

   我们正在调查影响某些API请求的高误差率. 一些列表和检索端点可能回回断断续续的500个回复,同时我们努力找出根源.

2. 2026年7月11日 UTC 15:38resolved

   我们查明并解决了这一问题,导致在清单和检索终点上出现间歇500个答复。 已部署最新配置,以防止提前终止请求,我们正在继续监测这项服务.

自动翻译自官方事件更新。

1. 2026年7月8日 UTC 15:22investigating

   Spreedly发现了一个可能导致网络源信贷交易业绩下降的问题。 我们提供这份早期通知是为了尽快提醒你,但我们仍在调查实际的范围和影响。 一旦掌握更多细节,我们将立即提供最新情况.

2. 2026年7月8日 UTC 16:08identified

   我们发现在协调世界时下午3:00至15:05之间发生的瞬间DNS解析出错,在斯普雷德利服务中造成500个选择出错. 我们继续调查其全部影响,并将在获得补充资料后提供补充资料.

3. 2026年7月8日 UTC 16:54resolved

   在密切监测和确认所有系统都按预期稳定和运作之后,认为这一事件已得到解决。 预计不会对客户产生进一步的影响。 我们正在完成对事件起因和任何剩余影响的调查。 我们为任何不便或干扰道歉.

自动翻译自官方事件更新。

1. 2026年6月17日 UTC 14:55investigating

   Spreedly检测出一个问题,它有可能与同步交易产生装饰性能. 我们正在提供早期通知,以尽快通知你们,与此同时,我们仍在调查实际的范围和影响,并将在获得更多细节后立即提供最新情况。 谢谢你的耐心.

2. 2026年6月17日 UTC 15:00monitoring

   并稳定了同步交易。 我们的小组正在密切监测局势,以确保没有其他影响。 请继续沿用我们的状态Page以获取最新消息.

3. 2026年6月17日 UTC 16:03monitoring

   我们正在密切监测任何其他问题或其他影响。 请继续沿用我们的最新消息.

4. 2026年6月17日 UTC 21:42resolved

   经过一段时间的监测,我们确认这个问题已经解决,系统正在正常运行。 如果你继续遇到任何问题,请联系支助.

5. 2026年6月24日 UTC 20:10postmortem

   2026年6月24日——同步交易处理中断. ** 高级别摘要** 2026年6月17日,从大约10:02AM到10:41AM UTC,斯普雷德利经历了一次干扰,影响了同步网关交易和3DS认证流量. 在39分钟的窗口中,约有1 634个交易处于未确认结果的待决状态,占这一期间交易总量的不到1%。 ** 发生了什么事** 代码部署引入了新部署和以前运行的应用程序实例之间的兼容性问题. 在回滚过程中,一些请求通过混合代码版本被路由,导致处理同步交易状态更新时失败. 受影响的交易类型包括同步网关交易\(通道特定3DS,以及其它同步验证,授权和购买\)和3DS Global认证. 在此窗口中输入受影响流量的交易可能反映不确定的结果, 您网关上的交易状态可能与斯普雷德利报告的不同 。 ** 下一步** 我们建议审查6月17日协调世界时10:02-10:41的AM窗口交易, 核查可以安全地重新进行。 重新试验前, 应在网关上取消授权 。 采购有重复的风险,我们建议在重新试验之前退货。 如果您需要帮助确定受影响的交易或确定适当的补救路径,请联系Spreedly支持。 我们正在积极努力,通过改进部署保障和自动回滚探测,防止这类问题。 - 间谍队

自动翻译自官方事件更新。

1. 2026年5月8日 UTC 18:00monitoring

   A fix has been implemented and we are monitoring results.

2. 2026年5月8日 UTC 18:05resolved

   This incident has been resolved.

1. 2026年3月19日 UTC 15:13identified

   We are currently investigating performance of the Dodgeball service.

2. 2026年3月19日 UTC 15:30identified

   We are working to resolve an issue with delayed processing of requests to the Dodgeball services.

3. 2026年3月19日 UTC 15:55monitoring

   We’ve implemented a fix and stabilized the Dodgeball component that was experiencing a partial outage. We are continuing to monitor

4. 2026年3月19日 UTC 16:07monitoring

   After closely monitoring and confirming that all systems are stabilized and functioning as expected, this incident is considered resolved. No further customer impact is expected. We apologize for any inconvenience or disruption.

5. 2026年3月19日 UTC 16:09resolved

   This incident has been resolved.

1. 2026年3月16日 UTC 17:45investigating

   We are currently investigating elevated errors on API endpoints

2. 2026年3月16日 UTC 17:51monitoring

   A fix has been implemented and we are monitoring to ensure services are operating correctly

3. 2026年3月16日 UTC 18:07resolved

   After closely monitoring and confirming that all systems are stabilized and functioning as expected, this incident is considered resolved. No further customer impact is expected. We are completing our investigation concerning the causes of the incident and any residual impact. We apologize for any inconvenience or disruption.

4. 2026年3月19日 UTC 19:07postmortem

   ### Mar 18th 2026 — _Decryption Errors on API endpoints_ ### Summary Between 5:40 PM and 5:43 PM UTC on March 16, 2026, we experienced a brief issue during a key rotation that caused a small subset of API requests to fail due to decryption errors. These errors occurred within our system before any external processing took place. The issue was quickly identified, and traffic was immediately redirected to a stable environment, restoring normal service. ### What Happened At 5:36 PM UTC, we routed traffic to a newly deployed cluster of our encryption service running with the latest security keys. Shortly after, we observed decryption errors affecting a small subset of records, which resulted in some requests returning 500 error responses. We promptly redirected traffic back to the existing cluster, restoring normal operation. Further investigation determined that these records were still associated with a key in the process of rotation, and we updated them as a follow-up action to prevent the issue from recurring. ### Next Steps We have enhanced our process to ensure that a key’s usage is fully verified before it is removed from the encryption service. -The Spreedly Team

1. 2026年2月26日 UTC 16:45investigating

   We are currently investigating an issue affecting customers using Dodgeball. The production database backing Dodgeball has reached critically low available storage, which is expected to result in elevated error rates. Customers may begin experiencing errors within the next 10 minutes as the system’s ability to process requests degrades. Our engineering team is actively investigating and working to restore normal capacity as quickly as possible. We will provide further updates as more information becomes available.

2. 2026年2月26日 UTC 16:59monitoring

   The engineering team has successfully resolved the storage issue affecting the Dodgeball production PostgreSQL instance. Capacity has been restored and the system is now operating normally. At this time, we are not observing any elevated error rates for customers using Dodgeball. We will continue to monitor the system closely to ensure stability. Further updates will be provided if necessary.

3. 2026年2月26日 UTC 18:44resolved

   The storage issue affecting the Dodgeball production instance has been fully resolved. Capacity has been restored and systems are operating normally. We are no longer observing elevated error rates for customers using Dodgeball. Thank you for your patience while we worked to resolve this issue.

1. 2026年2月25日 UTC 19:18investigating

   We are currently investigating performance of the Dodgeball service.

2. 2026年2月25日 UTC 19:48identified

   We are working to resolve an issue with processing of Checkpoints that is causing latency and errors for requests to the Dodgeball services.

3. 2026年2月25日 UTC 20:35monitoring

   Service has resumed and we are monitoring for further impact. We apologize for any inconvenience this has caused

4. 2026年2月25日 UTC 21:52resolved

   After closely monitoring and confirming that all systems are stabilized and functioning as expected, this incident is considered resolved. No further customer impact is expected. We are completing our investigation concerning the causes of the incident and any residual impact. We apologize for any inconvenience or disruption.

1. 2026年2月5日 UTC 15:56investigating

   We are currently investigating an issue where credit transactions referencing Capture transactions are failing in production. Some customers may see errors when attempting to issue credits against Capture transactions. Capture transactions themselves are processing successfully. Our engineering team is actively investigating this behavior to determine the cause and confirm whether this is a regression or an unintended change. We will provide updates as soon as more information is available.

2. 2026年2月5日 UTC 16:25monitoring

   We’ve deployed a revert to address the issue where credit transactions referencing Capture transactions were failing. We’re actively monitoring to ensure credit requests referencing Captures are processing as expected. At this time, we’re not seeing further errors related to this behavior.

3. 2026年2月5日 UTC 17:01resolved

   After monitoring the system post-deploy, we’re no longer seeing errors related to credits referencing Capture transactions, and processing has returned to normal. Thank you for your patience.

4. 2026年2月10日 UTC 17:22postmortem

   ### February 10, 2026 — Credits Referencing Capture Transactions Failing ### ‌High Level Summary Between approximately 1:56 PM UTC and 4:15 PM UTC on February 5, 2026, credit transactions where the reference transaction was a capture failed with the error "The specified `reference_token` corresponds to a transaction which cannot be used as a reference." ### What Happened At approximately 1:56 PM UTC on February 5, 2026, Spreedly rolled out a change to support credits on the CyberSource Rest gateway. This included a change in the way we validated reference transaction types used in credits. Specifically, there was an unintended side-effect which caused captures to no longer be treated as a valid reference type for credits on gateways other than CyberSource Rest. Once this was discovered, the code was reverted at 4:01PM UTC. By 4:15 PM UTC, credit requests against captures were fully functional. ### Next Steps * Any credit attempts that failed with this error are safe to retry * We are making improvements to our code review processes and regression test suite to ensure we catch similar errors before they make it into production * We have identified a gap in our observability surrounding validation errors that we will be prioritizing ‌ -The Spreedly Team

1. 2026年2月4日 UTC 18:15investigating

   Spreedly has detected an issue that is affecting multiple customers using Payflow Pro where some transactions fail with the error: “Invalid Request. Malicious string found in XML document”. While we are providing an early notification in an effort to alert you as quickly as possible, we are still investigating the actual scope and impact and will provide an update as soon as more details are available. Thank you for your patience.

2. 2026年2月4日 UTC 19:52monitoring

   We are no longer seeing errors at this time, and the issue appears to have subsided. We are working with Payflow Pro to identify the root cause. Our team continues to monitor the situation closely to ensure stability. Please continue to follow our StatusPage for the latest updates.

3. 2026年2月4日 UTC 19:56resolved

   This incident has been resolved.

1. 2026年1月27日 UTC 19:59investigating

   Spreedly has detected an issue related to a dodgeball component experiencing a partial outage. While we are providing an early notification in an effort to alert you as quickly as possible, we are still investigating the actual scope and impact and will provide an update as soon as more details are available. Thank you for your patience.

2. 2026年1月27日 UTC 20:12resolved

   We’ve implemented a fix and stabilized the Dodgeball component that was experiencing a partial outage. After closely monitoring and confirming that all systems are stabilized and functioning as expected, this incident is considered resolved. No further customer impact is expected. We apologize for any inconvenience or disruption.

1. 2026年1月27日 UTC 19:08investigating

   Spreedly has detected an issue potentially resulting in a high number of pending 3DS2 Global transactions. While we are providing an early notification in an effort to alert you as quickly as possible, we are still investigating the actual scope and impact and will provide an update as soon as more details are available. Thank you for your patience.

2. 2026年1月27日 UTC 19:25monitoring

   We’ve implemented a fix and stabilized the 3DS2 Global transactions. Our team is monitoring the situation closely to ensure there are no additional impacts. Please continue to follow our StatusPage for the latest update.

3. 2026年1月27日 UTC 19:29resolved

   After closely monitoring and confirming that all systems are stabilized and functioning as expected, this incident is considered resolved. No further customer impact is expected. We are completing our investigation concerning the causes of the incident and any residual impact. We apologize for any inconvenience or disruption.

4. 2026年2月3日 UTC 15:38postmortem

   ### February 3rd, 2026 — 3DS2 Global Transactions Pending ‌ Between 5:40 pm UTC and 7:24 PM UTC on January 27, 2026 the latest version of the Spreedly iFrame \(`iframe-v1`\) was configured to use version 1.180 which contained a bug related to the handling of device fingerprinting for `Sca::Authentications` in an end user's browser. As a result there was an elevated number of `Sca::Authentications` stuck in a pending flow with a `required_action` of `device_fingerprint`. These transactions were held in a pending state and never completed. Our team identified the issue and updated the version of iFrame \(`iframe-v1`\) to utilize `1.179`. ### What Happened At 5:40 pm UTC on January 27th, 2026 a new version of iFrame was released \(1.180\) and updated the version to be used in the `iframe-v1.min.js` asset to point to `1.180`. This change was meant to improve the error handling for the loading of 3rd party critical SDKs however it caused the SDK to enter a permanently error-ed state when attempting to perform device fingerprinting due to a race condition with loading the SDKs. At 7:20 pm UTC on January 27th, 2026 the change was rolled back and by 7:24 pm UTC, 2026 the `iframe-v1.min.js` asset globally was pointing to version `1.179`. ### Next Steps We have identified a gap in our observability and alerting within the iFrame service related to the managing of 3DS lifecycle flows that we will be prioritizing. -The Spreedly Team

1. 2026年1月9日 UTC 18:46investigating

   We are currently investigating this issue.

2. 2026年1月9日 UTC 18:52investigating

   We are continuing to investigate this issue.

3. 2026年1月9日 UTC 19:07monitoring

   A fix has been deployed and we are seeing systems recover. We will continue to monitor to ensure stability and will provide further updates as needed.

4. 2026年1月9日 UTC 19:10resolved

   After a period of monitoring, we have confirmed that the issue has been resolved and systems are operating normally. If you continue to experience any issues, please reach out to Support.

1. 2025年12月29日 UTC 16:50resolved

   Between 7:00 PM EST on Saturday, December 27th, and 1:00 AM EST on Sunday, December 28th, we identified a period of higher-than-normal latency caused by a pre-planned customer event that resulted in a significant traffic spike. Although we did not observe any service failures or timeouts, our systems experienced degraded service due to the increased latency. Systems recovered quickly once peak traffic subsided and are now operating normally. We expect performance to remain stable moving forward.

1. 2025年12月16日 UTC 21:56resolved

   Between 3:59 and 4:03 EST, customers may have experienced degraded performance when processing transactions that required 3D Secure (3DS) authentication via our Global 3DS service. Performance has since returned to normal, and no further impact has been observed. If you have any questions about potential impact to your transactions during this time window, please reach out to Spreedly Support for assistance.

2. 2025年12月18日 UTC 21:26postmortem

   # _December 18, 2025 — Delayed Processing of Some 3DS Global Transactions_ ‌ _A targeted update intended for Gateway-Specific 3DS processing impacted a subset of 3DS Global transactions. This caused a small number of transactions to remain in a "Pending" state rather than progressing to completion._ ### What Happened Between 08:59 PM and 09:03 PM UTC on December 16, 2025, a deployment intended to update Gateway-Specific 3DS logic inadvertently affected the processing of 3DS Global transactions. As a result, 52 transactions failed to reach a final state and remained "pending." During this 4-minute window, customers attempting a 3DS-authenticated operation would have encountered an interruption, preventing their transactions from completing. Our team identified the impact immediately and rolled back the change to restore normal service. ### Next Steps _We have successfully identified and corrected the underlying logic error within the change. We are also enhancing our automated deployment tests to better catch cross-module dependencies of this nature. We sincerely apologize for any disruption this caused to your operations._ ‌ — The Spreedly Team

1. 2025年12月9日 UTC 15:58investigating

   We are investigating an issue where roughly 30–40% of transactions are not appearing in Spreedly reporting since 12/03. Primary datastores and secondary services show no data loss, indicating the issue lies within the reporting ingestion process. Please reach out to Spreedly Support if you need assistance.

2. 2025年12月10日 UTC 06:53investigating

   Our team continues to investigate the root cause of the missing transaction data appearing in Spreedly reporting. All primary datastores and transactional processing are functioning normally, and there is no indication of data loss at the source. The issue appears to be isolated to the reporting ingestion process.

3. 2025年12月10日 UTC 14:42monitoring

   We have identified and fixed the issue that was causing some transaction data to be missing from Spreedly reporting. Our team is now working on backfilling the missing reporting data to ensure completeness and accuracy. All primary processing systems continue to operate normally, and there is no impact to transaction processing.

4. 2025年12月10日 UTC 20:04resolved

   Our team has completed the backfill process, and all missing transaction data has now been fully restored in Spreedly reporting. Transaction reporting is up to date and functioning as expected. All primary processing systems continued to operate normally throughout this incident, and no transaction processing impact occurred.

5. 2026年1月2日 UTC 21:44postmortem

   ### _Postmortem: January 2, 2026_ — _Transaction Reporting Degraded_ ### ‌High Level Summary Select data related to transactions did not make it into our downstream systems, causing inaccurate reports in [app.spreedly.com](http://app.spreedly.com) ### What Happened After completing system updates on December 3, 2025, approximately 30-40% of data related to transactions did not make it into our reporting tools due to a latent bug in one of our supporting applications. This caused inaccurate reports in [app.spreedly.com](http://app.spreedly.com). This did not impact any primary system functionality and no data was lost. ### Next Steps * Our team has since identified and fixed the affected application. The missing data was backfilled and subsequent data continues to flow normally * We are adding additional monitoring in order to more quickly flag discrepancies * We are continuing to make updates to simplify our data pipeline that will prevent similar issues in the future -The Spreedly Team

1. 2025年10月20日 UTC 15:41identified

   The major AWS outage on October 20, 2025, has been largely resolved for most affected services, though some residual issues may persist. Spreedly’s core platform remains stable and fully available. However, due to downstream impact from the AWS outage, some of Spreedly’s indexing endpoints and callback processing are experiencing delays. This is related to dependencies on third-party systems that were affected by the outage and are still undergoing recovery. As a result, you may continue to see intermittent latency or delayed responses for API requests involving these systems. We are actively monitoring the situation as AWS and its downstream services continue recovery efforts. For ongoing updates, please refer to the AWS status page here

2. 2025年10月20日 UTC 20:22resolved

   After closely monitoring and confirming that all systems are stabilized and functioning as expected, this incident is considered resolved. No further customer impact is expected.

3. 2025年10月23日 UTC 18:27postmortem

   ### Postmortem: October 24, 2025 — _AWS outage_ Monday's AWS outage had downstream impact on some 3rd-party systems utilized by Spreedly's platform. Namely, our indexing endpoints and callbacks lagged behind due to the outage and recovery steps. ### What Happened Spreedly's 3DS global service and legacy smart routing product both faced a period of failures due to 3rd-party systems relying on AWS. ### Next Steps Spreedly will improve its response to our alerts platform, and are actively working on multi-region support slated for 2026, which should mitigate the impact of similar events going forward. -The Spreedly Team