Pricing
Status historyRSS updatesStatus API

Powered by Uptimus

    Insided 中断历史

    54 Insided incidents · 2023年5月 — official updates, affected components, duration and resolution details.

    返回当前状态

    PHP 脆弱性宣布 - 上传能力已禁用

    开始时间 2026年10月21日 UTC 11:53 · 进行中

    Pending
    1. 2026年10月21日 UTC 11:53identified

      作为预防措施,我们暂时取消了社区上传功能。 在此期间,成员将无法上传文件,包括内置图像,剖面图等,并张贴附件. 所有其他社区功能,包括阅读、张贴和答复,都不受影响。 我们没有看到对客户数据或社区可用性有任何影响。 我们期望在工作完成后恢复上传功能,一旦获得更多信息,我们将立即更新这一页面.

    2. 2026年10月21日 UTC 12:40identified

      我们正在继续密切监测该平台。 我们的调查没有发现任何证据证明我们正在处理的安全问题被利用了。 作为防范措施,文件上传仍然暂时失效,我们正在努力尽快恢复。 我们将随着形势的发展不断更新这一页.

    3. 2026年10月21日 UTC 13:00identified

      PHP 脆弱性( PHP2 Shell): 上传暂时禁用 Gainsight客户社区使用PHP. 10月20日,PHP本身的两个关键弱点(CVE-2026-109618和CVE-2026-109619,简称PHP2Shell)被披露在正式固定之前. PHP安全团队今日10月21日16:00CEST(协调世界时14:00)发布定点. 我们所做的: 禁用所有社区的所有文件上传,包括内置图像,剖面图和后附. 针对这些攻击中使用的请求类型设置额外的防火墙保护. 检查了我们的日志 发现开采的迹象。 目前我们还没有找到任何证据, 接下来会发生什么: 一旦发布, 我们就会在EMEA和美国的基础设施上应用PHP安全更新。 并分享结果。 一旦补丁被应用和核实,我们将恢复上传。 正常地阅读、张贴和答复工作。 我们下一次更新时间是17:00 CEST,如果有任何变化,则提前进行.

    4. 2026年10月21日 UTC 14:07identified

      我们正在继续努力解决这个问题。

    5. 2026年10月21日 UTC 14:08identified

      更新:PHP安全团队已发布定点. 我们现在正在根据我们的平台验证和测试它。 测试完成后,我们将把它部署在我们的EMEA和美国的基础设施中。 上传文件暂时被禁用,直到安装并核实固定状态. 我们的监测继续显示,没有任何证据表明我们平台上利用了这种脆弱性。 阅读、张贴和答复仍然正常。 我们的下一次更新是5PM CEST, 或者如果有什么变化,更早.

    6. 2026年10月21日 UTC 15:30identified

      更新:我们正在测试释放的定律,以确保不引入任何回归. 上传文件暂时被禁用,直到安装并核实固定状态. 我们的监测继续显示,没有任何证据表明我们平台上利用了这种脆弱性。 阅读、张贴和答复仍然正常。 我们的下一次更新将在 6PM CEST, 或者如果有什么变化更早.

    7. 2026年10月21日 UTC 16:16identified

      更新:我们正在测试释放的定律,以确保不引入任何回归. 上传文件暂时被禁用,直到安装并核实固定状态. 我们的监测继续显示,没有任何证据表明我们平台上利用了这种脆弱性。 阅读、张贴和答复仍然正常。 我们的下一次更新将是7PM CEST, 或者如果有什么变化,更早.

    8. 2026年10月21日 UTC 18:18identified

      Update: We are completing the testing in staging before deploying to production. File uploads remain temporarily disabled until the fix is deployed and verified. Our monitoring continues to show no evidence that the vulnerability was exploited on our platform. Reading, posting and replying continue to work as normal. Our next update will be by 8:30PM CEST, or sooner if anything changes.

    9. 2026年10月21日 UTC 18:53monitoring

      Update: The PHP security fix has been deployed to production and we are continuing to monitor closely. We have found no evidence that the vulnerability was exploited on our platform. File uploads remain temporarily disabled out of an abundance of caution. Reading, posting and replying work as normal. Next update by 08:00AM CEST, or sooner if anything changes.

    10. 2026年10月22日 UTC 06:12monitoring

      Update: We are continuing to monitor and have found no evidence that the vulnerability was exploited on our platform. File uploads remain temporarily disabled out of an abundance of caution. Reading, posting and replying work as normal. Next update by 10:00AM CEST, or sooner if anything changes.

    11. 2026年10月22日 UTC 08:17monitoring

      Update: We are continuing to monitor and have found no evidence that the vulnerability was exploited on our platform. File uploads remain temporarily disabled out of an abundance of caution. Reading, posting and replying work as normal. Next update by 12:00PM CEST, or sooner if anything changes.

    12. 2026年10月22日 UTC 11:19monitoring

      Update: We are continuing to monitor and have found no evidence that the vulnerability was exploited on our platform. File uploads remain temporarily disabled out of an abundance of caution. Reading, posting and replying work as normal. Next update by 3:00PM CEST, or sooner if anything changes.

    13. 2026年10月22日 UTC 14:18monitoring

      Update: We are continuing to monitor and have found no evidence that the vulnerability was exploited on our platform. File uploads remain temporarily disabled out of an abundance of caution. Reading, posting and replying work as normal. Next update by 6:00PM CEST, or sooner if anything changes.

    14. 2026年10月22日 UTC 18:43monitoring

      Update: We are continuing to monitor and have found no evidence that the vulnerability was exploited on our platform. File uploads that render in the page remain temporarily disabled out of an abundance of caution. Reading, posting and replying work as normal. Attachments of files pdf, zip, txt is again possible. Next update by 10:00PM CEST, or sooner if anything changes.

    15. 2026年10月22日 UTC 20:02monitoring

      Update: We are continuing to monitor and have found no evidence that the vulnerability was exploited on our platform. File uploads that render in the page remain temporarily disabled out of an abundance of caution. Reading, posting and replying work as normal. Attachments of files pdf, zip, txt is again possible. Next update by 10:00AM CEST, or sooner if anything changes.

    16. 2026年10月23日 UTC 11:13monitoring

      Update: We are continuing to monitor and have found no evidence that the vulnerability was exploited on our platform. Image file uploads that render in the page remain temporarily disabled out of an abundance of caution. Reading, posting and replying work as normal. We will have an update on restoring image file uploads by end of day. Attachments of files pdf, zip, txt is again possible. Next update by 6:00PM CEST, or sooner if anything changes.

    17. 2026年10月23日 UTC 19:44monitoring

      Update: We are continuing to monitor and have found no evidence that the vulnerability was exploited on our platform. Image file uploads that render in the page remain temporarily disabled out of an abundance of caution. Reading, posting and replying work as normal. We have started reworking the concerned services to ensure we can restore image uploads as fast as possible. We will have a firm ETA by 25 Sunday 12pm CET Attachments of files pdf, zip, txt is again possible. Next update by 12:00PM CEST, or sooner if anything changes.

    18. 2026年10月26日 UTC 04:39monitoring

      Uploads restored: Uploads are available again in all Gainsight Customer Communities. No action is needed. We will continue to monitor closely.

    自动翻译自官方事件更新。

    [欧盟和美国群 登录错误500和504

    开始时间 2026年9月29日 UTC 11:05 · 1h 30m

    Outage重大事件
    受影响的组件
    Status of our EMEA Community InfrastructureStatus of our US Community Infrastructure
    1. 2026年9月29日 UTC 11:05identified

      这一问题已经确定,一个解决办法正在实施之中.

    2. 2026年9月29日 UTC 11:28monitoring

      一项措施已经执行,我们正在监测结果.

    3. 2026年9月29日 UTC 12:35resolved

      这个事件已经解决了——我们为这次对平台的干扰表示真诚的歉意,我们将在完成后将RCA文件上传到这里.

    自动翻译自官方事件更新。

    US - 用户概览更新缓慢

    开始时间 2026年8月24日 UTC 07:35 · 1d 5h

    Issues轻微事件
    受影响的组件
    User Overview
    1. 2026年8月24日 UTC 07:35investigating

      我们目前正在调查这个问题, 美国社群的用户概况显示新用户的时间比通常要长.

    2. 2026年8月25日 UTC 13:21resolved

      这一事件已经得到解决.

    自动翻译自官方事件更新。

    [EU + US communities] Topics failing to load

    开始时间 2026年8月4日 UTC 07:57 · 3h 30m

    Outage严重事件
    受影响的组件
    Status of our EMEA Community InfrastructureStatus of our US Community Infrastructure
    1. 2026年8月4日 UTC 07:57investigating

      We are currently investigating an issue with topics not loading correctly, we will update this page asap with more information once known.

    2. 2026年8月4日 UTC 08:08monitoring

      A fix has been implemented and we are monitoring the results.

    3. 2026年8月4日 UTC 11:27resolved

      This issue has been resolved - time of the incident (CEST) 9:33am - 10:00am; a 27 minute period in which most topics were returning a 404 error page instead of the topic. It was caused by an unexpected side affect of a downstream change to one of our platform security mechanisms, we apologise for this temporary disruption.

    Intermittent timeouts and slowness are identified in our EU communities

    开始时间 2026年7月8日 UTC 09:45 · 2h 9m

    Outage严重事件
    受影响的组件
    Status of our EMEA Community Infrastructure
    1. 2026年7月8日 UTC 09:45identified

      Intermittent timeouts and slowness are identified in our EU communities. We identified the issue and are mitigating it. We will follow up and update this page as soon as we have more information.

    2. 2026年7月8日 UTC 11:54resolved

      We have traced the root cause of the recent downtime to an AWS hardware failure. We apologize for the inconvenience this caused. Our team is already putting structural measures in place to ensure that similar hardware issues will not impact our platform going forward.

    Investigating AI Answers / AI Summary Services

    开始时间 2026年7月2日 UTC 17:28 · 2h 7m

    Pending
    受影响的组件
    Community SearchSearch Functionality
    1. 2026年7月2日 UTC 17:28investigating

      We are currently investigating an issue affecting AI Summary features, which may result in unexpected or incomplete responses. Search results are not impacted.

    2. 2026年7月2日 UTC 19:34resolved

      The issue affecting AI Summary features has been successfully resolved, and normal functionality has been fully restored. Thank you for your patience while we worked to fix this.

    Events Module Unavailable for US Communities

    开始时间 2026年5月29日 UTC 19:21 · 24m

    Outage严重事件
    受影响的组件
    Events Module
    1. 2026年5月29日 UTC 19:21investigating

      We are currently investigating reports that the Events module is not working or failing to load for users within our US communities. Our engineering team is looking into the root cause, and we will provide an update within the next 30 minutes. We apologize for the inconvenience and appreciate your patience while we work to resolve this.

    2. 2026年5月29日 UTC 19:45resolved

      The issue affecting the Events module in US communities has been fully resolved, and normal functionality has been restored. We appreciate your patience and apologize for any disruption this caused to your workflow. If you continue to experience any issues, please reach out to our support team at [email protected]

    Salesforce Connection Failure

    开始时间 2025年11月21日 UTC 12:47 · 117d 3h

    Issues轻微事件
    受影响的组件
    Salesforce IntegrationSalesforce Integration
    1. 2025年11月21日 UTC 12:47investigating

      We are investigating reports of Salesforce Connection failures and we will update here as soon as we have more details.

    2. 2025年11月21日 UTC 14:01identified

      This issue has been tied to the temporary revocation of Salesforce authentication credentials by Salesforce - we are working on getting these reinstated asap, no community data should be lost and events can be replayed again on our side to fill in the gaps once the authentication credentials have been reinstated.

    3. 2025年11月22日 UTC 02:57identified

      Work is ongoing to get Salesforce authentication reinstated. No community data should be lost and events can be replayed again on our side to fill in the gaps once the authentication credentials have been reinstated.

    4. 2025年11月22日 UTC 19:36identified

      The Salesforce and Gainsight incident remains ongoing. Our teams are actively working to resolve the issue and restore functionality as quickly and safely as possible. For the latest updates and FAQs, please visit the main status page: https://status.gainsight.com/incidents/gvng0kly8vwf

    5. 2025年11月22日 UTC 19:48identified

      We are continuing to work on a fix for this issue.

    6. 2025年12月2日 UTC 17:42identified

      The Salesforce and Gainsight incident remains ongoing. Rest assured our teams are working around the clock to resolve the issue and restore functionality as quickly and safely as possible. In the meantime, we are preparing a workaround and steps to re-enable services. For the latest updates and FAQs, please visit the main status page: https://status.gainsight.com/incidents/gvng0kly8vwf

    7. 2025年12月13日 UTC 22:43monitoring

      The Gainsight CC<>Salesforce integration has now been re-enabled and customers can proceed with re-authenticating their connection. To restore full functionality, please follow the step-by-step reconnection guide available here: https://communities.gainsight.com/customer-communities-cc-17/communities-salesforce-integration-restoration-guide-29962

    8. 2026年3月18日 UTC 16:35resolved

      This incident has been resolved.

    Issue with content creation and edits

    开始时间 2025年11月17日 UTC 09:07 · 4h 42m

    Outage重大事件
    受影响的组件
    Status of our EMEA Community InfrastructureStatus of our US Community Infrastructure
    1. 2025年11月17日 UTC 09:07investigating

      We are currently investigating an issue on some communities where content cannot be created or edited; either via the control administration area or sometimes from within the community destination pages.

    2. 2025年11月17日 UTC 12:03identified

      The issue has been identified and a fix is being implemented.

    3. 2025年11月17日 UTC 12:20monitoring

      A fix has been implemented and we are monitoring the results.

    4. 2025年11月17日 UTC 13:49resolved

      This issue has been resolved - we traced the source of the issue to some database indexes that were lost when a subsequent database update wiped out these indexes. We estimate that only around 10% of customers had this issue happen to their community and the problem was certainly not widespread. Any left over issues please contact [email protected] and we apologies for any inconvenience caused by this.

    Custom community pages may not load properly

    开始时间 2025年11月10日 UTC 18:38 · 5h 16m

    Issues轻微事件
    受影响的组件
    Status of our EMEA Community InfrastructureStatus of our US Community Infrastructure
    1. 2025年11月10日 UTC 18:38investigating

      We’re investigating an issue where custom community pages may not load correctly.

    2. 2025年11月10日 UTC 20:15identified

      Our engineering team has identified the root cause and is actively working on a fix.

    3. 2025年11月10日 UTC 21:08identified

      We have applied the initial fix which resolves this issue for the majority of customers. We are actively working on a fix that continues to impact custom pages for less than 1% of CC customers.

    4. 2025年11月10日 UTC 22:19identified

      We are continuing to work on a fix for this issue.

    5. 2025年11月10日 UTC 23:54resolved

      This incident has been resolved.

    Embedded Media Not Working

    开始时间 2025年10月6日 UTC 15:18 · 4h 37m

    Issues轻微事件
    受影响的组件
    Text EditorText Editor
    1. 2025年10月6日 UTC 15:18identified

      Embedded media is currently not working for some video providers (e.g., Vimeo, Twitter, Vidyard, etc.). YouTube embeds remain unaffected and continue to work as expected. We’ve identified the root cause of the issue, and our engineering team is actively working on a fix. We’ll share an update as soon as the resolution is deployed.

    2. 2025年10月6日 UTC 19:25monitoring

      The fix was deployed, we are now monitoring this issue.

    3. 2025年10月6日 UTC 19:55resolved

      This incident has been resolved.

    Embedded media (e.g., Vimeo videos) is currently not working across communities

    开始时间 2025年9月26日 UTC 08:32 · 1h 39m

    Issues轻微事件
    受影响的组件
    Text EditorText Editor
    1. 2025年9月26日 UTC 08:32investigating

      Embedded media (e.g., Vimeo videos) is currently not working across communities. Users may see “Media not available” when trying to play embedded videos. Our engineering team is actively investigating the issue and working on a fix.

    2. 2025年9月26日 UTC 10:12resolved

      The incident affecting embedded media has been resolved. Embedded videos should now display correctly across all affected communities.

    US region degraded performance

    开始时间 2025年9月3日 UTC 16:49 · 4h 7m

    Issues轻微事件
    受影响的组件
    Status of our US Community Infrastructure
    1. 2025年9月3日 UTC 16:49identified

      We identified a degraded performance especially on the homepages of our US communities. The investigation is ongoing into what is the root cause to this.

    2. 2025年9月3日 UTC 17:43identified

      The root cause was identified, and the performance is back to expected performance. We will continue monitoring this incident.

    3. 2025年9月3日 UTC 17:44identified

      We are continuing to work on a fix for this issue.

    4. 2025年9月3日 UTC 17:44monitoring

      A fix has been implemented and we are monitoring the results.

    5. 2025年9月3日 UTC 20:56resolved

      This incident has been resolved.

    [EU + US communities] Issues with Linkedin + Facebook Login

    开始时间 2025年8月29日 UTC 08:48 · 2h 42m

    Outage重大事件
    受影响的组件
    Single Sign On (SSO)Single Sign On (SSO)
    1. 2025年8月29日 UTC 08:48identified

      We have identified an issue with LinkedIn and Facebook login functionality - it can be temporarily mitigated by adding a new 'Authorized redirect URL for your app', adding a http version of the url e.g: http://[yourcommunityurl.com]/sso/return. We are working on a full fix for this now though and will update this page with more information once available. Please note that this issue will not affect normal SSO logins / registrations which continue to function normally.

    2. 2025年8月29日 UTC 10:36monitoring

      A fix has been implemented and we are monitoring the results.

    3. 2025年8月29日 UTC 11:30resolved

      This incident has been resolved and both Linkedin and Facebook logins are back to normal, we sincerely apologise for the disruption caused by this.

    EU + US Communities - Elevated error rates on topic load

    开始时间 2025年8月6日 UTC 14:05 · 3h 22m

    Outage重大事件
    受影响的组件
    Status of our EMEA Community InfrastructureStatus of our US Community Infrastructure
    1. 2025年8月6日 UTC 14:05investigating

      We are investigating an issue where topics are not loading and and error page is shown, it seems to be affecting Articles and Product Updates.

    2. 2025年8月6日 UTC 14:24identified

      The issue has been identified and a fix is being implemented.

    3. 2025年8月6日 UTC 15:33monitoring

      A fix has been implemented and we are monitoring the results.

    4. 2025年8月6日 UTC 17:28resolved

      This incident has been resolved.

    Community Pages Not Loading Properly

    开始时间 2025年7月28日 UTC 17:29 · 3h 55m

    Outage重大事件
    受影响的组件
    Status of our EMEA Community InfrastructureStatus of our US Community Infrastructure
    1. 2025年7月28日 UTC 17:29identified

      We’re currently experiencing an issue where community pages are not loading correctly. Our engineering team has identified the root cause and is actively working on a fix. We’re aiming to restore full functionality as soon as possible and will share updates here as progress is made. Thank you for your patience and understanding.

    2. 2025年7月28日 UTC 21:24resolved

      This incident has been resolved.

    Issues with Third Party Scripts due to Front End performance improvements

    开始时间 2025年7月24日 UTC 07:00 · 0m

    Pending
    1. 2025年8月11日 UTC 10:21resolved

      Incident Summary: On 24 July 2025, we deployed a performance improvement designed to speed up page loading by changing when certain parts of our application load in the browser. As a result of this change, some page elements loaded later than before. Certain third party scripts used by customers depended on those elements being available straight away, so those scripts failed to run correctly. This caused integration errors and broken functionality for affected customers. Impact: Seven customers experienced issues with their integrations. In these cases, third party scripts could not access the expected page elements, which led to errors and features not working as intended. Timeline: - 24 July 2025 9:00 AM (CET) - Deployment completed with new loading behaviour as part of performance improvements. - 25 July 2025 - First customer reports of integration issues received. An internal investigation began to identify the cause. - 26 to 29 July 2025 - Investigation confirmed that lazy loading of certain elements meant they were not available when customer scripts ran. Compatibility checks were carried out to assess possible fixes and confirm the scope of the impact. - 30 July 2025 15:30 AM (CET) - Decision made to revert the change to restore the original loading behaviour. Reversion deployed to production. Monitoring confirmed resolution for all affected accounts. Impacted customers were contacted and confirmed the issue was resolved. Resolution: We rolled back the performance change so that all elements load as they did before. This restored the expected functionality for all affected customers. Mitigation and Next Steps: - Implement a compatibility aware rollout strategy for future performance changes. - Conduct an audit of existing third party scripts to identify other potential compatibility risks. - Include broader customer impact reviews before releasing performance optimisations. - Give customers and support teams more notice of significant changes that may affect integrations. Lessons Learned: There are a wide range of third party scripts in use across all customer instances, and changes to page performance can have unintended side effects. Future performance updates will be introduced in phased rollouts with additional compatibility checks, and customer impact will be reviewed more thoroughly before release.

    EU Communities - elevated error rate

    开始时间 2025年7月15日 UTC 15:14 · 20m

    Outage重大事件
    受影响的组件
    Status of our EMEA Community Infrastructure
    1. 2025年7月15日 UTC 15:14investigating

      We are currently investing an issue with an elevated error rate (15-20%) affecting EU communities, we will update here once more information is available.

    2. 2025年7月15日 UTC 15:24identified

      The issue has been identified and a fix is being implemented.

    3. 2025年7月15日 UTC 15:34resolved

      This issue has been resolved and the platform is stable again. Any more follow up issues please contact [email protected] - apologies for this short disruption.

    SSO Login issues

    开始时间 2025年6月27日 UTC 07:26 · 2h 13m

    Outage重大事件
    受影响的组件
    Status of our EMEA Community InfrastructureStatus of our US Community Infrastructure
    1. 2025年6月27日 UTC 07:26investigating

      We are currently investigating an issue with SSO logins leading to an error 500 page after login.

    2. 2025年6月27日 UTC 07:42identified

      The issue has been identified and a fix is being implemented.

    3. 2025年6月27日 UTC 08:08monitoring

      A fix has been implemented and we are monitoring the results.

    4. 2025年6月27日 UTC 09:39resolved

      This incident has been resolved - we sincerely apologise for the disruption caused by this, any more issues or questions please email [email protected].

    US Communities - Currently experiencing issues with community loading

    开始时间 2025年6月13日 UTC 15:19 · 4h 44m

    Outage重大事件
    受影响的组件
    Status of our US Community Infrastructure
    1. 2025年6月13日 UTC 15:19investigating

      We are currently investigating this issue.

    2. 2025年6月13日 UTC 15:45investigating

      We are continuing to investigate this issue - we have identified a couple of infrastructure issues thought to be causing this. We appreciate your continued patience,

    3. 2025年6月13日 UTC 16:46investigating

      We are continuing to investigate the source of the problem, and have identified a few possibilities for it. As soon as we know more information we will post it here asap.

    4. 2025年6月13日 UTC 17:37investigating

      We are continuing to investigate this issue.

    5. 2025年6月13日 UTC 18:11investigating

      User experience is improving at this time but we are still investigating.

    6. 2025年6月13日 UTC 18:44investigating

      Users may experience degraded performance in some areas.

    7. 2025年6月13日 UTC 18:56identified

      The issue has been identified, and a fix was deployed. Performance is steadily improving.

    8. 2025年6月13日 UTC 19:05monitoring

      We are monitoring the results of our fix, and performance is back to normal.

    9. 2025年6月13日 UTC 20:03resolved

      This incident has been resolved. Details will be added once available.

    10. 2025年6月19日 UTC 10:58postmortem

      **Redis Cache Degradation and Service Impact - June 13, 2025** ### **Overview** On **June 13, 2025**, from **17:09 to 21:13 CET**, some of our services in the **US region** experienced significant disruptions due to a cascading infrastructure failure. The issue began with degraded performance in our Redis caching layer, triggered by underlying **AWS infrastructure instability**, and ultimately led to elevated error rates and service outages across several systems. ‌ ### **What Happened** A sudden degradation in our Redis cache - likely caused by transient **network issues within AWS infrastructure** - resulted in dramatically slower response times and dropped connections. As Redis became unavailable, systems attempted to query the main database directly, which caused that database to hit its connection limits and become overloaded. This caused broader performance issues across multiple services. ### **Customer Impact** * **High error rates** \(HTTP 5XX\) across several services in the US region * **Slower API response times** and degraded system performance * **Stale or delayed data** due to failed or missed caching * **Up to 4 hours of partial or full service disruption** * **No customer data was lost**, but availability and performance were impacted ### **Timeline of Events \(CEST\)** * **17:09** - Error rates spike and initial alarms trigger * **17:13** - Incident response begins * **17:30-19:00** - Teams work to stabilise the system, including service rollbacks and traffic load reduction * **18:38** - AWS confirms a related infrastructure/network event affecting Redis connectivity * **19:30** - Redis instance upgraded to a higher-capacity configuration * **21:10** - Services fully recover * **21:13** - Incident officially resolved ### **What Caused the Issue?** Our Redis cache experienced performance degradation and stopped serving traffic effectively. Based on internal logs and external confirmations, this was **likely due to transient network and infrastructure issues within AWS**, which impacted bandwidth and connection handling for Redis. As a result, traffic was redirected to the main database, which then became overloaded, creating a cascading service failure. ### **How Was It Resolved?** * Redis was scaled up to handle more traffic and recover performance * Non-essential services were temporarily paused to reduce system load * Once Redis recovered, database pressure eased and systems began returning to normal ### **Could This Have Been Prevented?** Partially. Some safeguards helped limit impact \(such as CDN caching for guest users\), but earlier detection of Redis degradation and better isolation between cache and core systems could have reduced downtime. The unexpected AWS infrastructure event contributed significantly to the incident’s scope. ‌ ### **Key Lessons and Next Steps** * **Improve monitoring and alerting** for Redis resource exhaustion and network saturation * **Isolate caching infrastructure** by service or use case to prevent single points of failure * **Enhance fallback logic** to reduce stress on primary databases when caching layers fail * **Work closely with AWS** to better understand and mitigate infrastructure-related disruptions * **Explore more resilient multi-layer caching strategies** to handle future spikes gracefully We sincerely apologise for the disruption and appreciate your patience. We are taking meaningful steps to strengthen our infrastructure and improve our detection and response capabilities to prevent similar events in the future. For any more information or issues please email [[email protected]](mailto:[email protected]).