고급 성능 - 관리자 및 Storefronts
Issues경미한 인시던트
- resolved
약 19:35 및 19:39 UTC 10 월 18, 플랫폼은 관리자 대시보드 및 storefronts에 영향을 미치는 성능의 간단한 기간을 경험했습니다. 서비스 자동 회복 및 완전히 작동 유지. 서비스는 몇 분 안에 정상으로 돌려줍니다.
공식 인시던트 업데이트를 자동 번역했습니다.
54 Uscreen incidents · 2022년 6월 — official updates, affected components, duration and resolution details.
약 19:35 및 19:39 UTC 10 월 18, 플랫폼은 관리자 대시보드 및 storefronts에 영향을 미치는 성능의 간단한 기간을 경험했습니다. 서비스 자동 회복 및 완전히 작동 유지. 서비스는 몇 분 안에 정상으로 돌려줍니다.
공식 인시던트 업데이트를 자동 번역했습니다.
10 월 14, 23 : 35에서 23 : 46 UTC (7 : 35 PM에서 7 : 46 PM EDT), 일부 사용자는 관리자 대시보드 및 상점에 "Rate가 초과 된"오류 및 느리게-단간로드 시간을 보았습니다. 원인은 우리의 인프라 제공 업체 중 하나에 서비스 중단이었다. admin 대쉬보드와 storefront의 서비스들은 완전히 복원되고 운영됩니다.
공식 인시던트 업데이트를 자동 번역했습니다.
관리자 대시보드, storefronts 및 API에 영향을 미치는 오류 및 서비스의 보고서를 볼 수 있습니다. 우리의 팀은 현재 조사입니다.
우리의 엔지니어링 팀은 관리자 대시보드, storefronts 및 API에 영향을 미치는 오류의 원인을 확인했습니다. 서비스는 영향을받는 표면 전체에 걸쳐 복원되었으며, 팀은 더 넓은 수정 작업을 계속합니다.
우리의 기술설계 팀은 더 넓은 고침에 일하기 위하여 계속됩니다. 서비스는 admin 대쉬보드, storefronts 및 API를 통해 제공됩니다.
수정이 구현되었습니다. 현재 관리자 대시보드, storefronts 및 API를 통해 서비스의 전체 복원을 볼 수 있습니다. 우리의 팀은 안정성을 지키는 바싹 감시자를 계속할 것입니다.
이 사건은 해결되었습니다. admin 대쉬보드, storefronts 및 API를 통한 서비스는 완전히 복원되고 운영됩니다. 우리의 엔지니어링 팀은 이것을 통해 일하면서 당신의 인내력을 주셔서 감사합니다.
**문의:** Storefront 오류 및 degraded 성능 **일시:** 9월 13, 2026 **Duration:** 12:09–12:47 UTC and 13:18–13:26 UTC, 고객 영향의 약 45 분 ## 요약 몇몇 고객 및 최종 사용자는 9월 13일에 2 기간 도중 intermittent storefront 과실 및 degraded 성과를 경험했습니다. 서비스는 복원되었습니다, 추가 트래픽 보호가 구현되었으며, 플랫폼은 일반적으로 작동합니다. ## 무슨 일이 있었는지 우리의 인프라는 플랫폼의 정상적인 운영 범위를 초과하는 들어오는 트래픽에서 비정상적인 스파이크를 경험했습니다. 증가된 하중은 공유 플랫폼 자원에 압력을 가하고, 저장소 앞의 간헐적 오류 및 저하된 성능을 유발합니다. ## 행사 일정 * 12:09 UTC: 고객 가능 평가 시작. * 12:42 UTC: 추가 트래픽 제어가 적용되었습니다. * 12:47 UTC: 서비스 영향의 첫 번째 기간에서 복구. * 13:18 UTC: 축하의 두 번째 기간 시작. * 13:22 UTC: 교통 통제는 더 강화되었습니다. * 13:26 UTC: 서비스는 복원되었습니다. * 16:05 UTC: 추가 보호는 장소에서 확인되었으며, 플랫폼은 일반적으로 작동되었습니다. ## 뿌리 원인 비정상적인 트래픽 패턴은 플랫폼의 정상 작동 범위를 넘어 들어오는 요청의 급격한 볼륨을 생성합니다. 이 요청을 처리하는 것은 공유 플랫폼 리소스에 과도한 압력을 배치, 이는 저장 공간 성능과 간헐적인 오류를 압니다. ## 우리가 한 것 우리의 엔지니어링 팀은 트래픽 패턴, 적용 트래픽 제어 및 복원 된 서비스를 조사했습니다. 우리는 다음을 통해 합법적 인 트래픽을 허용하는 추가 보호를 활성화하고 중단 된 요청을 제한하고 플랫폼 안정성을 확인하기 위해 모니터링. ## 우리가 recurrence를 방지하기 위해 노력하고 있는 것 우리는 4개의 지역에 있는 플랫폼을 강화하고 있습니다: * 이상 트래픽 및 높은 플랫폼 자원 사용을위한 경고 확장. * storefront에 걸쳐 트래픽 처리 및 보호 강화. * 예상치 못한 트래픽의 리소스 비용을 줄이기 위해 storefront 캐싱 및 애플리케이션 효율성을 향상. * 증가 서비스 고립 그래서 플랫폼의 한 부분에 교통 압력은 다른 표면에 영향을 미칠 가능성이 적습니다. ## 고객 영향 * **Affected 표면: ** * 관리자 포털 * 상점 * API - V1 * API - V2 * **User-facing 경험:** 과도한 과실, degraded 성과 및 임시 불균형 * ** 데이터 무결성:** 데이터 손실 없음, 보안 위반 없음, 고객 계정 또는 저장 된 콘텐츠에 대한 영향 우리의 엔지니어링 팀은 이것을 통해 일하면서 당신의 인내력을 주셔서 감사합니다. 우리는 플랫폼 신뢰성을 위해 고수준을 보유하고, 우리는 우리가 여기에서 배운 무슨을 적용하고 있습니다.
공식 인시던트 업데이트를 자동 번역했습니다.
제3자 CDN 파트너는 비디오 재생 오류를 발생시키는 문제점을 경험하고 있습니다. 일부 동영상은 예상대로 재생할 수 없습니다. 우리는 우리의 CDN 파트너와 긴밀한 커뮤니케이션에서 우리가 그(것)들을 가진 갱신을 제공할 것입니다.
이 문제를 조사하고 있습니다.
우리의 타사 CDN 파트너는 수정을 구현하고 현재 결과를 모니터링하고 있습니다. 우리는 영상 재생을 밀접하게 보고 우리가 풀 해결책을 한 번 확인하는 것을 계속합니다.
이 사건은 해결되었습니다. CDN 파트너는 수정을 확인하고 비디오 재생은 정상으로 돌아 왔습니다. 우리는이 해결 된 동안 당신의 인내를 평가합니다.
공식 인시던트 업데이트를 자동 번역했습니다.
We're investigating reports that some pages are loading without proper styling for a subset of visitors on certain customer websites. Our team is actively working to identify the root cause and resolve it, and we'll post updates here as we learn more
We are monitoring the situation. If you're still seeing pages load without styling, try a hard refresh (Ctrl/Cmd + Shift + R) or clear your browser cache — this should resolve it immediately
We've applied a change that has resolved the immediate issue — visitors on affected customer websites should no longer see pages loading without proper styling. The underlying cause appears to be related to how DNS records for one of our infrastructure providers were being resolved. We've reached out to them for more detail and will share an update once we have a full explanation. If you're still seeing unstyled pages, your browser or device may have cached a previous, incorrect DNS result. Forcing a fresh lookup resolves it immediately: Chrome: 1. Paste into the address bar and press Enter: chrome://settings/security 2. Scroll to "Use secure DNS" and turn it ON. 3. Select "Cloudflare (1.1.1.1)" from the dropdown. 4. Refresh. Edge: 1. Paste: edge://settings/privacy 2. Scroll to "Use secure DNS" and turn it ON. 3. Select "Cloudflare (1.1.1.1)". 4. Refresh. Firefox: 1. Paste: about:preferences#privacy 2. Scroll to "DNS over HTTPS", set to "Max Protection". 3. Choose "Cloudflare". 4. Refresh.
We are currently investigating an issue causing degraded performance in Communities. We'll post updates as soon as they are available. Thank you for your patience.
We are continuing to investigate this issue.
We've identified the cause of the slow page loads and errors affecting the storefront, admin portal, and APIs. Following an earlier database upgrade, a subset of queries are running significantly slower than expected. Our team is implementing a fix now. We'll share another update as soon as the fix is deployed and we begin monitoring recovery.
We've rolled out a fix and performance is returning to normal, while we improving the cache ratio. We're continuing to monitor closely to make sure everything stays stable. We'll share a final update once we're confident the issue is fully resolved.
The earlier fix did not hold and we're seeing degradation return. Engineering is investigating further. We'll update as soon as we have more.
Recovery is underway and performance is improving across our services. We're proceeding carefully to ensure stability as everything returns to normal. You may still see some degradation for a short while longer. We'll update as soon as full recovery is confirmed.
Good news: storefronts and the Admin Portal are recovering. We're now focused on bringing the APIs that power mobile and TV apps back online. We'll update again as soon as more is restored.
Storefronts and the Admin Portal are now fully back online and running normally. Thanks for your patience while we worked through this. The APIs that power mobile and TV apps are still affected — that's our team's full focus right now. We'll update again as soon as those are restored.
Mobile and TV app APIs are coming back online. We're continuing to monitor as services fully recover. We'll share another update once everything is operating normally.
All services are back online and operating normally. We're continuing to monitor closely to ensure everything remains stable. We'll share a final update once we're confident the issue is fully resolved.
This incident is now resolved. All services are operating normally. A detailed post-mortem will be shared shortly. Thank you for your patience.
# Incident Post-Mortem: May 4, 2026 Database Upgrade ## A note from our team On Monday, May 4, 2026, Uscreen experienced a significant performance incident following a planned PostgreSQL database upgrade. Storefronts, the Admin Portal, and the APIs that power our mobile and TV apps were degraded or unavailable for portions of the day for many of our customers. We know that when your storefront is slow, or your viewers can’t watch, it’s not just a technical issue — it affects your business, your subscribers, and your reputation. We’re deeply sorry for the disruption, and we owe you a clear, honest account of what happened, how we resolved it, and what we’re doing to make sure this doesn’t happen again. This document is that account. ## What happened, in plain language We perform major database upgrades approximately once per year to keep the platform secure, fast, and on a supported version. The upgrade itself completed successfully and on schedule. The problem started **after** the upgrade, when production traffic returned to the freshly upgraded database. After a major version upgrade, a database needs to “warm up” — its internal memory cache, which normally holds the most frequently accessed data ready for instant retrieval, starts empty. As traffic returned, our database needed to load enormous amounts of data into that cache simultaneously. We had not anticipated how severely this cold-start effect would compound with our current workload, and the database fell behind. Requests began to queue. Queries that normally took milliseconds took seconds. The platform slowed down dramatically for many customers. It’s worth being direct about this: **we tested this upgrade beforehand using a canary approach** — running it in a non-production environment to validate it. And we performed a similar major upgrade last year without significant issues. What changed between then and now was the **size of our data**. Our platform has grown substantially over the past year, and the cold-start memory pressure that was manageable last year became severe at our current scale. Our pre-production environment did not faithfully replicate that scale, so it did not expose the problem before it hit production. This is the specific gap we are closing — see “What we’re changing” below. Compounding the cold-start problem, a small number of high-volume application queries — ones that performed acceptably under normal conditions — became significantly slower under memory pressure. This created a feedback loop: slow queries held connections open, those connections couldn’t serve other requests, and the backlog grew faster than the system could clear it. It took our team longer than we would have liked to correctly diagnose the structural cause. Our initial hypotheses — that statistics needed rebuilding, that specific code-level query fixes would resolve it — turned out to be only partial answers. The real fix required more decisive action: stopping the application entirely to let the database stabilize, expanding the database’s memory capacity, and then gradually bringing services back online. ## Timeline \(all times Eastern\) | Time | What happened | | --- | --- | | **12:00 AM** | Planned maintenance window begins. Database upgrade initiated. | | **12:15 AM** | Upgrade phase completes successfully — faster than planned. | | **~12:15–3:30 AM** | Backups and replica provisioning continue in the background. Application traffic returns. | | **~3:30 AM** | Performance begins to degrade as production traffic ramps up. Database memory cache is cold and under heavy load. | | **4:00–6:00 AM** | Severe degradation. Average response times rise to 4 seconds. End-user experience is heavily affected, particularly for customers in European time zones whose business day is starting. | | **6:00–11:00 AM** | Partial recovery as cache warms naturally, but performance remains unstable with intermittent slowness and errors. | | **7:37 AM** | First public status page acknowledgment of degraded performance. _\(We should have posted earlier — see “What we’re changing” below.\)_ | | **8:31 AM** | Root cause identified as post-upgrade query performance issues. | | **9:12 AM** | First targeted fix deployed — code-level changes to reduce load from a high-volume query pattern, plus routing more read traffic to a replica database. | | **9:42 AM** | The first fix improved things briefly but did not hold. Performance degraded again. We publicly acknowledged this on the status page and resumed investigation. | | **~10:00–10:30 AM** | Decision to take more decisive action. Application stopped to let the database fully drain queued work. Database resized to provide additional memory headroom. | | **10:30 AM** | Application services begin coming back online in stages, allowing the database cache to warm progressively rather than all at once. | | **10:47 AM** | Storefronts and Admin Portal recovering. | | **10:58 AM** | Storefronts and Admin Portal fully restored. | | **11:22 AM** | Mobile and TV app APIs recovering. | | **11:31 AM** | All services restored to normal operation. | | **Through afternoon** | Continued close monitoring. Performance held stable. | ## What was affected Different services were affected at different times and to different degrees: * **Storefronts** — Slow page loads and intermittent errors during the worst windows \(approximately 4:00–11:00 AM ET, with severity peaking 4:00–6:00 AM and again briefly around midday\) * **Admin Portal** — Severe slowness and “Unexpected Server Error” pages for many creators * **Mobile and TV apps \(API V1 and V2\)** — Slow or failing requests, recovering last * **Communities** — The initial area where our investigation began, where high-volume queries were among the early contributors to the database load Throughout the incident, **no customer data was lost, no security boundaries were affected, and no payment or subscription data was compromised.** The issue was performance, not integrity. ## What we did to resolve it Our response unfolded in two layers. The application-layer fixes \(the “minor contributors”\) helped, but the database-layer actions \(the “major contributors”\) actually ended the incident. ### Major contributors to recovery 1. **Stopped the application entirely.** When new slow queries were arising faster than we could resolve them, we made the deliberate decision to halt all application traffic for a short window so the database could finish its work and return to a stable baseline. This was a planned, controlled outage of approximately 10–15 minutes — chosen because partial mitigation was no longer keeping pace with the problem. 2. **Resized the database to add memory capacity.** We moved to a larger database instance with significantly more memory available for the cache. This addressed the structural root cause: our previous configuration didn’t have sufficient memory headroom for the platform’s current workload during a cold start. 3. **Brought services back online gradually.** Rather than restoring all traffic at once and risking a repeat of the cold-cache problem, we restarted application servers in stages. This lets the database cache warm progressively, services come back in a controlled order, and confidence builds at each step. Storefronts came back first, followed by the Admin Portal, then the mobile and TV APIs. 4. **Disabled a database optimization feature \(JIT compilation\)** that, under our specific workload during the incident, was adding overhead rather than reducing it. This will be re-evaluated and tuned separately. ### Minor contributors to recovery These code-level fixes shipped during the incident genuinely helped — they reduced load on the database and improved query performance — but on their own, they couldn’t resolve the structural problem. 1. **High-volume query optimization.** A frequently-fired query that scanned a large amount of historical data on every call was bound to only look at a recent time window, dramatically reducing the work required per request. 2. **Read traffic routing.** We routed more of our API V2 read traffic to replica databases, taking pressure off the primary. 3. **Targeted index improvement.** An index was added to support a frequent storefront query. We are continuing to ship these and similar improvements this week. They make the platform faster and more resilient regardless of upgrades. ## Why did this take longer than it should have? We want to be honest about this: the incident lasted longer than it needed to. Three things contributed: 1. **Our initial diagnosis was incomplete.** The first hypotheses we pursued — that database statistics needed rebuilding, that specific code-level query fixes would resolve it — were partial truths. The structural cause \(insufficient memory for cold-cache recovery under our current workload\) wasn’t identified until later in the incident. We chased the symptoms before fully understanding the system pressure. 2. **We were slower to communicate publicly than we should have been.** Our first status page update came hours after degradation began. Customers experiencing slowness deserved to hear from us sooner, even if all we could say was “we’re investigating.” 3. **Our pre-production testing didn’t catch the scale-dependent failure mode.** We did run this upgrade in a canary environment first, and a comparable upgrade last year went smoothly. But our test environment didn’t replicate the full size of our current production data, so the cold-start memory pressure that hit us at production scale never showed up in testing. Our growth over the past year crossed a threshold our procedures hadn’t been updated to account for. We’re owning these honestly because they are exactly the things we can change. ## What we’re changing so this doesn’t happen again This is the most important section of this document. Several improvements are already in motion. ### Database capacity and configuration * **The larger database instance is now permanent.** We are not returning to the previous configuration. Our memory headroom is now sized appropriately for our current and projected workload. * **We are reviewing additional database configuration** settings \(memory tuning, JIT settings, connection limits\) in light of the workload patterns we observed during the incident. * **Specific high-volume queries are being permanently optimized,** including the ones that became hotspots during the incident — independent of any future upgrade. ### Upgrade procedures * **A formal major-version upgrade runbook is being written** with mandatory pre- and post-upgrade steps, including cache pre-warming, staged traffic restoration, and an extended observation period before declaring success. * **We are upgrading our pre-production canary environment to better mirror production scale.** Our canary process worked correctly for last year’s upgrade, but production data growth has outpaced our test data. Going forward, the canary will be sized to faithfully reproduce production-scale memory pressure, query patterns, and connection load, so that scale-dependent failure modes show up before they reach customers. * **Pre-upgrade query performance baselines** will be captured, so any regression after an upgrade is immediately visible by comparison. * **Capacity reviews are now part of every major upgrade plan.** Before any future major upgrade, we will explicitly verify that the database's memory and connection capacity are sized for cold-start recovery, not just for steady-state operation. ### Monitoring and alerting * **New alerts on response-time and database-pressure metrics** so that degradation is detected automatically and the on-call team is paged within minutes of onset, not after customer reports accumulate. * **Database memory pressure and lock contention** are being added to our standard dashboards as first-class signals. ### Communication * **Our internal incident playbook now requires a public status page update within 15 minutes** of any confirmed customer-facing degradation, regardless of whether we have a diagnosis yet. * **We will publish post-mortems for significant incidents** as a standard practice, not just when asked. ## Looking forward We want to leave you with three things: **This kind of major maintenance happens approximately once per year.** It’s not a routine event, and it’s not the new normal. We performed a similar upgrade last year that went smoothly, and most days, weeks, and months on Uscreen pass without anything like this incident. Today was a bad day, not a sign of bad days ahead. **Every change above is real.** We’re not gesturing at improvement — we are funding it, scheduling it, and assigning it to specific people. The next major database upgrade will be a fundamentally different operation from this one. **We take this seriously because you trust us with your business.** Your subscribers, your revenue, your reputation — they all depend on Uscreen being available and fast. We did not meet that standard yesterday. We are going to. If you have questions or concerns, or were materially affected and want to discuss the impact on your business, please reach out to your account contact or to [[email protected]](mailto:[email protected]). We are here, and we want to talk. — The Uscreen Engineering and Operations team ## Appendix: Technical detail \(for technical customers\) For customers with engineering teams who want a deeper view: * **Database:** Managed PostgreSQL, recently upgraded to the current major version * **Working set:** Approximately 1 TB OLTP database * **Pre-incident memory cache configuration:** Undersized for the cold-start scenario after a major upgrade; resized post-incident to provide significant additional headroom * **Peak symptoms during the incident:** Database connections fully saturated, recent cache hit ratio dropped to ~36%, significant lock contention from concurrent backends competing for buffer access, dozens of concurrent long-running queries \(>10s\), application-server p95 latency at 5.25s, Apdex bottoming at 0.27 * **Recovery state:** Connections at normal idle levels, cache hit ratio 99.4–99.8%, p95 latency 0.29s, Apdex 0.99 * **Initial diagnostic dead-ends ruled out during the incident:** missing planner statistics \(replica counters were misleading — actual statistics had replicated correctly\), collation library drift \(verified intact\), and major configuration drift \(verified carried over correctly from the prior version\) * **Application-layer fixes shipped during the incident:** bounded a high-volume time-series query to a recent time window, routed additional read traffic to replicas, and addressed related code paths * **Outstanding follow-up technical work** includes additional indexes on hot lookup columns, improved indexing for a JSONB lookup pattern, partitioning of a large time-series table, and enabling automatic GET-request routing to replicas at the framework level
We have identified an issue following a scheduled maintenance. Some functionality may be degraded while we work on a fix. We'll post updates as soon as they are available. Thank you for your patience.
We are continuing to work on a fix for this issue.
The issue has been resolved and we are monitoring for any further errors. Thank you for your patience.
This incident has been resolved.
We're seeing reports of intermittent errors and slower-than-usual load times affecting admin dashboards and storefronts. Our team is currently investigating.
Our engineering team has identified the cause of the slowness and intermittent errors affecting admin dashboards and storefronts, and is actively implementing a resolution.
A fix has been implemented, and we're currently seeing full restoration of services across admin dashboards and storefronts. Our team will continue to monitor closely to ensure stability.
This incident has been resolved. Services across admin dashboards and storefronts are fully restored and operating normally. Thank you for your patience while our engineering team worked through this.
We experienced a brief service disruption lasting approximately 2 minutes. The issue has been fully resolved and all systems are operating normally. We apologize for any inconvenience caused.
We’re monitoring an issue with a third-party provider that may impact video uploads. Some customers may see upload failures while the provider works to restore normal service. Our team is tracking this closely and will post an update as soon as we have more details.
The third-party provider has implemented a fix and is currently monitoring results. We’re continuing to monitor on our side as well, and we’ll share another update as soon as we can confirm full resolution.
The third-party provider has confirmed that the earlier issue has been resolved. Video uploads are now being processed normally.
We're aware that some storefronts hosted on Uscreen subdomains may be experiencing slow load times or intermittent errors. This is due to an issue with one of our third-party providers. Our team is actively monitoring the situation. We'll post an update here as soon as we have more information from the 3rd party provider.
Our third-party partner has resolved the issue, and our team continues to monitor for any changes. All storefronts are operational.
We experienced a brief performance degradation due to a DDoS attack that impacted platform performance for approximately 4 minutes. Our team responded quickly, mitigated the attack, and deployed additional protections to prevent further impact. Performance has been fully restored and remains stable. Thank you for your patience.
A third-party CDN partner is experiencing issues that are causing video playback errors. Some videos may not play as expected. We're in close communication with our CDN partner and will provide updates as we have them.
Our CDN partner has implemented a fix for the video playback errors. We are currently monitoring to ensure videos are functioning normally. We will provide a final update once we confirm the issue is fully resolved.
The video playback issues have been fully resolved. Our CDN partner's fix has been successfully implemented, and we have confirmed that videos are functioning normally.
A third-party CDN partner is experiencing issues that are causing video playback errors. Some videos and livestreams may not play as expected. We're in close communication with our CDN partner and will provide updates as we have them.
Our CDN partner has identified the issue causing video playback errors. They are currently working on implementing a fix. Some videos and livestreams may still not play as expected while the fix is being deployed. We will continue to provide updates as we receive them.
Our CDN partner has implemented a fix for the video playback errors. We are currently monitoring to ensure videos and livestreams are functioning normally. We will provide a final update once we confirm the issue is fully resolved.
The video playback issues have been fully resolved. Our CDN partner's fix has been successfully implemented and we have confirmed that videos and livestreams are functioning normally.
We are currently experiencing an issue that may affect some of our services. Our team is actively investigating and working to resolve it as quickly as possible. Thank you for your patience.
Our team continues to investigate an issue with one of our cloud service partners that may cause error messages in both the Admin Area and storefronts. Our engineering team is actively working with the provider to identify the cause and restore normal service as quickly as possible. We will share updates as soon as we have more information.
Our team continues to actively work with our cloud service partner to investigate the issue impacting the Admin Area and storefronts. We are monitoring the situation closely and remain focused on restoring normal service as quickly as possible. Further updates will be shared as soon as more details are available. Thank you for your continued patience.
Our team has identified the issue with our cloud service partner that was causing errors in the Admin Area and storefronts. We are actively working on restoring the platform and are beginning to see improvements. We will continue to monitor progress closely and provide further updates as recovery continues.
Services in the Admin Area and storefronts are now back online. We are continuing to monitor the platform closely to ensure stability and confirm that the issue is fully resolved.
The issue impacting access to the Admin Area and storefronts has been fully resolved. Our team has confirmed that services are stable and no further issues have been identified. Thank you for your patience and understanding while we worked to restore service.
# Summary On September 6, 2025, Uscreen experienced a service outage that affected access to the Admin Area, storefronts, and API. Our engineering team identified the root cause, implemented a workaround, and fully restored services within approximately 1 hour and 45 minutes. # Root Cause The outage was triggered by an issue with one of our third-party service providers responsible for email delivery. A network failure on the service provider's end prevented our platform from connecting to the provider, which in turn caused issues with our own infrastructure, preventing Uscreen from functioning properly. # Resolution Our engineering team isolated the dependency and deployed a temporary fix that allowed the platform to operate independently of the impacted service provider. This restored storefronts, the Admin Area, and API functionality while email delivery remained degraded. Once the provider resolved their issue, email services returned to normal and all systems were fully restored. # Preventative Measures To reduce the likelihood of similar incidents in the future, we are making the following improvements: * Adjusting our system architecture so external services cannot cause full outages. * Updating health checks to ensure that Uscreen remains available even if non-critical services are unavailable. * Introducing redundancy and fallback mechanisms for critical email communications delivery. * Reviewing other external service dependencies to ensure resilience. # Moving Forward We know how disruptive incidents like this can be and we sincerely apologize for the impact it had. While every online platform relies on certain third-party providers, our focus is on making sure that interruptions from those services have limited impact for Uscreen. Thank you for your continued trust and for being part of the Uscreen community.
Some functionality such as granting access after a purchase or content moderation of newly uploaded videos is working with significant delays.
Our team identified a scaling issue of the background job processing system: more tasks were created than the job could process.
All systems have since recovered and are operating normally, the service was successfully scaled. We are monitoring the system.
This incident has been resolved.
Some users experienced issues with parts of the platform loading slowly or not responding as expected. This was due to a temporary disruption with a third-party server provider and lasted for approximately 45 minutes. All systems have since recovered and are operating normally. We’ll continue monitoring to ensure everything remains stable.
Our CDN partner is currently experiencing issues that are impacting video playback and ingest across the platform. You may notice: - Live streams not starting or loading properly - Video in your catalog failing to play or taking longer to load - Delays when uploading new videos We’re actively monitoring the situation and working with our CDN partner as they implement a fix. We’ll continue to share updates as we have them.
Our CDN partner has implemented a fix for the playback and ingest issues. At this time, video loading and streaming should begin returning to normal. They are continuing to monitor the situation closely to ensure full stability. We’ll post another update if any further issues arise.
The issue has been fully resolved. Video playback, live streams, and uploads are all functioning normally again. Our CDN partner has confirmed that the fix is stable and no further issues are expected. Thanks for your patience while this was being addressed.
Timeframe: 11:14 AM – 11:23 AM Eastern We experienced a brief disruption due to a DDoS attack that impacted platform performance for approximately 9 minutes. Our team responded quickly, mitigated the attack, and deployed additional protections to prevent further impact. Service has been fully restored and remains stable. Thank you for your patience.
One of our cloud infrastructure providers, Google, is currently experiencing a widespread outage affecting services across the internet. As a result, you may notice: - Slow loading times - Errors accessing certain parts of the platform - Issues uploading content or viewing analytics We’re closely monitoring the situation and will post updates as we learn more. Thanks for your patience as Google works to resolve this.
We’re starting to see signs of recovery as Google works to restore its cloud services. Some areas of the platform should begin functioning normally again. We’ll continue monitoring closely and share further updates as things stabilize.
Our cloud provider has made improvements to their systems, and we’re now seeing full stability across our platform. While the broader outage may still be impacting some services online, everything is functioning normally for our services. We’ll continue to monitor, but at this time, all systems are operational on our end.