Uscreen 장애 기록

We're investigating reports that some pages are loading without proper styling for a subset of visitors on certain customer websites. Our team is actively working to identify the root cause and resolve it, and we'll post updates here as we learn more

We are monitoring the situation. If you're still seeing pages load without styling, try a hard refresh (Ctrl/Cmd + Shift + R) or clear your browser cache — this should resolve it immediately

We've applied a change that has resolved the immediate issue — visitors on affected customer websites should no longer see pages loading without proper styling. The underlying cause appears to be related to how DNS records for one of our infrastructure providers were being resolved. We've reached out to them for more detail and will share an update once we have a full explanation. If you're still seeing unstyled pages, your browser or device may have cached a previous, incorrect DNS result. Forcing a fresh lookup resolves it immediately: Chrome: 1. Paste into the address bar and press Enter:  chrome://settings/security 2. Scroll to "Use secure DNS" and turn it ON. 3. Select "Cloudflare (1.1.1.1)" from the dropdown. 4. Refresh. Edge: 1. Paste:  edge://settings/privacy 2. Scroll to "Use secure DNS" and turn it ON. 3. Select "Cloudflare (1.1.1.1)". 4. Refresh. Firefox: 1. Paste:  about:preferences#privacy 2. Scroll to "DNS over HTTPS", set to "Max Protection". 3. Choose "Cloudflare". 4. Refresh.

We are currently investigating an issue causing degraded performance in Communities. We'll post updates as soon as they are available. Thank you for your patience.

We are continuing to investigate this issue.

We've identified the cause of the slow page loads and errors affecting the storefront, admin portal, and APIs. Following an earlier database upgrade, a subset of queries are running significantly slower than expected. Our team is implementing a fix now. We'll share another update as soon as the fix is deployed and we begin monitoring recovery.

We've rolled out a fix and performance is returning to normal, while we improving the cache ratio. We're continuing to monitor closely to make sure everything stays stable. We'll share a final update once we're confident the issue is fully resolved.

The earlier fix did not hold and we're seeing degradation return. Engineering is investigating further. We'll update as soon as we have more.

Recovery is underway and performance is improving across our services. We're proceeding carefully to ensure stability as everything returns to normal. You may still see some degradation for a short while longer. We'll update as soon as full recovery is confirmed.

Good news: storefronts and the Admin Portal are recovering. We're now focused on bringing the APIs that power mobile and TV apps back online. We'll update again as soon as more is restored.

Storefronts and the Admin Portal are now fully back online and running normally. Thanks for your patience while we worked through this. The APIs that power mobile and TV apps are still affected — that's our team's full focus right now. We'll update again as soon as those are restored.

Mobile and TV app APIs are coming back online. We're continuing to monitor as services fully recover. We'll share another update once everything is operating normally.

All services are back online and operating normally. We're continuing to monitor closely to ensure everything remains stable. We'll share a final update once we're confident the issue is fully resolved.

This incident is now resolved. All services are operating normally. A detailed post-mortem will be shared shortly. Thank you for your patience.

# Incident Post-Mortem: May 4, 2026 Database Upgrade ## A note from our team On Monday, May 4, 2026, Uscreen experienced a significant performance incident following a planned PostgreSQL database upgrade. Storefronts, the Admin Portal, and the APIs that power our mobile and TV apps were degraded or unavailable for portions of the day for many of our customers. We know that when your storefront is slow, or your viewers can’t watch, it’s not just a technical issue — it affects your business, your subscribers, and your reputation. We’re deeply sorry for the disruption, and we owe you a clear, honest account of what happened, how we resolved it, and what we’re doing to make sure this doesn’t happen again. This document is that account. ## What happened, in plain language We perform major database upgrades approximately once per year to keep the platform secure, fast, and on a supported version. The upgrade itself completed successfully and on schedule. The problem started **after** the upgrade, when production traffic returned to the freshly upgraded database. After a major version upgrade, a database needs to “warm up” — its internal memory cache, which normally holds the most frequently accessed data ready for instant retrieval, starts empty. As traffic returned, our database needed to load enormous amounts of data into that cache simultaneously. We had not anticipated how severely this cold-start effect would compound with our current workload, and the database fell behind. Requests began to queue. Queries that normally took milliseconds took seconds. The platform slowed down dramatically for many customers. It’s worth being direct about this: **we tested this upgrade beforehand using a canary approach** — running it in a non-production environment to validate it. And we performed a similar major upgrade last year without significant issues. What changed between then and now was the **size of our data**. Our platform has grown substantially over the past year, and the cold-start memory pressure that was manageable last year became severe at our current scale. Our pre-production environment did not faithfully replicate that scale, so it did not expose the problem before it hit production. This is the specific gap we are closing — see “What we’re changing” below. Compounding the cold-start problem, a small number of high-volume application queries — ones that performed acceptably under normal conditions — became significantly slower under memory pressure. This created a feedback loop: slow queries held connections open, those connections couldn’t serve other requests, and the backlog grew faster than the system could clear it. It took our team longer than we would have liked to correctly diagnose the structural cause. Our initial hypotheses — that statistics needed rebuilding, that specific code-level query fixes would resolve it — turned out to be only partial answers. The real fix required more decisive action: stopping the application entirely to let the database stabilize, expanding the database’s memory capacity, and then gradually bringing services back online. ## Timeline \(all times Eastern\) | Time | What happened | | --- | --- | | **12:00 AM** | Planned maintenance window begins. Database upgrade initiated. | | **12:15 AM** | Upgrade phase completes successfully — faster than planned. | | **~12:15–3:30 AM** | Backups and replica provisioning continue in the background. Application traffic returns. | | **~3:30 AM** | Performance begins to degrade as production traffic ramps up. Database memory cache is cold and under heavy load. | | **4:00–6:00 AM** | Severe degradation. Average response times rise to 4 seconds. End-user experience is heavily affected, particularly for customers in European time zones whose business day is starting. | | **6:00–11:00 AM** | Partial recovery as cache warms naturally, but performance remains unstable with intermittent slowness and errors. | | **7:37 AM** | First public status page acknowledgment of degraded performance. _\(We should have posted earlier — see “What we’re changing” below.\)_ | | **8:31 AM** | Root cause identified as post-upgrade query performance issues. | | **9:12 AM** | First targeted fix deployed — code-level changes to reduce load from a high-volume query pattern, plus routing more read traffic to a replica database. | | **9:42 AM** | The first fix improved things briefly but did not hold. Performance degraded again. We publicly acknowledged this on the status page and resumed investigation. | | **~10:00–10:30 AM** | Decision to take more decisive action. Application stopped to let the database fully drain queued work. Database resized to provide additional memory headroom. | | **10:30 AM** | Application services begin coming back online in stages, allowing the database cache to warm progressively rather than all at once. | | **10:47 AM** | Storefronts and Admin Portal recovering. | | **10:58 AM** | Storefronts and Admin Portal fully restored. | | **11:22 AM** | Mobile and TV app APIs recovering. | | **11:31 AM** | All services restored to normal operation. | | **Through afternoon** | Continued close monitoring. Performance held stable. | ## What was affected Different services were affected at different times and to different degrees: * **Storefronts** — Slow page loads and intermittent errors during the worst windows \(approximately 4:00–11:00 AM ET, with severity peaking 4:00–6:00 AM and again briefly around midday\) * **Admin Portal** — Severe slowness and “Unexpected Server Error” pages for many creators * **Mobile and TV apps \(API V1 and V2\)** — Slow or failing requests, recovering last * **Communities** — The initial area where our investigation began, where high-volume queries were among the early contributors to the database load Throughout the incident, **no customer data was lost, no security boundaries were affected, and no payment or subscription data was compromised.** The issue was performance, not integrity. ## What we did to resolve it Our response unfolded in two layers. The application-layer fixes \(the “minor contributors”\) helped, but the database-layer actions \(the “major contributors”\) actually ended the incident. ### Major contributors to recovery 1. **Stopped the application entirely.** When new slow queries were arising faster than we could resolve them, we made the deliberate decision to halt all application traffic for a short window so the database could finish its work and return to a stable baseline. This was a planned, controlled outage of approximately 10–15 minutes — chosen because partial mitigation was no longer keeping pace with the problem. 2. **Resized the database to add memory capacity.** We moved to a larger database instance with significantly more memory available for the cache. This addressed the structural root cause: our previous configuration didn’t have sufficient memory headroom for the platform’s current workload during a cold start. 3. **Brought services back online gradually.** Rather than restoring all traffic at once and risking a repeat of the cold-cache problem, we restarted application servers in stages. This lets the database cache warm progressively, services come back in a controlled order, and confidence builds at each step. Storefronts came back first, followed by the Admin Portal, then the mobile and TV APIs. 4. **Disabled a database optimization feature \(JIT compilation\)** that, under our specific workload during the incident, was adding overhead rather than reducing it. This will be re-evaluated and tuned separately. ### Minor contributors to recovery These code-level fixes shipped during the incident genuinely helped — they reduced load on the database and improved query performance — but on their own, they couldn’t resolve the structural problem. 1. **High-volume query optimization.** A frequently-fired query that scanned a large amount of historical data on every call was bound to only look at a recent time window, dramatically reducing the work required per request. 2. **Read traffic routing.** We routed more of our API V2 read traffic to replica databases, taking pressure off the primary. 3. **Targeted index improvement.** An index was added to support a frequent storefront query. We are continuing to ship these and similar improvements this week. They make the platform faster and more resilient regardless of upgrades. ## Why did this take longer than it should have? We want to be honest about this: the incident lasted longer than it needed to. Three things contributed: 1. **Our initial diagnosis was incomplete.** The first hypotheses we pursued — that database statistics needed rebuilding, that specific code-level query fixes would resolve it — were partial truths. The structural cause \(insufficient memory for cold-cache recovery under our current workload\) wasn’t identified until later in the incident. We chased the symptoms before fully understanding the system pressure. 2. **We were slower to communicate publicly than we should have been.** Our first status page update came hours after degradation began. Customers experiencing slowness deserved to hear from us sooner, even if all we could say was “we’re investigating.” 3. **Our pre-production testing didn’t catch the scale-dependent failure mode.** We did run this upgrade in a canary environment first, and a comparable upgrade last year went smoothly. But our test environment didn’t replicate the full size of our current production data, so the cold-start memory pressure that hit us at production scale never showed up in testing. Our growth over the past year crossed a threshold our procedures hadn’t been updated to account for. We’re owning these honestly because they are exactly the things we can change. ## What we’re changing so this doesn’t happen again This is the most important section of this document. Several improvements are already in motion. ### Database capacity and configuration * **The larger database instance is now permanent.** We are not returning to the previous configuration. Our memory headroom is now sized appropriately for our current and projected workload. * **We are reviewing additional database configuration** settings \(memory tuning, JIT settings, connection limits\) in light of the workload patterns we observed during the incident. * **Specific high-volume queries are being permanently optimized,** including the ones that became hotspots during the incident — independent of any future upgrade. ### Upgrade procedures * **A formal major-version upgrade runbook is being written** with mandatory pre- and post-upgrade steps, including cache pre-warming, staged traffic restoration, and an extended observation period before declaring success. * **We are upgrading our pre-production canary environment to better mirror production scale.** Our canary process worked correctly for last year’s upgrade, but production data growth has outpaced our test data. Going forward, the canary will be sized to faithfully reproduce production-scale memory pressure, query patterns, and connection load, so that scale-dependent failure modes show up before they reach customers. * **Pre-upgrade query performance baselines** will be captured, so any regression after an upgrade is immediately visible by comparison. * **Capacity reviews are now part of every major upgrade plan.** Before any future major upgrade, we will explicitly verify that the database's memory and connection capacity are sized for cold-start recovery, not just for steady-state operation. ### Monitoring and alerting * **New alerts on response-time and database-pressure metrics** so that degradation is detected automatically and the on-call team is paged within minutes of onset, not after customer reports accumulate. * **Database memory pressure and lock contention** are being added to our standard dashboards as first-class signals. ### Communication * **Our internal incident playbook now requires a public status page update within 15 minutes** of any confirmed customer-facing degradation, regardless of whether we have a diagnosis yet. * **We will publish post-mortems for significant incidents** as a standard practice, not just when asked. ## Looking forward We want to leave you with three things: **This kind of major maintenance happens approximately once per year.** It’s not a routine event, and it’s not the new normal. We performed a similar upgrade last year that went smoothly, and most days, weeks, and months on Uscreen pass without anything like this incident. Today was a bad day, not a sign of bad days ahead. **Every change above is real.** We’re not gesturing at improvement — we are funding it, scheduling it, and assigning it to specific people. The next major database upgrade will be a fundamentally different operation from this one. **We take this seriously because you trust us with your business.** Your subscribers, your revenue, your reputation — they all depend on Uscreen being available and fast. We did not meet that standard yesterday. We are going to. If you have questions or concerns, or were materially affected and want to discuss the impact on your business, please reach out to your account contact or to [support@uscreen.tv](mailto:support@uscreen.tv). We are here, and we want to talk. — The Uscreen Engineering and Operations team ## Appendix: Technical detail \(for technical customers\) For customers with engineering teams who want a deeper view: * **Database:** Managed PostgreSQL, recently upgraded to the current major version * **Working set:** Approximately 1 TB OLTP database * **Pre-incident memory cache configuration:** Undersized for the cold-start scenario after a major upgrade; resized post-incident to provide significant additional headroom * **Peak symptoms during the incident:** Database connections fully saturated, recent cache hit ratio dropped to ~36%, significant lock contention from concurrent backends competing for buffer access, dozens of concurrent long-running queries \(>10s\), application-server p95 latency at 5.25s, Apdex bottoming at 0.27 * **Recovery state:** Connections at normal idle levels, cache hit ratio 99.4–99.8%, p95 latency 0.29s, Apdex 0.99 * **Initial diagnostic dead-ends ruled out during the incident:** missing planner statistics \(replica counters were misleading — actual statistics had replicated correctly\), collation library drift \(verified intact\), and major configuration drift \(verified carried over correctly from the prior version\) * **Application-layer fixes shipped during the incident:** bounded a high-volume time-series query to a recent time window, routed additional read traffic to replicas, and addressed related code paths * **Outstanding follow-up technical work** includes additional indexes on hot lookup columns, improved indexing for a JSONB lookup pattern, partitioning of a large time-series table, and enabling automatic GET-request routing to replicas at the framework level

We have identified an issue following a scheduled maintenance. Some functionality may be degraded while we work on a fix. We'll post updates as soon as they are available. Thank you for your patience.

We are continuing to work on a fix for this issue.

The issue has been resolved and we are monitoring for any further errors. Thank you for your patience.

This incident has been resolved.

We're seeing reports of intermittent errors and slower-than-usual load times affecting admin dashboards and storefronts. Our team is currently investigating.

Our engineering team has identified the cause of the slowness and intermittent errors affecting admin dashboards and storefronts, and is actively implementing a resolution.

A fix has been implemented, and we're currently seeing full restoration of services across admin dashboards and storefronts. Our team will continue to monitor closely to ensure stability.

This incident has been resolved. Services across admin dashboards and storefronts are fully restored and operating normally. Thank you for your patience while our engineering team worked through this.

We experienced a brief service disruption lasting approximately 2 minutes. The issue has been fully resolved and all systems are operating normally. We apologize for any inconvenience caused.

We’re monitoring an issue with a third-party provider that may impact video uploads. Some customers may see upload failures while the provider works to restore normal service. Our team is tracking this closely and will post an update as soon as we have more details.

The third-party provider has implemented a fix and is currently monitoring results. We’re continuing to monitor on our side as well, and we’ll share another update as soon as we can confirm full resolution.

The third-party provider has confirmed that the earlier issue has been resolved. Video uploads are now being processed normally.

We're aware that some storefronts hosted on Uscreen subdomains may be experiencing slow load times or intermittent errors. This is due to an issue with one of our third-party providers. Our team is actively monitoring the situation. We'll post an update here as soon as we have more information from the 3rd party provider.

Our third-party partner has resolved the issue, and our team continues to monitor for any changes. All storefronts are operational.

We experienced a brief performance degradation due to a DDoS attack that impacted platform performance for approximately 4 minutes. Our team responded quickly, mitigated the attack, and deployed additional protections to prevent further impact. Performance has been fully restored and remains stable. Thank you for your patience.

A third-party CDN partner is experiencing issues that are causing video playback errors. Some videos may not play as expected. We're in close communication with our CDN partner and will provide updates as we have them.

Our CDN partner has implemented a fix for the video playback errors. We are currently monitoring to ensure videos are functioning normally. We will provide a final update once we confirm the issue is fully resolved.

The video playback issues have been fully resolved. Our CDN partner's fix has been successfully implemented, and we have confirmed that videos are functioning normally.

A third-party CDN partner is experiencing issues that are causing video playback errors. Some videos and livestreams may not play as expected. We're in close communication with our CDN partner and will provide updates as we have them.

Our CDN partner has identified the issue causing video playback errors. They are currently working on implementing a fix. Some videos and livestreams may still not play as expected while the fix is being deployed. We will continue to provide updates as we receive them.

Our CDN partner has implemented a fix for the video playback errors. We are currently monitoring to ensure videos and livestreams are functioning normally. We will provide a final update once we confirm the issue is fully resolved.

The video playback issues have been fully resolved. Our CDN partner's fix has been successfully implemented and we have confirmed that videos and livestreams are functioning normally.

We are currently experiencing an issue that may affect some of our services. Our team is actively investigating and working to resolve it as quickly as possible. Thank you for your patience.

Our team continues to investigate an issue with one of our cloud service partners that may cause error messages in both the Admin Area and storefronts. Our engineering team is actively working with the provider to identify the cause and restore normal service as quickly as possible. We will share updates as soon as we have more information.

Our team continues to actively work with our cloud service partner to investigate the issue impacting the Admin Area and storefronts. We are monitoring the situation closely and remain focused on restoring normal service as quickly as possible. Further updates will be shared as soon as more details are available. Thank you for your continued patience.

Our team has identified the issue with our cloud service partner that was causing errors in the Admin Area and storefronts. We are actively working on restoring the platform and are beginning to see improvements. We will continue to monitor progress closely and provide further updates as recovery continues.

Services in the Admin Area and storefronts are now back online. We are continuing to monitor the platform closely to ensure stability and confirm that the issue is fully resolved.

The issue impacting access to the Admin Area and storefronts has been fully resolved. Our team has confirmed that services are stable and no further issues have been identified. Thank you for your patience and understanding while we worked to restore service.

# Summary On September 6, 2025, Uscreen experienced a service outage that affected access to the Admin Area, storefronts, and API. Our engineering team identified the root cause, implemented a workaround, and fully restored services within approximately 1 hour and 45 minutes. # Root Cause The outage was triggered by an issue with one of our third-party service providers responsible for email delivery. A network failure on the service provider's end prevented our platform from connecting to the provider, which in turn caused issues with our own infrastructure, preventing Uscreen from functioning properly. # Resolution Our engineering team isolated the dependency and deployed a temporary fix that allowed the platform to operate independently of the impacted service provider. This restored storefronts, the Admin Area, and API functionality while email delivery remained degraded. Once the provider resolved their issue, email services returned to normal and all systems were fully restored. # Preventative Measures To reduce the likelihood of similar incidents in the future, we are making the following improvements: * Adjusting our system architecture so external services cannot cause full outages. * Updating health checks to ensure that Uscreen remains available even if non-critical services are unavailable. * Introducing redundancy and fallback mechanisms for critical email communications delivery. * Reviewing other external service dependencies to ensure resilience. # Moving Forward We know how disruptive incidents like this can be and we sincerely apologize for the impact it had. While every online platform relies on certain third-party providers, our focus is on making sure that interruptions from those services have limited impact for Uscreen. Thank you for your continued trust and for being part of the Uscreen community.

Some functionality such as granting access after a purchase or content moderation of newly uploaded videos is working with significant delays.

Our team identified a scaling issue of the background job processing system: more tasks were created than the job could process.

All systems have since recovered and are operating normally, the service was successfully scaled. We are monitoring the system.

This incident has been resolved.

Some users experienced issues with parts of the platform loading slowly or not responding as expected. This was due to a temporary disruption with a third-party server provider and lasted for approximately 45 minutes. All systems have since recovered and are operating normally. We’ll continue monitoring to ensure everything remains stable.

Our CDN partner is currently experiencing issues that are impacting video playback and ingest across the platform. You may notice: - Live streams not starting or loading properly - Video in your catalog failing to play or taking longer to load - Delays when uploading new videos We’re actively monitoring the situation and working with our CDN partner as they implement a fix. We’ll continue to share updates as we have them.

Our CDN partner has implemented a fix for the playback and ingest issues. At this time, video loading and streaming should begin returning to normal. They are continuing to monitor the situation closely to ensure full stability. We’ll post another update if any further issues arise.

The issue has been fully resolved. Video playback, live streams, and uploads are all functioning normally again. Our CDN partner has confirmed that the fix is stable and no further issues are expected. Thanks for your patience while this was being addressed.

Timeframe: 11:14 AM – 11:23 AM Eastern We experienced a brief disruption due to a DDoS attack that impacted platform performance for approximately 9 minutes. Our team responded quickly, mitigated the attack, and deployed additional protections to prevent further impact. Service has been fully restored and remains stable. Thank you for your patience.

One of our cloud infrastructure providers, Google, is currently experiencing a widespread outage affecting services across the internet. As a result, you may notice: - Slow loading times - Errors accessing certain parts of the platform - Issues uploading content or viewing analytics We’re closely monitoring the situation and will post updates as we learn more. Thanks for your patience as Google works to resolve this.

We’re starting to see signs of recovery as Google works to restore its cloud services. Some areas of the platform should begin functioning normally again. We’ll continue monitoring closely and share further updates as things stabilize.

Our cloud provider has made improvements to their systems, and we’re now seeing full stability across our platform. While the broader outage may still be impacting some services online, everything is functioning normally for our services. We’ll continue to monitor, but at this time, all systems are operational on our end.

We’re currently investigating an issue causing increased loading times for some users. Our team is looking into it and working to identify the root cause. We’ll share more updates as soon as we have them. Thanks for your patience!

Our team has identified the root cause of the increased loading times and is working on a resolution. We’ll provide another update as soon as we have more information. Thank you for your continued patience.

We’ve deployed a fix, and the platform is currently recovering. Our team will continue to monitor performance closely to ensure everything remains stable. Thank you for your continued patience!

The issue impacting loading times has been fully resolved. Platform performance has continued to be stable. Thanks again for your patience and understanding!

Our engineering team is actively investigating the issue and working on a resolution. This affects access to the Admin Area and Catalog pages.

Our engineering team is investigating the issue and is working on implementing a solution. We're starting to see improvements.

Our team has restored service to normal operations, however, they continue to investigate the root cause of the issue. We appreciate your continued patience.

Our team has identified the issue's root cause and implemented a fix. Thank you for being so patient while the team worked to fully resolve the issue.

# **What Happened?** On 3/13/2025, we detected a service disruption affecting all users. During this time, some catalog pages and the Uscreen admin area experienced errors, preventing store functionality from working as expected. The issue lasted approximately 19 minutes, with a complete outage of 5-6 minutes before services began recovering. # **What Was the Impact?** * **Total Duration:** ~19 minutes \(15:01 - 15:20 UTC\) * **Full Service Disruption:** ~5-6 minutes * **Degraded Performance:** 15:08 - 15:20 UTC * **Scope:** All stores * **Recovery:** Initial recovery started at 15:08 UTC, with full restoration by 15:20 UTC # **Is Everything Working Now?** Yes, the service has been fully restored and is operating normally. # **What Caused the Issue?** After investigating, we identified that: * A feature in the community section was not optimized for stores with a high number of users. * The system was executing resource-intensive operations, causing requests to take longer than expected. * Under heavy load, the database became overwhelmed, leading to errors. * Our monitoring system did not provide useful diagnostics during the incident, delaying troubleshooting. # **What Are We Doing to Prevent This in the Future?** We’ve taken immediate steps to mitigate the issue and are working on long-term improvements: ## **Immediate Fixes:** * Applied a temporary adjustment to reduce the server load caused by the feature. * Upgraded our monitoring system for better incident visibility and faster response times. ## **Planned Improvements:** * Optimizing the affected feature to handle high-traffic scenarios efficiently. * Implementing measures in our mobile apps to limit excessive requests. ‌ We sincerely apologize for any disruption this may have caused and appreciate your patience as we work to enhance system stability.

We experienced an issue with a build release that affected the functionality of modal window popups in the Admin area of Uscreen, causing disruptions for some users. Our team identified the root cause and implemented a fix. The issue has been fully resolved, and normal functionality has been restored for these modal windows.

We are currently looking into an issue causing slow loading times for both the Admin and Storefront pages.

Our team has identified the root cause of the slowdown affecting the Admin and storefront pages. We are currently working on a fix.

Our team continues to work towards a fix for the issue causing slow loading times for the catalog and admin pages.

Our team continues to work towards a solution for this issue. Thank you for your continued patience.

A fix has been implemented, and we are seeing improvements in the Admin Area and Storefront pages. Our team is actively monitoring the situation and tracking the improvements.

The issue has been successfully resolved. Thank you for your patience during this process.

# Postmortem **Date of Incident: 1/6/25** ## Summary: We recently experienced degraded performance and limited service interruptions caused by unprecedented traffic growth and an issue within our database infrastructure. A CPU resource leak in our database provider’s system contributed to high resource utilization, compounding the challenge of meeting demand. This incident tested our system’s capacity, and while it caused temporary disruptions, it also highlighted opportunities for immediate and long-term improvements. ## Root Cause * **Increased Demand:** * Platform traffic and user activity tripled compared to typical levels, leading to an unexpected surge in database load. * This spike caused memory saturation and an overabundance of connections. * **CPU Resource Leak:** * A resource leak in the database provider’s infrastructure led to persistent CPU spikes, limiting system efficiency. * This issue prevented the system from scaling effectively to handle workloads. * **Inefficient Resource Allocation:** * Idle database connections and unoptimized queries further stressed the infrastructure, reducing its ability to respond to peak demand. ## Resolution ### **Immediate Actions:** * **Increased Database Capacity:** * Doubled CPU and memory resources in our database instance to handle the increased load. * **Instance Restarts:** * Refreshed instances to eliminate stale connections and stabilize performance. * **Cleared Stale Connections:** * Optimized active connection counts, reducing strain on the system. ### Short-Term Adjustments: * Adjusted query handling to improve load distribution. * Enhanced monitoring tools to identify potential resource leaks sooner. ‌ This incident underscores the challenges of balancing rapid growth with infrastructure resilience. By immediately increasing database capacity and addressing inefficiencies, we stabilized the platform for now. Moving forward, we are committed to strengthening our systems through proactive scaling, deeper collaboration with our providers, and better resource management to support your continued success on our platform.