Crowdcast 장애 기록

1. 2026년 8월 13일 AM 1:04 UTCinvestigating

   We’re currently investigating an issue that is preventing users from watching events from the Crowdcast mobile app. Our team is actively working to resolve it as quickly as possible. **In the meantime, you can continue watching events and replays from a mobile or desktop browser.** We’ll share another update as soon as we have more information. Thank you for your patience.

1. 2026년 8월 3일 PM 6:54 UTCresolved

   Crowdcast는 호스트가 라이브 및 녹음에 등장하는 이슈를 경험했지만 참석자는 방송을 볼 수 없습니다. 엔지니어는 신속하게 과부하된 데이터베이스로 원인을 확인했습니다. 서비스 복원에, 그들은 데이터베이스 구성을 전환, 간단한 결과로 (~5 분) 이벤트가 재설정 및 서비스 복구 전에 사이트 중단. 우리는 미래의 유사한 문제를 방지하기 위해 큰 사건에 대한 인프라 제공을 개선하고 있습니다.

공식 인시던트 업데이트를 자동 번역했습니다.

1. 2026년 5월 6일 PM 12:05 UTCinvestigating

   The team is actively investigating an issue that is affecting some hosts when broadcasting live. Attendees may have to refresh to view the live feed. We'll update you as soon as we can.

2. 2026년 5월 6일 PM 1:40 UTCresolved

   We identified and resolved an issue that prevented events from loading correctly on the initial page load. As a result, some users needed to refresh the page before the livestream would appear. This has now been fixed, and events should load properly the first time.

1. 2026년 4월 29일 PM 4:17 UTCinvestigating

   The team is actively investigating an issue affecting our service that is causing increased buffering in live streams. This does not affect recordings, and we’ll provide an update as soon as possible.

2. 2026년 4월 29일 PM 5:14 UTCidentified

   We’ve deployed an update that mitigates the issue and are continuing to work on resolving the underlying cause of increased buffering in some sessions.

3. 2026년 4월 29일 PM 9:11 UTCmonitoring

   We’ve confirmed the issue as originating with our CDN provider, Cloudflare, and have implemented a temporary mitigation. We’re actively working with the provider to resolve the underlying cause and will share another update as soon as more information is available.

4. 2026년 4월 30일 PM 4:27 UTCresolved

   Our updates remain stable, and we are continuing to implement additional improvements to help ensure that issues at Cloudflare have a reduced impact on our services.

1. 2026년 4월 23일 PM 2:58 UTCresolved

   We experienced an issue that temporarily prevented some sessions from going live due to a brief compute disruption within our recording infrastructure. The problem was limited in duration, and we are actively implementing a patch to address the underlying cause and help ensure it does not happen again.

1. 2026년 3월 17일 PM 5:17 UTCinvestigating

   The team is actively investigating an issue affecting our service. We'll update you as soon as we can.

2. 2026년 3월 17일 PM 5:33 UTCidentified

   We’re currently experiencing a capacity shortage with our cloud provider (Hetzner). Hetzner has run out of available capacity in the region(s) we use, which is limiting our ability to provision additional resources and may impact scaling and recovery actions. • **Current status:** Investigating / Mitigating • **Impact:** Potential service degradation due to constrained scaling capacity (e.g., slower provisioning, delayed failover, or limited ability to add nodes). • **What we’re doing:** Actively working to secure additional capacity, including requesting more allocation from Hetzner and evaluating alternative capacity options. • **Next update:** We’ll share another update as soon as additional capacity is confirmed or our mitigation plan changes.

3. 2026년 3월 17일 PM 5:51 UTCidentified

   We’ve identified an issue affecting our services where some hosts may experience difficulty going live. Our team is actively investigating, and we’ll share an update as soon as more information is available.

4. 2026년 3월 17일 PM 5:52 UTCidentified

   ** Identified** We’re currently experiencing a capacity shortage with our cloud provider (Hetzner). Hetzner has run out of available capacity in the region(s) we use, which is limiting our ability to provision additional resources and may impact scaling and recovery actions. • **Current status:** Investigating / Mitigating • **Impact:** Potential service degradation due to constrained scaling capacity (e.g., slower provisioning, delayed failover, or limited ability to add nodes). • **What we’re doing:** Actively working to secure additional capacity, including requesting more allocation from Hetzner and evaluating alternative capacity options. • **Next update:** We’ll share another update as soon as additional capacity is confirmed or our mitigation plan changes.

5. 2026년 3월 17일 PM 7:39 UTCmonitoring

   We’ve secured additional compute capacity, and the platform is currently stable. We’re continuing to monitor key signals (capacity headroom, provisioning queues, and error rates) to ensure stability remains sustained. • **Current status:** Monitoring • **Impact:** No current impact; capacity constraints have been mitigated for now. • **What we’re doing:** Ongoing monitoring and readiness to scale further if demand or capacity conditions change. • **Next update:** We’ll provide another update if conditions change or once we confirm sustained stability.

6. 2026년 3월 17일 PM 7:55 UTCresolved

   We’ve confirmed the scaled-up resources are holding under current load, and service remains stable. The immediate capacity constraints are **resolved for now**. • **Current status:** Resolved • **Impact:** No current impact; services are operating normally. • **What we’re doing next:** We’re investigating and designing a multi-cloud capacity strategy to reduce dependence on a single provider’s regional availability and enable provisioning across multiple cloud providers concurrently.

1. 2026년 3월 11일 PM 2:21 UTCresolved

   At **8:54 AM ET**, we experienced a **9-minute service disruption** in our livestreaming infrastructure that impacted **real-time audience playback**. During this window, viewers may have seen interruptions or inability to watch live content. • **Start time:** 8:54 AM ET • **Duration:** 9 minutes • **Impacted:** Live audience streaming sessions • **Not impacted:** Recording capture and stored playback assets The incident was caused by a previously unobserved failure mode. Mitigation has been applied, and we are implementing additional patches and safeguards to prevent recurrence.

1. 2026년 3월 5일 AM 10:28 UTCinvestigating

   The team is actively investigating an issue affecting our service. We'll update you as soon as we can.

2. 2026년 3월 5일 PM 12:27 UTCmonitoring

   We've scaled our systems to handle significantly higher loads from multiple simultaneous large-scale events. Our systems are currently stable, and we continue to monitor them closely.

3. 2026년 3월 5일 PM 2:32 UTCresolved

   **All systems remain stable.** We have identified the root cause of the cascading issues that occurred under high load and are deploying an update today to prevent this from happening again. Thank you for your continued patience.

1. 2026년 2월 13일 AM 1:27 UTCinvestigating

   The team is actively investigating an issue affecting our service. We'll update you as soon as we can.

2. 2026년 2월 13일 AM 1:51 UTCidentified

   We’re hitting scale limits on our real-time chat and interaction services due to a large event. We’re increasing capacity now and will restore full service as quickly as possible.

3. 2026년 2월 13일 AM 2:53 UTCmonitoring

   Services are coming back online, and we’re monitoring performance.

4. 2026년 2월 13일 AM 3:10 UTCresolved

   **Impact:** Some real-time updates were delayed for a portion of users. No data was lost. **Timeline:** Started at 7pm, peaked at 7:10pm, and resolved by 9:30pm ET. **Cause:** A sudden traffic spike from a high-scale event pushed past our autoscaling limits. **What we did:** Added capacity and tuned autoscaling. **Now:** Everything is stable and performing normally. **What’s next:** Stronger scaling safeguards, smarter rate limiting, and improved monitoring.

1. 2025년 11월 18일 AM 11:47 UTCidentified

   We’re actively investigating a service disruption caused by our third‑party provider, Cloudflare. This issue is affecting us and many other services, including Zoom, Uber, Twitter, OpenAI, and AWS. We’ll share updates as soon as we have more information.

2. 2025년 11월 18일 PM 12:09 UTCmonitoring

   Cloudflare has acknowledged a global outage and you can follow updates [here](https://www.cloudflarestatus.com/incidents/8gmgl950y3h7). We'll continue to update as we have more information.

3. 2025년 11월 18일 PM 12:28 UTCmonitoring

   Cloudflare’s ongoing issue is still impacting Crowdcast. Access from some regions is improving, and we’re continuing to monitor and will share updates as we learn more.

4. 2025년 11월 18일 PM 1:45 UTCresolved

   Things look stable now—Crowdcast is back to normal. We’ll keep monitoring to make sure we’re fully in the clear.

1. 2025년 10월 20일 AM 8:54 UTCidentified

   A [major outage](https://health.aws.amazon.com/health/status) at Amazon Web Services (AWS) has taken down our [upstream hosting](https://www.vercel-status.com/) provider as well as large portions of the internet. We're monitoring the situation and will update you as soon as we can.

2. 2025년 10월 20일 AM 9:23 UTCmonitoring

   Our systems are back online. We're continuing to monitor the situation.

3. 2025년 10월 20일 AM 9:47 UTCresolved

   All systems are back to normal.

4. 2025년 10월 20일 PM 4:40 UTCidentified

   [AWS is having a massive outage](https://health.aws.amazon.com/health/status) again that's affecting us and [many other platforms](https://downdetector.com/). We'll update you as soon as we can.

5. 2025년 10월 20일 PM 7:53 UTCresolved

   Our systems are back to normal. We're continuing to monitor the situation at AWS for any other updates.

1. 2025년 9월 29일 PM 11:19 UTCinvestigating

   The web API is timing out and the team is investigating.

2. 2025년 9월 29일 PM 11:19 UTCresolved

   All systems are back to normal. During a busy period on Crowdcast, part of our system that checks if everything is healthy started timing out because of the extra traffic. Those timeouts made our infrastructure think something was wrong, so it restarted parts of the system. These restarts caused brief disruptions that affected access to event data and led to slower or interrupted service for some live events. We’ve adjusted how the health checks work so they’re more tolerant during heavy traffic and have added safeguards to keep the system stable.

1. 2025년 9월 17일 PM 5:18 UTCinvestigating

   The team is actively investigating an issue preventing new events from going live. We'll update you as soon as we can.

2. 2025년 9월 17일 PM 6:27 UTCresolved

   The issue has been resolved. The issue was caused by a change from one of our external providers, which unexpectedly disrupted part of our system. While the cause was outside of Crowdcast, our team resolved it quickly and has already put new safeguards in place to prevent this type of issue from happening again.

1. 2025년 8월 21일 PM 6:57 UTCinvestigating

   We’re experiencing an overall slowdown in the app’s performance. We suspect this may be related to an issue with [Cloudflare’s DNS services](https://www.cloudflarestatus.com/). Our team is actively investigating and will provide updates as soon as possible.

2. 2025년 8월 21일 PM 7:45 UTCresolved

   We've confirmed that the issues were related to an outage at Cloudflare which has now been resolved, and the app is now operating at normal levels.

1. 2025년 7월 9일 PM 2:05 UTCinvestigating

   The team is actively investigating an issue affecting our service. We'll update you as soon as we can.

2. 2025년 7월 9일 PM 2:57 UTCidentified

   We've identified that the issue resides with our cloud provider (Hetzner) not having enough compute resources. We're monitoring the situation closely and working on spinning up our servers on other cloud providers. You can monitor one of the issues here: https://status.hetzner.com/incident/8b1ceeb2-9daa-4971-a1a1-5346fb8fc45a

3. 2025년 7월 9일 PM 3:57 UTCmonitoring

   We were able to spin up additional resources so some sessions are starting to be able to go live. The core issue with Hetzner remains so we'll update you as soon as it's fully resolved. We recommend running a test event before your scheduled event.

4. 2025년 7월 9일 PM 10:51 UTCresolved

   Sessions are back online, and we’re actively adding support for more cloud providers to prevent this kind of issue in the future.

1. 2025년 7월 6일 PM 11:49 UTCinvestigating

   The team is actively investigating an issue affecting our service. We'll update you as soon as we can.

2. 2025년 7월 6일 PM 11:58 UTCresolved

   All systems are back to normal. We’re still digging into what happened to make sure it doesn’t happen again.

1. 2025년 6월 12일 PM 1:24 UTCresolved

   Our backend services recently encountered rate limits due to an unexpected policy change by a vendor. The issue has been resolved, and we’ve implemented safeguards to prevent similar disruptions in the future. All systems are now fully operational.

1. 2025년 5월 13일 PM 4:17 UTCinvestigating

   The team is actively investigating an issue affecting our service. We'll update you as soon as we can.

2. 2025년 5월 13일 PM 4:31 UTCidentified

   We have identified that the issue is with our cloud provider, Vultr, specifically at one of their U.S. data centers that we utilize. We are collaborating with their team to resolve the problem and restore service. We will provide updates as soon as possible.

3. 2025년 5월 13일 PM 4:58 UTCresolved

   The issue with our cloud provider, Vultr, specifically at one of their U.S. data centers that we utilize, has been resolved. We collaborated with their team to resolve the problem and restore service.

1. 2025년 5월 12일 PM 11:13 UTCresolved

   Recording and streaming servers are back online. The issue was caused by an unexpected outage with our downstream cloud provider. We’re actively working on transitioning to a more reliable provider to prevent future disruptions.

2. 2025년 5월 12일 PM 11:13 UTCmonitoring

   We're monitoring an issue with our recording and streaming servers.

1. 2025년 3월 5일 PM 5:18 UTCidentified

   The team is actively investigating an issue affecting our service. We'll update you as soon as we can.

2. 2025년 3월 5일 PM 5:25 UTCidentified

   We are currently experiencing an outage due to a third-party service issue. This is impacting the ability to go live in the browser and via RTMP. We recommend notifying attendees from within the event and considering rescheduling if needed. Our team is actively monitoring the situation. Please check the [status page](#) for updates. Thank you for your patience.

3. 2025년 3월 5일 PM 6:15 UTCmonitoring

   We believe the incident is resolved and we are monitoring.

4. 2025년 3월 5일 PM 6:18 UTCresolved

   The issue has been resolved. We will continue to monitor