26 Crowdcast incidents · wrzesień 2024 — official updates, affected components, duration and resolution details.
We’re currently investigating an issue that is preventing users from watching events from the Crowdcast mobile app. Our team is actively working to resolve it as quickly as possible. **In the meantime, you can continue watching events and replays from a mobile or desktop browser.** We’ll share another update as soon as we have more information. Thank you for your patience.
Crowdcast doświadczył problemu, w którym gospodarze pojawili się na żywo i nagrywali, ale uczestnicy nie mogli oglądać transmisji. Inżynierowie szybko zidentyfikowali przyczynę jako przeciążoną bazę danych. Aby przywrócić usługę, przełączyli konfiguracje bazy danych, co spowodowało krótką (~ 5 minut) przerwę w miejscu przed resetem zdarzenia i odzyskaniem usługi. Ulepszamy zaopatrzenie infrastruktury w duże wydarzenia, aby zapobiec podobnym problemom w przyszłości.
Automatyczne tłumaczenie oficjalnej aktualizacji incydentu.
The team is actively investigating an issue that is affecting some hosts when broadcasting live. Attendees may have to refresh to view the live feed. We'll update you as soon as we can.
We identified and resolved an issue that prevented events from loading correctly on the initial page load. As a result, some users needed to refresh the page before the livestream would appear. This has now been fixed, and events should load properly the first time.
The team is actively investigating an issue affecting our service that is causing increased buffering in live streams. This does not affect recordings, and we’ll provide an update as soon as possible.
We’ve deployed an update that mitigates the issue and are continuing to work on resolving the underlying cause of increased buffering in some sessions.
We’ve confirmed the issue as originating with our CDN provider, Cloudflare, and have implemented a temporary mitigation. We’re actively working with the provider to resolve the underlying cause and will share another update as soon as more information is available.
Our updates remain stable, and we are continuing to implement additional improvements to help ensure that issues at Cloudflare have a reduced impact on our services.
We experienced an issue that temporarily prevented some sessions from going live due to a brief compute disruption within our recording infrastructure. The problem was limited in duration, and we are actively implementing a patch to address the underlying cause and help ensure it does not happen again.
The team is actively investigating an issue affecting our service. We'll update you as soon as we can.
We’re currently experiencing a capacity shortage with our cloud provider (Hetzner). Hetzner has run out of available capacity in the region(s) we use, which is limiting our ability to provision additional resources and may impact scaling and recovery actions. • **Current status:** Investigating / Mitigating • **Impact:** Potential service degradation due to constrained scaling capacity (e.g., slower provisioning, delayed failover, or limited ability to add nodes). • **What we’re doing:** Actively working to secure additional capacity, including requesting more allocation from Hetzner and evaluating alternative capacity options. • **Next update:** We’ll share another update as soon as additional capacity is confirmed or our mitigation plan changes.
We’ve identified an issue affecting our services where some hosts may experience difficulty going live. Our team is actively investigating, and we’ll share an update as soon as more information is available.
** Identified** We’re currently experiencing a capacity shortage with our cloud provider (Hetzner). Hetzner has run out of available capacity in the region(s) we use, which is limiting our ability to provision additional resources and may impact scaling and recovery actions. • **Current status:** Investigating / Mitigating • **Impact:** Potential service degradation due to constrained scaling capacity (e.g., slower provisioning, delayed failover, or limited ability to add nodes). • **What we’re doing:** Actively working to secure additional capacity, including requesting more allocation from Hetzner and evaluating alternative capacity options. • **Next update:** We’ll share another update as soon as additional capacity is confirmed or our mitigation plan changes.
We’ve secured additional compute capacity, and the platform is currently stable. We’re continuing to monitor key signals (capacity headroom, provisioning queues, and error rates) to ensure stability remains sustained. • **Current status:** Monitoring • **Impact:** No current impact; capacity constraints have been mitigated for now. • **What we’re doing:** Ongoing monitoring and readiness to scale further if demand or capacity conditions change. • **Next update:** We’ll provide another update if conditions change or once we confirm sustained stability.
We’ve confirmed the scaled-up resources are holding under current load, and service remains stable. The immediate capacity constraints are **resolved for now**. • **Current status:** Resolved • **Impact:** No current impact; services are operating normally. • **What we’re doing next:** We’re investigating and designing a multi-cloud capacity strategy to reduce dependence on a single provider’s regional availability and enable provisioning across multiple cloud providers concurrently.
At **8:54 AM ET**, we experienced a **9-minute service disruption** in our livestreaming infrastructure that impacted **real-time audience playback**. During this window, viewers may have seen interruptions or inability to watch live content. • **Start time:** 8:54 AM ET • **Duration:** 9 minutes • **Impacted:** Live audience streaming sessions • **Not impacted:** Recording capture and stored playback assets The incident was caused by a previously unobserved failure mode. Mitigation has been applied, and we are implementing additional patches and safeguards to prevent recurrence.
The team is actively investigating an issue affecting our service. We'll update you as soon as we can.
We've scaled our systems to handle significantly higher loads from multiple simultaneous large-scale events. Our systems are currently stable, and we continue to monitor them closely.
**All systems remain stable.** We have identified the root cause of the cascading issues that occurred under high load and are deploying an update today to prevent this from happening again. Thank you for your continued patience.
The team is actively investigating an issue affecting our service. We'll update you as soon as we can.
We’re hitting scale limits on our real-time chat and interaction services due to a large event. We’re increasing capacity now and will restore full service as quickly as possible.
Services are coming back online, and we’re monitoring performance.
**Impact:** Some real-time updates were delayed for a portion of users. No data was lost. **Timeline:** Started at 7pm, peaked at 7:10pm, and resolved by 9:30pm ET. **Cause:** A sudden traffic spike from a high-scale event pushed past our autoscaling limits. **What we did:** Added capacity and tuned autoscaling. **Now:** Everything is stable and performing normally. **What’s next:** Stronger scaling safeguards, smarter rate limiting, and improved monitoring.
We’re actively investigating a service disruption caused by our third‑party provider, Cloudflare. This issue is affecting us and many other services, including Zoom, Uber, Twitter, OpenAI, and AWS. We’ll share updates as soon as we have more information.
Cloudflare has acknowledged a global outage and you can follow updates [here](https://www.cloudflarestatus.com/incidents/8gmgl950y3h7). We'll continue to update as we have more information.
Cloudflare’s ongoing issue is still impacting Crowdcast. Access from some regions is improving, and we’re continuing to monitor and will share updates as we learn more.
Things look stable now—Crowdcast is back to normal. We’ll keep monitoring to make sure we’re fully in the clear.
A [major outage](https://health.aws.amazon.com/health/status) at Amazon Web Services (AWS) has taken down our [upstream hosting](https://www.vercel-status.com/) provider as well as large portions of the internet. We're monitoring the situation and will update you as soon as we can.
Our systems are back online. We're continuing to monitor the situation.
All systems are back to normal.
[AWS is having a massive outage](https://health.aws.amazon.com/health/status) again that's affecting us and [many other platforms](https://downdetector.com/). We'll update you as soon as we can.
Our systems are back to normal. We're continuing to monitor the situation at AWS for any other updates.
The web API is timing out and the team is investigating.
All systems are back to normal. During a busy period on Crowdcast, part of our system that checks if everything is healthy started timing out because of the extra traffic. Those timeouts made our infrastructure think something was wrong, so it restarted parts of the system. These restarts caused brief disruptions that affected access to event data and led to slower or interrupted service for some live events. We’ve adjusted how the health checks work so they’re more tolerant during heavy traffic and have added safeguards to keep the system stable.
The team is actively investigating an issue preventing new events from going live. We'll update you as soon as we can.
The issue has been resolved. The issue was caused by a change from one of our external providers, which unexpectedly disrupted part of our system. While the cause was outside of Crowdcast, our team resolved it quickly and has already put new safeguards in place to prevent this type of issue from happening again.
We’re experiencing an overall slowdown in the app’s performance. We suspect this may be related to an issue with [Cloudflare’s DNS services](https://www.cloudflarestatus.com/). Our team is actively investigating and will provide updates as soon as possible.
We've confirmed that the issues were related to an outage at Cloudflare which has now been resolved, and the app is now operating at normal levels.
The team is actively investigating an issue affecting our service. We'll update you as soon as we can.
We've identified that the issue resides with our cloud provider (Hetzner) not having enough compute resources. We're monitoring the situation closely and working on spinning up our servers on other cloud providers. You can monitor one of the issues here: https://status.hetzner.com/incident/8b1ceeb2-9daa-4971-a1a1-5346fb8fc45a
We were able to spin up additional resources so some sessions are starting to be able to go live. The core issue with Hetzner remains so we'll update you as soon as it's fully resolved. We recommend running a test event before your scheduled event.
Sessions are back online, and we’re actively adding support for more cloud providers to prevent this kind of issue in the future.
The team is actively investigating an issue affecting our service. We'll update you as soon as we can.
All systems are back to normal. We’re still digging into what happened to make sure it doesn’t happen again.
Our backend services recently encountered rate limits due to an unexpected policy change by a vendor. The issue has been resolved, and we’ve implemented safeguards to prevent similar disruptions in the future. All systems are now fully operational.
The team is actively investigating an issue affecting our service. We'll update you as soon as we can.
We have identified that the issue is with our cloud provider, Vultr, specifically at one of their U.S. data centers that we utilize. We are collaborating with their team to resolve the problem and restore service. We will provide updates as soon as possible.
The issue with our cloud provider, Vultr, specifically at one of their U.S. data centers that we utilize, has been resolved. We collaborated with their team to resolve the problem and restore service.
Recording and streaming servers are back online. The issue was caused by an unexpected outage with our downstream cloud provider. We’re actively working on transitioning to a more reliable provider to prevent future disruptions.
We're monitoring an issue with our recording and streaming servers.
The team is actively investigating an issue affecting our service. We'll update you as soon as we can.
We are currently experiencing an outage due to a third-party service issue. This is impacting the ability to go live in the browser and via RTMP. We recommend notifying attendees from within the event and considering rescheduling if needed. Our team is actively monitoring the situation. Please check the [status page](#) for updates. Thank you for your patience.
We believe the incident is resolved and we are monitoring.
The issue has been resolved. We will continue to monitor