История сбоев Crowdcast

1. 13 августа 2026 г. в 01:04 UTCinvestigating

   В настоящее время мы расследуем проблему, которая мешает пользователям смотреть события из мобильного приложения Crowdcast. Наша команда активно работает над тем, чтобы решить ее как можно быстрее. ** В то же время вы можете продолжать смотреть события и воспроизводить их с мобильного или настольного браузера.** Мы поделимся еще одним обновлением, как только у нас будет больше информации. Спасибо за ваше терпение.

2. 13 августа 2026 г. в 15:29 UTCresolved

   Мы определили проблему и внедрили исправление. Функционал событий в мобильном приложении снова работает. Спасибо за ваше терпение и понимание.

Автоматический перевод официального обновления инцидента.

1. 3 августа 2026 г. в 18:54 UTCresolved

   Crowdcast столкнулся с проблемой, когда ведущие появлялись в прямом эфире и записывались, но участники не могли просматривать трансляцию. Инженеры быстро определили причину как перегруженную базу данных. Чтобы восстановить обслуживание, они переключали конфигурации базы данных, что привело к короткому (~ 5 минут) прерыванию сайта до сброса и восстановления обслуживания. Мы улучшаем обеспечение инфраструктуры для крупных мероприятий, чтобы предотвратить подобные проблемы в будущем.

Автоматический перевод официального обновления инцидента.

1. 6 мая 2026 г. в 12:05 UTCinvestigating

   The team is actively investigating an issue that is affecting some hosts when broadcasting live. Attendees may have to refresh to view the live feed. We'll update you as soon as we can.

2. 6 мая 2026 г. в 13:40 UTCresolved

   We identified and resolved an issue that prevented events from loading correctly on the initial page load. As a result, some users needed to refresh the page before the livestream would appear. This has now been fixed, and events should load properly the first time.

1. 29 апреля 2026 г. в 16:17 UTCinvestigating

   The team is actively investigating an issue affecting our service that is causing increased buffering in live streams. This does not affect recordings, and we’ll provide an update as soon as possible.

2. 29 апреля 2026 г. в 17:14 UTCidentified

   We’ve deployed an update that mitigates the issue and are continuing to work on resolving the underlying cause of increased buffering in some sessions.

3. 29 апреля 2026 г. в 21:11 UTCmonitoring

   We’ve confirmed the issue as originating with our CDN provider, Cloudflare, and have implemented a temporary mitigation. We’re actively working with the provider to resolve the underlying cause and will share another update as soon as more information is available.

4. 30 апреля 2026 г. в 16:27 UTCresolved

   Our updates remain stable, and we are continuing to implement additional improvements to help ensure that issues at Cloudflare have a reduced impact on our services.

1. 23 апреля 2026 г. в 14:58 UTCresolved

   We experienced an issue that temporarily prevented some sessions from going live due to a brief compute disruption within our recording infrastructure. The problem was limited in duration, and we are actively implementing a patch to address the underlying cause and help ensure it does not happen again.

1. 17 марта 2026 г. в 17:17 UTCinvestigating

   The team is actively investigating an issue affecting our service. We'll update you as soon as we can.

2. 17 марта 2026 г. в 17:33 UTCidentified

   We’re currently experiencing a capacity shortage with our cloud provider (Hetzner). Hetzner has run out of available capacity in the region(s) we use, which is limiting our ability to provision additional resources and may impact scaling and recovery actions. • **Current status:** Investigating / Mitigating • **Impact:** Potential service degradation due to constrained scaling capacity (e.g., slower provisioning, delayed failover, or limited ability to add nodes). • **What we’re doing:** Actively working to secure additional capacity, including requesting more allocation from Hetzner and evaluating alternative capacity options. • **Next update:** We’ll share another update as soon as additional capacity is confirmed or our mitigation plan changes.

3. 17 марта 2026 г. в 17:51 UTCidentified

   We’ve identified an issue affecting our services where some hosts may experience difficulty going live. Our team is actively investigating, and we’ll share an update as soon as more information is available.

4. 17 марта 2026 г. в 17:52 UTCidentified

   ** Identified** We’re currently experiencing a capacity shortage with our cloud provider (Hetzner). Hetzner has run out of available capacity in the region(s) we use, which is limiting our ability to provision additional resources and may impact scaling and recovery actions. • **Current status:** Investigating / Mitigating • **Impact:** Potential service degradation due to constrained scaling capacity (e.g., slower provisioning, delayed failover, or limited ability to add nodes). • **What we’re doing:** Actively working to secure additional capacity, including requesting more allocation from Hetzner and evaluating alternative capacity options. • **Next update:** We’ll share another update as soon as additional capacity is confirmed or our mitigation plan changes.

5. 17 марта 2026 г. в 19:39 UTCmonitoring

   We’ve secured additional compute capacity, and the platform is currently stable. We’re continuing to monitor key signals (capacity headroom, provisioning queues, and error rates) to ensure stability remains sustained. • **Current status:** Monitoring • **Impact:** No current impact; capacity constraints have been mitigated for now. • **What we’re doing:** Ongoing monitoring and readiness to scale further if demand or capacity conditions change. • **Next update:** We’ll provide another update if conditions change or once we confirm sustained stability.

6. 17 марта 2026 г. в 19:55 UTCresolved

   We’ve confirmed the scaled-up resources are holding under current load, and service remains stable. The immediate capacity constraints are **resolved for now**. • **Current status:** Resolved • **Impact:** No current impact; services are operating normally. • **What we’re doing next:** We’re investigating and designing a multi-cloud capacity strategy to reduce dependence on a single provider’s regional availability and enable provisioning across multiple cloud providers concurrently.

1. 11 марта 2026 г. в 14:21 UTCresolved

   At **8:54 AM ET**, we experienced a **9-minute service disruption** in our livestreaming infrastructure that impacted **real-time audience playback**. During this window, viewers may have seen interruptions or inability to watch live content. • **Start time:** 8:54 AM ET • **Duration:** 9 minutes • **Impacted:** Live audience streaming sessions • **Not impacted:** Recording capture and stored playback assets The incident was caused by a previously unobserved failure mode. Mitigation has been applied, and we are implementing additional patches and safeguards to prevent recurrence.

1. 5 марта 2026 г. в 10:28 UTCinvestigating

   The team is actively investigating an issue affecting our service. We'll update you as soon as we can.

2. 5 марта 2026 г. в 12:27 UTCmonitoring

   We've scaled our systems to handle significantly higher loads from multiple simultaneous large-scale events. Our systems are currently stable, and we continue to monitor them closely.

3. 5 марта 2026 г. в 14:32 UTCresolved

   **All systems remain stable.** We have identified the root cause of the cascading issues that occurred under high load and are deploying an update today to prevent this from happening again. Thank you for your continued patience.

1. 13 февраля 2026 г. в 01:27 UTCinvestigating

   The team is actively investigating an issue affecting our service. We'll update you as soon as we can.

2. 13 февраля 2026 г. в 01:51 UTCidentified

   We’re hitting scale limits on our real-time chat and interaction services due to a large event. We’re increasing capacity now and will restore full service as quickly as possible.

3. 13 февраля 2026 г. в 02:53 UTCmonitoring

   Services are coming back online, and we’re monitoring performance.

4. 13 февраля 2026 г. в 03:10 UTCresolved

   **Impact:** Some real-time updates were delayed for a portion of users. No data was lost. **Timeline:** Started at 7pm, peaked at 7:10pm, and resolved by 9:30pm ET. **Cause:** A sudden traffic spike from a high-scale event pushed past our autoscaling limits. **What we did:** Added capacity and tuned autoscaling. **Now:** Everything is stable and performing normally. **What’s next:** Stronger scaling safeguards, smarter rate limiting, and improved monitoring.

1. 18 ноября 2025 г. в 11:47 UTCidentified

   We’re actively investigating a service disruption caused by our third‑party provider, Cloudflare. This issue is affecting us and many other services, including Zoom, Uber, Twitter, OpenAI, and AWS. We’ll share updates as soon as we have more information.

2. 18 ноября 2025 г. в 12:09 UTCmonitoring

   Cloudflare has acknowledged a global outage and you can follow updates [here](https://www.cloudflarestatus.com/incidents/8gmgl950y3h7). We'll continue to update as we have more information.

3. 18 ноября 2025 г. в 12:28 UTCmonitoring

   Cloudflare’s ongoing issue is still impacting Crowdcast. Access from some regions is improving, and we’re continuing to monitor and will share updates as we learn more.

4. 18 ноября 2025 г. в 13:45 UTCresolved

   Things look stable now—Crowdcast is back to normal. We’ll keep monitoring to make sure we’re fully in the clear.

1. 20 октября 2025 г. в 08:54 UTCidentified

   A [major outage](https://health.aws.amazon.com/health/status) at Amazon Web Services (AWS) has taken down our [upstream hosting](https://www.vercel-status.com/) provider as well as large portions of the internet. We're monitoring the situation and will update you as soon as we can.

2. 20 октября 2025 г. в 09:23 UTCmonitoring

   Our systems are back online. We're continuing to monitor the situation.

3. 20 октября 2025 г. в 09:47 UTCresolved

   All systems are back to normal.

4. 20 октября 2025 г. в 16:40 UTCidentified

   [AWS is having a massive outage](https://health.aws.amazon.com/health/status) again that's affecting us and [many other platforms](https://downdetector.com/). We'll update you as soon as we can.

5. 20 октября 2025 г. в 19:53 UTCresolved

   Our systems are back to normal. We're continuing to monitor the situation at AWS for any other updates.

1. 29 сентября 2025 г. в 23:19 UTCresolved

   All systems are back to normal. During a busy period on Crowdcast, part of our system that checks if everything is healthy started timing out because of the extra traffic. Those timeouts made our infrastructure think something was wrong, so it restarted parts of the system. These restarts caused brief disruptions that affected access to event data and led to slower or interrupted service for some live events. We’ve adjusted how the health checks work so they’re more tolerant during heavy traffic and have added safeguards to keep the system stable.

2. 29 сентября 2025 г. в 23:19 UTCinvestigating

   The web API is timing out and the team is investigating.

1. 17 сентября 2025 г. в 17:18 UTCinvestigating

   The team is actively investigating an issue preventing new events from going live. We'll update you as soon as we can.

2. 17 сентября 2025 г. в 18:27 UTCresolved

   The issue has been resolved. The issue was caused by a change from one of our external providers, which unexpectedly disrupted part of our system. While the cause was outside of Crowdcast, our team resolved it quickly and has already put new safeguards in place to prevent this type of issue from happening again.

1. 21 августа 2025 г. в 18:57 UTCinvestigating

   We’re experiencing an overall slowdown in the app’s performance. We suspect this may be related to an issue with [Cloudflare’s DNS services](https://www.cloudflarestatus.com/). Our team is actively investigating and will provide updates as soon as possible.

2. 21 августа 2025 г. в 19:45 UTCresolved

   We've confirmed that the issues were related to an outage at Cloudflare which has now been resolved, and the app is now operating at normal levels.

1. 9 июля 2025 г. в 14:05 UTCinvestigating

   The team is actively investigating an issue affecting our service. We'll update you as soon as we can.

2. 9 июля 2025 г. в 14:57 UTCidentified

   We've identified that the issue resides with our cloud provider (Hetzner) not having enough compute resources. We're monitoring the situation closely and working on spinning up our servers on other cloud providers. You can monitor one of the issues here: https://status.hetzner.com/incident/8b1ceeb2-9daa-4971-a1a1-5346fb8fc45a

3. 9 июля 2025 г. в 15:57 UTCmonitoring

   We were able to spin up additional resources so some sessions are starting to be able to go live. The core issue with Hetzner remains so we'll update you as soon as it's fully resolved. We recommend running a test event before your scheduled event.

4. 9 июля 2025 г. в 22:51 UTCresolved

   Sessions are back online, and we’re actively adding support for more cloud providers to prevent this kind of issue in the future.

1. 6 июля 2025 г. в 23:49 UTCinvestigating

   The team is actively investigating an issue affecting our service. We'll update you as soon as we can.

2. 6 июля 2025 г. в 23:58 UTCresolved

   All systems are back to normal. We’re still digging into what happened to make sure it doesn’t happen again.

1. 12 июня 2025 г. в 13:24 UTCresolved

   Our backend services recently encountered rate limits due to an unexpected policy change by a vendor. The issue has been resolved, and we’ve implemented safeguards to prevent similar disruptions in the future. All systems are now fully operational.

1. 13 мая 2025 г. в 16:17 UTCinvestigating

   The team is actively investigating an issue affecting our service. We'll update you as soon as we can.

2. 13 мая 2025 г. в 16:31 UTCidentified

   We have identified that the issue is with our cloud provider, Vultr, specifically at one of their U.S. data centers that we utilize. We are collaborating with their team to resolve the problem and restore service. We will provide updates as soon as possible.

3. 13 мая 2025 г. в 16:58 UTCresolved

   The issue with our cloud provider, Vultr, specifically at one of their U.S. data centers that we utilize, has been resolved. We collaborated with their team to resolve the problem and restore service.

1. 12 мая 2025 г. в 23:13 UTCresolved

   Recording and streaming servers are back online. The issue was caused by an unexpected outage with our downstream cloud provider. We’re actively working on transitioning to a more reliable provider to prevent future disruptions.

2. 12 мая 2025 г. в 23:13 UTCmonitoring

   We're monitoring an issue with our recording and streaming servers.

1. 5 марта 2025 г. в 17:18 UTCidentified

   The team is actively investigating an issue affecting our service. We'll update you as soon as we can.

2. 5 марта 2025 г. в 17:25 UTCidentified

   We are currently experiencing an outage due to a third-party service issue. This is impacting the ability to go live in the browser and via RTMP. We recommend notifying attendees from within the event and considering rescheduling if needed. Our team is actively monitoring the situation. Please check the [status page](#) for updates. Thank you for your patience.

3. 5 марта 2025 г. в 18:15 UTCmonitoring

   We believe the incident is resolved and we are monitoring.

4. 5 марта 2025 г. в 18:18 UTCresolved

   The issue has been resolved. We will continue to monitor