Störungsverlauf von Daily

We're seeing a recurrence of the problems with RegistryDNS.co and AT&T DNS from Monday. Users trying to join calls from affected regions using ISP DNS may get "Unable to join call" errors. We know specifically that AT&T DNS servers 68.94.157.1 and 68.94.156.1 are affected. We've been working hard on solving this problem since the incident on Monday. We're releasing daily-js 0.91.0 shortly to address this.

We expect to have daily-js 0.91.0 available within the next 60 to 90 minutes. The update includes automated failover to `dailywebrtc.com` or `dailywebrtc.net` if `daily.co` is unreachable. If you haven't updated daily-js in a while, you can do preliminary testing with 0.90.0 in order to be ready to release an update with 0.91 as soon as it's ready. 0.91 will only contain this feature, as well as a few dependency version updates for security. If you have customers with restrictive networks, they may need to add `*.dailywebrtc.com` and `*.dailywebrtc.net` to their firewall/security allowlists. These domains are relatively new, so they may be flagged by Cisco app firewalls and similar security appliances.

daily-js 0.91.0 is now available on npm and GitHub. If your app uses call object mode, you can update your dependency to that version and redeploy, and this issue should be resolved for you. We're publishing an update to our networking guide soon, but in the meantime, anything that mentions a `.daily.co` hostname in that doc is also available at `.dailywebrtc.com` and `.dailywebrtc.net`. We'll have updates for embedded Prebuilt and direct link customers soon. We'll leave this incident open while RegistryDNS.co remains offline.

The RegistryDNS.co servers continue to be intermittently unavailable, but AT&T's DNS servers seem to be more responsive. We encourage you to update to daily-js 0.91.0 as soon as possible to avoid future issues for your users.

Users that are using their ISP's DNS service (specifically AT&T users in the southeast US and/or Texas) are experiencing intermittent problems accessing any .co domain, including daily.co. This may result in those users seeing an “Unable to join call” error in the browser when trying to join a call. This is being caused by failures from the DNS servers that serve the .co TLD itself. Several third-party DNS providers like Google, Cloudflare, and Quad9 are working around this in order to continue to make .co domains available. To work around this, you can encourage your users to use a third-party DNS provider, like Cloudflare or Quad9.

We are working on implementing a fallback to a .com domain so that users on ISPs that are not fixing this will not see any issues starting calls.

CentralNIC confirmed that there was indeed an issue with the .co nameservers, and they resolved it at approximately 06:30 UTC today. Our monitoring of both their DNS servers and downstream providers confirmed this. We're continuing to work urgently on updating our infrastructure to make our services available on alternate hostnames. We'll post updates to our Networking Guide when we do: https://docs.daily.co/guides/privacy-and-security/corporate-firewalls-nats-allowed-ip-list We'll also provide a root cause analysis for this incident in the next few days.

We're receiving reports of some users having problems connecting to Daily calls. It seems to be localized to the Texas area. We're investigating.

We've identified an issue that's causing some users to fail to join calls. The daily-js library has to download a bundle of additional JavaScript as part of joining a call. This bundle is downloaded from c.daily.co, which is using Amazon's CloudFront CDN. They aren't reporting issues yet, but we're engaging with their support to figure out why this is happening.

We're receiving reports of a few other regions experiencing similar connection issues. If you're monitoring client errors and you see messages that start with "Failed to load call object bundle https://c.daily.co/....", you're being affected by this. We're working with AWS to get to the bottom of this.

We've confirmed through several different end users that this is a DNS issue. Affected users from multiple regions have been able to join calls by pointing DNS to Google or CloudFlare, using IPs like 1.1.1.1 or 8.8.8.8 for DNS. Obviously, this solution doesn't scale; we're working with AWS to identify the root cause of the DNS resolution issue for c.daily.co.

Cloudflare has posted an incident concerning intermittent DNS failures for .co domains: https://www.cloudflarestatus.com/incidents/z3b5zxjtp6g1 This aligns with the troubleshooting we've done so far. Some internet discussions suggest that it can be somewhat ISP-dependent, but this is unconfirmed. If this is indeed related to the .co domain itself, it would mean that it affects participants' ability to join calls, but it could also affect the ability to make API requests to api.daily.co. Updates to follow.

The .co registry appears to be experiencing issues. Cloudflare's recursive DNS is affected (and they've acknowledged it). Many regional ISPs, such as AT&T in the southeast US, rely on Cloudflare's DNS to power their own DNS. Unlike Cloudflare, other DNS providers, like Google and Quad9, are serving .co records from stale cache. If they stop doing this, then anyone using those DNS services will start to experience these same failures. This won't be fully resolved until the .co registry comes back online. In the meantime, different DNS providers may see problems come and go. All of Daily's services remain online and functional, so if your users can resolve .co hostnames, they can use our services without problems.

Cloudflare has posted that they've implemented a fix, and they are monitoring the results. We're still not certain that Cloudflare is the actual root cause of 100% of our affected users, but we'll be monitoring error rates.

Cloudflare has resolved their status incident, but we're unsure if that means the underlying issue is resolved. We're continuing to monitor our own error rates and health checks.

The rate of bundle download failures has decreased, but it's also tracking our overall usage volume through the day. The number of affected users appears to be very small, but we still have tests that can replicate the failure. Unfortunately this is completely out of our control. If you have users that are continuing to be affected, you can suggest that they use Google's or Quad9's DNS servers at 1.1.1.1 or 9.9.9.9.

Our tests just confirmed that DNS resolution is no longer returning errors for .co domains in our tests. This issue has been resolved!

This incident has been resolved.

We're seeing an elevated rate of errors for API requests and room joins. We're addressing it right now.

There was a sudden increase in activity across several databases. We've addressed the cause of the issue, and platform metrics are returning to normal. We're continuing to monitor for any further issues.

This incident has been resolved.

We're investigating elevated error rates for incoming PSTN/SIP calls.

SignalWire has identified an issue causing problems with incoming and outgoing PSTN/SIP calls for Daily customers. If you're making dialout calls, you may see an error that says "DialOut stopped: Remote busy or Remote did not answer or Remote ended the call". If you're using dialin, you may hear a message that says "the number you have dialed is not configured correctly and cannot receive calls" or something similar.

It seems that SignalWire has resolved their issue. We're monitoring for any other problems.

This incident has been resolved. We're still collecting more information on the root cause here; we'll update this status post when we have more info.

We're investigating an issue that may be causing delayed delivery for some webhooks.

We've identified an issue with the webhook system that was causing brief delays when finding meeting start and end events for webhooks. We've addressed that issue. Webhook delivery time was only minimally impacted during the event, and it has returned to normal as of about 15 minutes ago.

This incident has been resolved.

We are currently investigating an issue affecting SIP / PSTN inbound calls. Customers may encounter 480 “Temporarily Unavailable” errors when attempting to receive calls. Our engineering team is actively working to identify the root cause and restore service.

Our upstream carrier partner has identified the root cause of the issue impacting SIP/PSTN dial-in (inbound) calling and has implemented a fix. Service is beginning to recover, though some degradation may still be present while the fix fully propagates. We are actively monitoring and working to confirm full restoration of inbound calling. Outbound (dial-out) calling remains unaffected.

The fix from our upstream carrier partner has been applied, and service has been restored for SIP/PSTN dial-in (inbound) calling. We are seeing successful inbound call connections again, and our internal testing confirms recovery. We are continuing to monitor closely to ensure stability and full recovery.

This incident has been resolved.

We've seen an increase in "Failed to load call object bundle" errors within the past hour. This happens when the browser is unable to download our JavaScript bundle from the CDN. If this happens to you or your users, you should be able to refresh the page and join your call successfully. We're working with our CDN provider to identify the issue.

The initial increase in error rates has subsided. Based on reports from a few customers, it seems like the impact was primarily in the Texas area. We're continuing to work with AWS to determine the root cause.

The issue has been resolved. We’ve seen the number of “Failed to load call object bundle” errors return to normal levels, and call joins are operating as expected.

We're tracking a Deepgram incident that is causing problems with transcription events, as well as captions in Daily Prebuilt: https://status.deepgram.com/incidents/rp0jzltym9kw

We're continuing to see very brief occurrences of slow or missing transcription data from the Deepgram streaming endpoint. We're talking to Deepgram engineers and continuing to monitor the situation.

We're seeing extremely low error rates, but Deepgram hasn't resolved their incident yet, so we'll keep monitoring.

This incident has been resolved.

We are currently experiencing an abnormally large queue of webhooks awaiting delivery. Our team is investigating and working to restore normal processing.

We’ve identified the cause of the issue. While a backlog of webhooks remains, we’re actively working to reduce it. Webhook deliveries will continue to be delayed for some time, but no data is being lost.

We're continuing to work through a backlog of webhook events, and the delay is starting to decrease. It will still take some time to get fully caught up.

We've worked through the webhook backlog, and delivery times should be normal. We'll be monitoring for any further issues.

This incident has been resolved.

We're seeing an increased rate of errors when dialing out SIP/PSTN calls.

We've rolled back the release that was in process when we started seeing the errors, which seems to have resolved the issue. We'll be monitoring for any regressions.

This incident has been resolved.

We're seeing an abnormally large queue of webhooks to be delivered, so we've scaled up to work through the backlog. We'll post another update when delivery times return to normal.

We're continuing to work through the backlog of webhooks.

Webhook delivery times have returned to normal.

SignalWire experienced an outage earlier this morning. Their status site post is here: https://status.signalwire.com/incidents/MJVYwKKYYwbG Our first customer report of problems was at 1:51 AM PST (09:51 UTC). During the incident, many SignalWire SIP calls were returning 603 Declined. As we were investigating, SignalWire opened an incident on their status site at 2:26 AM PST (10:26 UTC). Their incident post didn't provide enough information for us to determine the overall impact to our customers. As we continued investigating our own logs and pressing SignalWire for more information, our affected customers started reporting that the issue was resolved around 2:35 AM PST. SignalWire ultimately resolved their incident at 03:27 PST.

AWS is addressing an issue with DynamoDB that is causing problems with many of their services in us-east-1. We have seen issues loading the dashboard and docs sites, and for a few hours we've been unable to log in to Statuspage to update this site. We're sorry about that delay. Up until a few minutes ago, our metrics suggested that core call functionality wasn't being impacted. We just saw a small number of call failures in us-east-1, though. We're continuing to monitor the situation and do what we can to minimize this outage's impact on you and your customers. We'll post more information here as we have it.

AWS is continuing to address issues in us-east-1. We're still seeing normal activity levels across our APIs and call servers for this time of day, so we have no reason to believe that any your customers' calls are actually being affected. Part of AWS's mitigation efforts involve limiting the rate at which we can start new call servers. Out of an abundance of caution, we're temporarily removing us-east-1 from our list of available regions for new call sessions. This means your users will likely connect to a call through us-west-2 instead. This failover happens automatically and transparently, and you don't need to do anything. We've already scaled up call servers in us-west-2 to handle the increase in sessions. We'll keep you posted as we continue to monitor the situation.

AWS has resolved their incident. We're continuing to run the same call servers in us-east-1 as we were before the incident started, but we're still routing the majority of US traffic through us-west-2. We plan to return things to normal tomorrow.

This incident has been resolved.

We've temporarily disabled displaying the API and webhook log views on the Dashboard Developer page. The Daily API is still fully operational, and these logs are still being collected. We expect to re-enable these log views in the next few hours. If you have an urgent need for this data, please contact [email protected].

This incident has been resolved.

We're experiencing delays in making log data available via the /logs REST API endpoint. We're not losing any log data, but the process that ingests logs for the REST API is bottlenecked. This means that the most recent data available from the /logs endpoint is about 6 hours old. We'll update this incident once the situation has been resolved.

This incident has been resolved.

We're investigating reports of SIP/PSTN dialin and dialout failing, or stopping immediately after starting.

We're in the process of rolling back a release to address this issue.

We've rolled back a server release, and the SIP error messages have stopped. We're monitoring for any further problems.

This incident has been resolved.

We're seeing alerts that webhooks are taking longer than normal to deliver.

We're aware of an infrastructure issue that may be causing some intermittent failures for Daily API requests.

We're continuing to investigate the root cause, but we've resolved all impact to our running services.

Daily is being affected by widespread internet outages in authentication services. This is currently affecting our ability to collect or view some call metrics. We'll post more as soon as we have more information.

This appears to be related to the Google Cloud outage posted here: https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1SsW We'll post here as we learn more.

We're seeing some increases in response time from our internal API endpoint that returns connection information to daily-js. This may be causing "room lookup timed out" errors for users attempting to join calls. We believe this is related to the ongoing GCP incident.

We had a brief spike in response time, but platform operations appear normal since approximately 20:05 UTC. We're continuing to monitor for any follow-on effects, and we'll leave this status post open until various upstream providers close their respective status incidents.

This incident has been resolved.

We're investigating elevated error rates from SignalWire when provisioning SIP/PSTN resources. If you're creating rooms with dial-in or dial-out enabled and it isn't absolutely necessary, you can remove those params from your room creation request to successfully create rooms.

SignalWire has confirmed that they made an unannounced change that's causing the problem, and they're working to revert it. We're also working on a quick production update to work around the issue if they can't revert quickly enough.

It appears that SignalWire rolled back the change, and the errors have stopped. We are monitoring for any further issues.

This incident has been resolved.