12 Tuple incidents · Gwengolo 2020 — official updates, affected components, duration and resolution details.
We're seeing a high P95 response times for requests to our contact endpoints. We are investigating the issue and will update as we know more.
Response times are back to normal. We are continuing to monitor performance.
TURN connections are currently impacted by Cloudflare's outage: https://www.cloudflarestatus.com/incidents/25r9t0vz99rp We are investigating temporary mitigations and monitoring the Cloudflare incident status.
Cloudflare is continuing to update their outage status: https://www.cloudflarestatus.com/incidents/25r9t0vz99rp
Cloudflare is still experiencing an outage, but we have mitigated the issue on our end so that there is no user impact while they resolve. Please contact support if you experience issues connecting to calls.
We are currently investigating degraded performance across the web dashboard and native applications due to an underlying issue with our database infrastructure.
We've implemented a fix and are monitoring the results.
This incident has been resolved. Please get in touch with [email protected] if you experience any issues.
We're seeing issues with our latest release, deployed today, that's causing a subset of users to experience app crashes while calling. We're reverting the release to get things back to normal; that should be ready and deployed in the next 15 minutes.
We've reverted to the previous release. If you're experiencing issues, please update your app. We'll resolve this once we're confident no further crashes are occurring.
Crash reports have ceased. We're considering this incident fully resolved!
We are seeing failed requests to our call URLs endpoint for specific teams. Not all users are affected, and we're currently investigating the issue to determine the root cause.
We've identified a performance issue with our Call URLs API route that is affecting a small number of teams. We've put a temporary fix in place for those teams that will restore application access with some diminished functionality around Call URLs, and the team is actively working on a permanent fix. The majority of Tuple users remain unaffected.
A fix has been implemented for affected teams, and response times for the affected API route are back to normal.
All client response times are back to normal and we're considering this incident resolved. Please get in touch with [email protected] if you experience any issues.
We're experiencing a large number of requests through our presence service that are overwhelming our web backend.
Presence service traffic was scaled to zero to allow the backend to stabilize. Things appear to be operational, but we're actively monitoring the situation to ensure the incident is fully resolved.
Services are operational, and we're continuing to monitor our web application metrics before considering this fully resolved.
This incident has been resolved.
We are currently investigating an issue with clients being unable to connect after a planned production deployment. Ongoing calls appear to be unaffected.
System health is recovering, and some users are successfully reconnecting. We're increasing infrastructure capacity slowly to accommodate the backlog of reconnecting clients.
We're fully back online and client connections are succeeding.
We've identified an issue with our signaling server that is preventing the team directory from loading. We're working to resolve the issue now.
We're continuing to work on a fix for this issue.
We've attempted to roll back our signaling service to a previous release, but are still experiencing issues with our load balancer correctly negotiating connections. Our team is still actively working to resolve this issue.
Tuple contact directories are loading. We're actively monitoring the service while clients reconnect to ensure that the issue is resolved.
We've seen some instances of stale presence data persisting in the directory and are monitoring the relevant queue as it is processed.
Directory status for all users is up-to-date, and the signaling server has continued to process requests successfully.
Our Email provider is experiencing an outage. Application emails are delayed.
This incident has been resolved.
We are currently down due to downtime at Heroku, an underlying provider of ours. You will likely be unable to sign in or make calls at this time. Their incident response can be tracked here: https://status.heroku.com/incidents/2402 (Unfortunately, their status page appears to be loading only intermittently.)
Service appears to have been restored.
We're aware that users are getting stuck in a login loop/crash after this morning's update. We're investigating.
We believe we have identified the issue. We're building a new release and are deploying it imminently.
We are continuing to work on a fix for this issue.
We've released a fix. Please restart Tuple to force an update to the latest version. This should resolve the issue.
This issue has been resolved. Sorry about that folks!
Some users are reporting that they cannot log in to the Tuple client.
This incident has been resolved.
A weekend deploy temporarily prevented unpaid guest users \(and users on a small number of older teams\) from signing in. We investigated Monday morning after reports started to roll in and resolved the issue approximately 20 minutes later. If you are still having trouble signing in, please contact support: [email protected].