51 Livestorm incidents · October 2022 — official updates, affected components, duration and resolution details.
We are currently investigating this issue.
The incident has been resolved. The root cause has been identified and we took measures to fix it.
We have identified an issue that prevented some attendees from completing registration on Livestorm event registration pages. During the affected period, clicking the "Receive access link" button did not complete the sign-up. Already-registered attendees and their access links were not affected, and no data was lost. We have identified the cause and applied a fix. Registration is being restored, and we are monitoring to confirm full recovery. We'll post an update shortly. We apologize for the inconvenience.
A fix has been deployed and event registration is now working normally. Attendees can complete registration and receive their access link as expected. We are monitoring closely to confirm everything remains stable. Thank you for your patience.
This incident has been resolved. Event registration is fully operational, and all attendees can register and receive their access link normally. Already-registered attendees were unaffected and no data was lost. We apologize for the disruption and appreciate your patience.
We are currently investigating an issue causing some users to encounter a 503 Service Temporarily Unavailable error when accessing the platform. Our team is actively working to identify the root cause and restore normal service as quickly as possible. We will provide further updates as soon as more information becomes available.
A change introduced during a Kubernetes cluster upgrade disrupted internal connections within our infrastructure, resulting in service unavailability. We are currently evaluating a rollback to restore connectivity and resume normal service as quickly as possible. We will provide another update as soon as remediation is underway.
The remediation has been applied and service availability has been restored. We are monitoring the platform closely to ensure that internal connectivity remains stable and that no further disruptions occur following the Kubernetes cluster upgrade issue.
This incident has been resolved. Service availability has been fully restored, and monitoring confirms that the platform is operating normally. The disruption was caused by a change introduced during a Kubernetes cluster upgrade, which affected internal cluster connectivity. We will continue reviewing the incident to prevent similar occurrences in the future. We apologize for the inconvenience.
We have identified an issue causing login attempts via Google SSO and Microsoft SSO to fail silently. Our team is actively investigating the root cause and working to mitigate the issue as quickly as possible.
We have identified the cause of the issue: recent changes made to strengthen reCAPTCHA security introduced an unintended side effect that disrupted SSO login. We are rolling back these changes and continuing to monitor the situation closely.
The rollback has been completed, and SSO login via Google and Microsoft is now functioning as expected. We are continuing to monitor the situation closely to ensure the issue is fully resolved.
The issue affecting Google and Microsoft SSO login has been resolved following the rollback. We have confirmed that SSO authentication is functioning as expected.
Our team has identified an ongoing issue where the platform is experiencing higher-than-usual loading times. At the moment, the Events list page may fail to load, and event creation may intermittently fail. Some users may also experience slow or unresponsive pages. We are actively investigating the root cause and rolling out fixes. We will share another update as soon as more information becomes available.
Our team was successfully able to mitigate the issue and the platform is now accessible as required. We're continuing to monitor the issue to ensure that users have no further impact in regards to this. We apologize for any inconvenience caused to this end.
Following our monitoring over the past few minutes, we can confirm that there is no longer any ongoing impact related to this issue. Users should now be able to access the platform and its features as expected. We sincerely apologize for any inconvenience caused and appreciate your patience while we worked to resolve this.
We have identified an issue with the event creation and registration. When attempting to create or register to an event, the operation does not complete successfully and the user sees an "Internal Server Error" We are currently investigating the issue and we will update you soon.
We have identified the root cause of this incident. An internal infrastructure component is experiencing saturation, causing event creation and registration operations to fail with an "Internal Server Error." We are actively working on mitigation and will provide updates as the situation progresses.
Event creation and registration should now be working as expected. We are monitoring the situation following the actions taken by our engineering team. Please let us know if you continue to experience any issues.
All systems remain stable, and key functionalities are operating as expected. Our teams are continuing to keep a close eye on performance and system behavior to ensure everything remains fully operational. We will share further updates if needed.
This incident has now been resolved. The root cause was identified as an internal component being overwhelmed by an unusually high volume of scheduled jobs originating from a single source. We have since removed the problematic jobs and scaled the infrastructure accordingly. All features, including event creation, registration, access to event rooms, and automated actions, are now fully operational. . Please note that during the time of the incident, some automated actions and emails may have failed or timed out due to the system overload. This could have resulted in certain emails not being sent and some automated actions not being triggered. Unfortunately, these actions cannot be retroactively reprocessed or retriggered. We sincerely apologize for the inconvenience and are taking steps to prevent this from happening again.
We have identified an issue with the event creation and registration. When attempting to create or register to an event, the operation does not complete successfully and the user sees an "Internal Server Error" We are currently investigating the issue and we will update you soon.
We have identified the root cause of this incident. An internal infrastructure component is experiencing saturation, causing event creation and registration operations to fail with an "Internal Server Error." We are actively working on mitigation and will provide updates as the situation progresses.
Event creation and registration should now be working as expected. We are monitoring the situation following the actions taken by our engineering team. Please let us know if you continue to experience any issues.
The infrastructure component has been stabilized and scaled. Event creation and registration should now be working as expected. We are continuing our investigation to determine the root cause of the saturation. We will provide a further update shortly.
This incident has been resolved. The root cause was identified as an internal component being overwhelmed by an abnormally large number of scheduled jobs originating from a single source. We have removed the offending jobs and scaled our infrastructure accordingly. All features, including event creation, registration, and automated actions, are now fully operational. We apologize for the inconvenience and will be taking steps to prevent this from happening again. Please note that during the time of the incident, some automated actions and emails may have failed or timed out due to the system overload. This could have resulted in certain emails not being sent and some automated actions not being triggered. Unfortunately, these actions cannot be retroactively reprocessed or retriggered. We sincerely apologize for the inconvenience and are taking steps to prevent this from happening again.
Our team has been alerted of an issue whereas users have been unable to join the room. We're investigating the cause and will provide an update as soon as possible. We are sorry for the inconvenience caused.
We recently deployed an update to one of our infrastructure components that caused this issue. This change has now been rolled back, users should be able to join the rooms at the moment, and we are continuing to closely monitor the situation.
The issue has now been resolved following the rollback of the recent infrastructure update. Normal behavior has been restored, and no further impact has been observed. We apologize for any inconvenience this may have caused and appreciate your patience. We will continue monitoring to ensure ongoing stability and prevent recurrence.
We are investigating an issue affecting a subset of users where the Events list fails to load. Not all users are impacted. We are actively investigating the root cause. No action is needed on your end. We will provide an update as soon as we have more information.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We are investigating an issue affecting a subset of users where the Events list fails to load. Not all users are impacted. We are actively investigating the root cause. No action is needed on your end. We will provide an update as soon as we have more information.
Resolved – Events List Degraded The issue causing the Events list to fail to load for a subset of users has been resolved. We rolled back a recent change which has restored normal service. We are continuing to investigate the root cause and will take steps to prevent recurrence. If you are still experiencing issues, please contact our support team.
We are continuing to investigate this issue.
The issue affecting the Events list for a subset of users has been resolved and the service is now operating normally. Our team will continue monitoring the situation to ensure everything remains stable.
We have identified an issue where duplicating an event fails for some users. When attempting to duplicate an existing Event, the operation does not complete successfully and the user sees an "Internal Server Error" This appears to be related to an internal URL generation error triggered during the event creation process, we have worked on a potential fix and will keep you posted, thanks!
We have identified the root cause of the issue. It was introduced by our recent Ruby upgrade which embeds a new version of OpenSSL, and is affecting event duplication in specific cases where there is a logo and/or a cover image on the registration page. We are rolling back the backend change to restore normal behavior.
The backend change that introduced the Ruby/OpenSSL upgrade has been rolled back. We are now monitoring the system to ensure stability and normal operation.
The incident has been resolved. The backend change introducing the Ruby/OpenSSL upgrade has been rolled back, and normal functionality has been restored. Monitoring confirms that the system is stable and event duplication is working as expected. We apologize for any inconvenience this may have caused.
We are currently investigating an issue preventing some users from logging into their Livestorm accounts. This issue was triggered by a recent change related to subscription handling. As a result, certain accounts are being incorrectly restricted during the authentication process.
The issue has been identified and we have rolled back the change causing this.
Login functionality has returned to normal, and affected users should now be able to access their accounts without issue. Our team is continuing to monitor authentication and subscription systems closely to ensure full stability.
The issue affecting login for some users has been fully resolved. Following the rollback of the subscription-related change, authentication flows have remained stable, and no further impact has been observed. We have completed our monitoring phase and confirmed that the platform is operating normally. As part of our follow-up, we are conducting an internal review to ensure additional safeguards are in place before reintroducing similar changes in the future. We sincerely apologize for the disruption this caused and thank you for your patience and understanding. If you experience any further issues, please contact us at [email protected].
Following the rollout of the new session roles & permissions feature, we’ve identified degraded performance across parts of the platform. Some users may experience slow loading times, delays, or intermittent errors when managing or interacting with session roles and permissions. Our engineering team is actively investigating the root cause and working on mitigations to stabilize performance.
We’ve identified that the degraded performance only occurs when managing session roles at the same time as event creation. Managing or updating session roles after the event has been created is not impacted and works as expected.
The issues affecting users’ ability to join events have been stabilized. This was caused by a temporary spike in activity impacting the Session Roles service. The performance issue related to managing roles during event creation remains under investigation, and we are currently deploying a fix to resolve it.
Managing roles during event creation has been resolved. We will continue to monitor the system to ensure complete stability.
We are continuing to monitor for any further issues.
The issue has now been fully resolved with a permanent fix. Event creation and role assignments are functioning normally. The situation is stable, and users can successfully manage session roles while participants are able to join their sessions without any issues. Thank you for your patience, and we apologize for any inconvenience this may have caused.
Livestorm is currently inaccessible. This appears to be related to an unusually high load on the platform. Our engineering team is actively investigating the situation and working to restore access as quickly as possible. We will share updates as more information becomes available. We apologize for the inconvenience and appreciate your patience.
Our team has mitigated the issue and is in the process of recovering the platform. The platform is now accessible but operating with degraded performance and limited functionality. We are continuing the recovery efforts to fully restore normal access and performance.
We are continuing to work on a fix for this issue.
The platform has now been fully re-stabilized and is accessible again. We can confirm that users are no longer experiencing issues accessing the platform, and full functionality has been restored. We apologize for the inconvenience this may have caused and thank you for your patience.
We are currently experiencing an issue in Livestorm Meeting rooms where viewers may not see other speakers on stage and may experience audio echos. Our engineering team is actively investigating the cause and working to restore full functionality. We apologize for the disruption and will provide updates as soon as we have more information.
We have rolled back the change that caused this issue. Speaker streams are now visible and audible as expected. Our team is continuing to work on a permanent fix.
The issue has been resolved. The disruption was caused by a new flow that was recently introduced. We have reverted the change and monitored the situation, and everything is now back to normal. We apologize for any inconvenience this may have caused and thank you for your patience.
We have identified an issue affecting our marketing website (https://livestorm.co/ ), which is currently unavailable. Please note that the Livestorm application (https://app.livestorm.co/ ) remains fully operational and is not impacted. Our team is actively working on resolving this issue and will provide updates as soon as more information becomes available. Thank you for your patience.
Our team is rolling back the latest release and applying a fix, our marketing website will be back soon
The marketing website has been restored and is operating normally. We will continue monitoring the situation to ensure continued stability. We apologize for any inconvenience this may have caused.
We are currently experiencing a disruption affecting the Live Captions feature. Our team is actively monitoring the situation and working to ensure the system returns to normal operation as soon as possible. We’ll keep you updated as we have more information.
A fix has been applied, and the system should return to normal operation shortly. We are currently monitoring the situation to ensure stability and will provide further updates if needed.
The issue affecting Live Captions has been resolved. The audio stream pipelines and consumers have recovered and are now fully in sync, so Live Captions are being generated again as expected. We apologize for any inconvenience this disruption may have caused and appreciate your patience while we resolved the issue.
We’ve identified an issue that is preventing new events from being created via the dashboard. As a result, attempts to create events may not complete successfully. We are currently investigating the root cause and rolling back any recent changes that might have triggered this issue. In the meantime, if you encounter difficulties, you can continue creating events using our Public APIs until the dashboard functionality is fully restored.
The issue has been identified and we are currently rolling back the change that have caused this
The issue has been identified and we are currently rolling back the change that have caused this
A fix has been implemented. Event creation from the dashboard is now fully operational. We are adding additional monitoring and safeguards to prevent similar issues in the future.
Event creation from the dashboard is now fully operational. We apologize for the inconvenience and appreciate your patience. If you continue to experience any issues, please contact our support team.
Our team has currently identified an issue whereas our platform is currently experiencing a higher loading time than usual, and may be causing pages to be slower or unresponsive. We're investigating the cause and will provide an update as soon as possible.
We are continuing to investigate this issue.
Our team has successfully mitigated the issue by rolling back the latest update, and the platform is now fully accessible. We’re continuing to monitor the situation closely to ensure there is no further impact on users. We sincerely apologize for any inconvenience this may have caused.
The incident was caused by recent changes related to our entitlement checks. We are actively monitoring the system to ensure stability and confirm that no further impact occurs. We sincerely apologize for the disruption and appreciate your patience while we worked to resolve this.
Following our monitoring over the last few minutes, our team has confirmed that there is no ongoing impact in regards to the issue and users should be able to access the platform as required.
We identified an issue where our recent migration job within our new billing system unintentionally triggered usage Consumption alerting emails, resulting in automated email warnings being sent to some clients. There is no impact on event performance, usage, or billing, However, some clients may have received unnecessary MAC limit warning emails.
We have canceled the migration job responsible for this and are now working to delete all scheduled alert emails to prevent further notifications.
We have deleted all the scheduled emails, no further unexpected emails will be triggered.