Störungsverlauf von Uservoice

UserVoice had a database cluster crash that caused an outage this morning. The service has recovered, however email notifications may be laggy while internal recovery completes.

Email notifications should be fully recovered now. Continuing to monitor and verify.

This incident has been resolved.

The admin settings page (and all affiliated pages) have now been restored. We are still actively monitoring and investigating.

This incident has been resolved.

We are investigating why our admin settings pages are currently unavailable. Please stand by.

We are currently investigating this issue.

The issue has been identified and a fix is being implemented.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

When creating a profile with Email & Password auth option, after receiving the verification email, when users click 'create account', the application loads continuously. We've identified the request to our email verification provider, Magic.link, is hanging and not responding. We're continuing to investigate the issue.

Thanks for your patience while we continue to investigate.

A fix has been implemented and we're monitoring. Please let us know if you continue to experience issues.

This incident has been resolved.

We are currently investigating an issue with Public Status Updates emailed to subscribed users via our Quick Actions panel.

The issue was identified by engineering team and a fix has been deployed.

This incident has been resolved.

We are investigating an issue where some small portion of requests are failing to resolve.

CloudFlare has identified the issue and is working on resolution: https://www.cloudflarestatus.com/incidents/55w67hz7kjw5

CloudFlare has applied a fix and we are monitoring to confirm full performance and stability.

This incident has been resolved.

We are currently investigating some issues with search on recently created suggestions.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

UserVoice went down due to a MySQL crash at 11:52 AM EST. A fix has been implemented and we are monitoring.

This incident has been resolved.

CSS and javascript assets were failing to load causing intermittent blank pages and errors across the site. The error has been resolved and we are monitoring

We are continuing to monitor for any further issues.

This incident has been resolved.

We have received reports and are investigating with Microsoft a higher volume of UserVoice notifications with delayed delivery or being quarantined when the recipient uses Microsoft365 for Email.

This incident has been resolved.

We are currently investigating this issue.

A fix has been implemented and we are monitoring the results.

A security hardening aspect of our cluster infrastructure inadvertently prevented a healthy restart of our load-balancing services. This caused a propogation failure that prevented traffic from being served. We have resolved the issue and are investigating additional testing resources to verify it can't reoccur.

We are currently investigating an infrastructure issue causing errors and slow performance.

We have mitigated DB issue and are monitoring to confirm.

This incident has been resolved.

We are currently investigating an issue resulting in a significantly increased error rate accessing the UserVoice application.

We are continuing to investigate this issue.

A fix has been implemented and we are confiming no existing issues remain.

This incident has been resolved.

Searches for ideas are showing partial results. Ideas which have been recently created or updated are showing up in search results. We are back-filling the data and will update you when search functionality is completely restored.

Functionality has been restored. Please contact support with any issues or concerns.

We are experiencing an issue where search is currently unavailable.

Search for suggestions, tickets, and users is returning partial results as we recover from our outage. We will update you when search is again fully operational.

Search has been fully recovered. Our engineers are currently verifying all search queries are behaving as expected.

This incident has been resolved.

We are currently investigating this issue.

The issue has been identified and a fix is being implemented.

We deployed a major upgrade to our backend about 20 minutes ago. This upgrade was reverted after about 5 minutes due to extra cautious error detection. After reverting, the backend was having issues rendering its already generated cached pages. This has now been resolved.

We have identified an issue with our upstream email provider which is delaying outgoing notifications for parts of our app.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

There was an issue between 9:00-9:40 EST that would have seen new user sign ups and password resets fail to send verification emails and would have blocked this functionality. The issue has been fully resolved.

The following notification emails are affected (Legacy HelpDesk/Ticketing features are not impacted): - New Comment - Supporter Message - Feature Update - Suggestion AutoResponse - Public Status Update - Internal Status Update - Discovery Account Create - Validation Account Create - Validation Invitation - One:One Comms - Outreach The provider we use to send these emails is currently investigating the issue.

The cause was identified and the issue has been resolved. Issue: The email template configured for these types of notification emails had an error during editing. Fix: The template has been reverted back to a working version.