Spiceworks 장애 기록

Our devops and engineering teams are investigating reports that some users are experiencing error pages when trying to access their help desk. Further updates to follow

We isolated the recent help desk access issues to a sudden spike in usage. To fix this, our team is rolling out capacity increases and performance optimizations. We're continuing to monitor closely though for the past 30 minutes global availability has been stable.

No further instances of page load issues have been reported or reproduced so we're considering this issue resolved. Please let support@spiceworks.com know if you are still seeing this issue.

Engineering and our ops team are investigating reports where an "oops" error page is displayed intermittently, along with the page loading but failing to display any tickets.

We're rolling back the underlying tech change from this morning as it is causing the performance issue.

The rollback is completed and performance has returned to normal. If you are still seeing the issue please let us know at support@spiceworks.com

No further issues have been detected. The engineering team is analyzing the root cause to ensure future stability before the code change is redeployed.

We're investigating reports of slow performance and timeouts on the Cloud Help Desk. We've confirmed this is a system-side issue and are working on a fix. Updates to follow.

We have successfully stabilized the Cloud Help Desk. Our teams will be monitoring for any further issues and investigating the root cause. Please submit a ticket via support@spiceworks.com if you still encounter excessive latency or timeouts.

We have seen no further reports of latency or timeouts, and our internal testing confirms that performance remains stable. We are now officially marking this incident as Resolved. Root Cause: The issue was triggered by a high-volume surge of API requests from a third-party data tool. This created an unexpected load on our database, specifically impacting the ticket queue and reporting sections. Prevention: To prevent a recurrence, we have implemented new traffic controls and rate-limiting measures. We will continue to monitor these systems to ensure long-term stability. If you are still experiencing any issues, please let our team know at support@spiceworks.com.

We've received reports of email notifications for tickets being delayed. Engineering are investigating the issue.

A fix has been deployed and the underlying issue is resolved. We are currently processing a backlog of delayed notifications. Delivery times will return to normal once the queue has cleared. Engineering is actively monitoring the recovery and has scaled resources to accelerate processing.

We have monitored the system and confirmed that notification delivery has returned to normal. As a result, we are marking this issue resolved. Note on Duplicates: Because of the way our retry logic handled the initial delays, some users may receive 1-2 duplicate notifications for older tickets. We appreciate your patience with this temporary "noise". If you continue to experience new delays, please contact us at support@spiceworks.com so we can investigate further.

The development team is currently investigating this issue. The following emails are impacted Ticket notifications New user invites Alert emails User portal access via email link

We are continuing to investigate this issue.

We have identified the root cause of this issue and are actively working on getting the email functionality restored. We will update this status page as we make progress.

We are still actively working on getting this resolved at the earliest as our top priority.

Our team is continuing work on the restoration of all email services (notifications, invites, and alerts). Due to the complexity of the restoration process, we cannot provide an exact time for completion yet, but we are making steady progress. We are working closely with our email provider to finalize the necessary steps. We apologize for the frustration this outage is causing your team this morning. We are fully dedicated to restoring full service as quickly as possible.

We greatly appreciate your patience while we continue to work with our email provider to resolve this issue at the earliest. This remains a top priority for us and we will continue posting status updates here.

While the issue with our emails is still unsolved, we are making progress in reaching a resolution with our service provider. We will follow up with another update tomorrow.

Resolution in progress: Outbound email notification issue (external ESP issue) We are continuing to work towards a resolution for the outbound email notification issue in the Spiceworks Cloud Help Desk (CHD). The root cause has been identified within our external email service provider (ESP) rather than our internal systems. Because the issue is localized to the ESP, only outgoing email-related notifications are impacted, and the core CHD platform remains operational. We are working closely with our ESP to complete the restoration process, with the immediate priority of returning full email delivery functionality as quickly as possible. What’s working • The Cloud Help Desk is operational: The core ticketing system remains active and accessible. -- Ticket submissions: End-users can still submit new tickets via email and through the User Portal. Please note that email authentication is affected; see the current limitations below. -- Managing tickets: Admins and technicians can create, update, and resolve tickets within CHD. Current limitations -- Update notifications: Outbound email notifications are not being sent to end-users, admins, or technicians. -- Passwordless User Portal logins: Passwordless logins for end-users to submit tickets via the User Portal are not working -- Admin and technician invitations: New user invitations are not working. Workaround: Contact Spiceworks Support for assistance with onboarding new users (this does not apply to end-users). -- Report notifications. Report completion email notifications are not being sent. Workaround: You can still use exports, but you will not receive an email when they’re ready. Simply refresh the page, and the download button will activate when the export is available. We are finalizing the remaining steps with our ESP and believe a resolution is near. Another update will be provided shortly. We sincerely apologize for the disruption this has caused your team’s workflow. We understand the critical importance of email notifications in maintaining effective communication with your end-users, and we truly appreciate your patience and continued support as we work to restore full functionality.

Email services have been restored. Our engineering team is currently monitoring the situation closely to ensure continued stability across all systems as services return to normal. We understand that you may have questions about the cause of this disruption. We are currently conducting a full review of the incident and will share more information once our investigation is complete. Thank you for your patience while we worked to resolve this issue with our service provider.

Following a period of stable monitoring, we have confirmed that the outbound email notification issue affecting the Spiceworks Cloud Help Desk (CHD) is fully resolved. All systems are operating normally, and we are officially moving this incident to "Resolved." All email-dependent features are now functioning as expected, including: • Ticket Notifications: End-users, admins, and technicians are receiving ticket update emails • User Access: Passwordless User Portal login links (magic links) are working • Onboarding: New admin and technician invitations are being sent successfully. • Reporting: Email notifications for report completions are being delivered correctly. We are now shifting our focus to a comprehensive review of the email service provider (ESP) disruption to ensure long-term stability and to prevent a recurrence. We sincerely apologize for the disruption this incident caused to your workflows and deeply appreciate your patience and understanding throughout the resolution process. Thank you for your continued support.

Cloud Help Desk has been affected by the issues with Amazon Web Services https://health.aws.amazon.com/health/status. Amazon has reported that most requests are now completing and in our testing Cloud Help Desk is back up and running. We are continuing to monitor the situtation.

This has now been resolved.

We're seeing reports that the management web page for connectivity dashboard is failing to load. Engineering have been made aware and are investigating.

Engineering were able to establish the root cause of this issue and implement a fix. Testing shows normal service is restored. If you're still seeing issues, try clearing your browser cache first. If the error still occurs, please open a ticket at support@spiceworks.com

Following reports of push notifications not being sent for the mobile app, engineering have identified the likely cause. Work is underway to fix this as soon as possible.

Our engineering team is actively working on resolving the issue with push notifications but we don't have an ETA for a fix yet. This remains a high priority, and we will provide another update as soon as more information is available. We apologize for the continued inconvenience.

Notifications are back up and running. We are continuing to monitor the situation.

We're seeing a small number of users reporting the issue started to happen again today. We reproduced it with an in house device and engineering are looking into the issue.

We kept this open to ensure the issue did not affect even a small number of users. The issue is now resolved and we've seen no further reports. If you are still seeing an issue with your mobile app notifications, please open a ticket at support@spiceworks.com

Engineering are investigating reports of slow/failed page loads. We will update this page as information becomes available.

The page load issue is now resolved and our teams are monitoring to ensure stability continues. Please let us know at support@spiceworks.com if you're still experiencing this problem

We're seeing no further issues and are considering this resolved. Please reach out to support@spiceworks.com if you're still seeing page load/timeout issues with cloud help desk.

The maintenance, which began at 12:30AM EDT, did not proceed as planned, and our systems remain offline. Current Impact: * Multi-Factor Authentication (MFA) for login remains temporarily disabled. You will still be able to log in using your username and password, but the second authentication step will not be required. * New account registration remains suspended. * All transactional emails (e.g., password resets, change email address) are currently not being sent. Our engineering team is actively investigating the issue with the highest priority and is working to restore full service as quickly as possible.

The maintenance is complete and transactional email operational.

The engineering team is investigating. Users have mentioned this affecting chrome/chromium browsers but not Firefox. You may also be able to log into the community or cloud help desk first, and then navigate to https://apps.spiceworks.com/tools/device-inventory/ directly

Code has been rolled back that was causing this issue. Our engineering team will establish a suitable fix so that the code can be redeployed at a later date.

The engineering team has been notified and are investigating the issue.

The backlog for reports has now cleared and it appears they are being processed normally.

Investigation is still ongoing as we saw this issue for a period of time again this morning.

Engineering have a potential fix that will be applied late Wednesday June 4th (Central US time). The impact of this will be monitored to establish if additional work is required.

Monitoring indicates the potential fix did not improve the issue. Engineering are treating this as a priority and will deploy additional logging tomorrow, in order to better trap the root cause of the issue.

The root cause of this issue has been identified. A fix is being developed in the current sprint. A further update will be made once that code is tested and deployed.

A fix has been implemented and affected users with support tickets have confirmed it is effective. If you are still seeing this issue please submit a ticket with support@spiceworks.com

While trying to improve deliverability of these emails, we wanted to communicate two things to folks which may help. 1. If you are stuck with a locked account from retrying codes or getting MFA emails in the wrong order, doing a password reset from our login page and changing your password will also unlock the account. This will save you waiting on support to unlock your account. 2. If you use Mimecast, its "greylisting" feature may be delaying these new MFA emails. More on that here https://community.spiceworks.com/t/cloud-help-desk-release-notes-mar-19-2025-mfa-for-cloud-help-desk-employee-accounts-is-here/1186674/14?u=craig-spiceworks

We're seeing lower ticket volumes today and it appears email filters are getting "used" to the new emails. If you are still having issues, let us know at support@spiceworks.com

We have notified our ops team of an issue where the cloud help desk intermittently displays a "bad gateway" or similar error. An investigation is underway.

We're seeing stability restored to the app. Please let us know at support@spiceworks.com if you are still having this issue.

Testing here and feedback from users is that we're back to full strength. The root cause appears to have been a network related issue causing the pods running cloud help desk to fail. We're altering the alerting mechanisms to give us more advance notice of this in future, so we can further avoid or minimize disruption. Apologies for the hassle! If you are still seeing issues, please get in touch with us at support@spiceworks.com

Our engineering teams are investigating an issue where some Spiceworks emails are not being sent out. Cloud help desk emails for tickets are unaffected as these use a different system.

We have been monitoring a fix for this issue which appears to have been successful. Please email support@spiceworks.com if you are still having issues.

Our development team is working on a fix for this issue. New users may not be able to complete their Spiceworks profile during signup. We have reports that Cloud help desk logins will work via a web or mobile web browser, but you will see an error while trying to login to the mobile app.

A fix has been posted. If you're still seeing the error when signing up, please let us know by emailing support@spiceworks.com

Feedback from affected folks indicate this issue has been resolved. If you were affected by this issue, our dev team would really like to get the results of the following URL's to ensure we prevent it in future. Results from visiting https://community.spiceworks.com/.well-known/conninfo IPv4 address from ip4.discourse.org IPv6 address from ip6.discourse.org Please send those results to support@spiceworks.com with the subject line "Registration Issue IP information". Thank you and sorry for the hassle.

This is not an outage, just to inform folks of the change with multiple communication methods. To improve account security and align with industry best practices, we are introducing a 7-day expiration period for session tokens for the Spiceworks Cloud Help Desk (CHD). Why This Change? This update reduces the risk of unauthorized access from stale or compromised tokens, ensuring enhanced protection for your accounts and data. What You Need to Know This update will take effect on January 8, 2025, and you will be required to log in on your first session after that update. Active sessions will remain valid for up to 7 days from the time of issuance. Expired tokens will trigger a logout, and you’ll be prompted to log in again. After expiration, users will be required to log in again to authenticate and obtain a new session token. This is specifically for CHD and our other IT tools; this will not affect your Spiceworks Community session token. This will only affect users who need to log into Cloud Help Desk and should not affect the end-users that you support. If you do not remember your password, you can reset it by going to your Spiceworks Account Management page. https://accounts.spiceworks.com/ If you have questions or need assistance with this change, please contact our Support Team by posting in the Cloud Help Desk support area or opening a support ticket by emailing support@spiceworks.com. Thank you for helping us maintain a secure environment for all members and users.

Our dev ops team has resolved this issue and feedback from the community shows access is restored. We apologize for the inconvenience. If you are still seeing the SSL error when logging into the help desk or browsing help desk pages, please let support@spiceworks.com know.

We have reports of slow page load times (or failure to load) when using the back option to move away from a single ticket to the ticket list. The development team is aware of and investigating the issue.

A fix has been put in place. Please let support@spiceworks.com know if you're still seeing the issue on the "no split" ticket view.

Indications from users testing, and our internal tests show the fix is effective.

Starting approx 70 minutes ago, agents began to experience delays and error responses when pushing updates. We are working to resolve this issue with handling inbound requests. This impacts all agents for the Inventory online and Connectivity Dashboard products. Viewing device information in Inventory online is unaffected, but new information is delayed. Connectivity Dashboard stats will not be displayed until the issue is resolved.

Agent updates are now flowing through normally. No action should be necessary on devices running the agent. Agents automatically retry on a set interval, so any failed attempts to push data will simply resend. You should see Connectivity Dashboard stats for online devices, now. Inventory online data updates should recover on the normal refresh interval.