53 Incident Iq incidents · 2022年3月 — official updates, affected components, duration and resolution details.
我们知道,有些地区在进入事件智商平台方面存在问题。 事件智商平台没有问题, 而是更大的互联网基础设施问题, Cloudflare报告了这一问题并正在积极进行补救。 https://www.cloudflarestatus.com/incicents/dk2obcyw2aew 互联网档案馆的存檔,存档日期2013-12-21
Cloudflare已经解决了第三方网络事件影响区接入. 所有服务都正常运行,各区现在应能毫无问题地进入事件智商。 有关该事件的详细情况,请参见https://www.cloudflarestatus.com/incicents/dk2obcyw2aew.
自动翻译自官方事件更新。
We want to inform you that we are currently experiencing a performance issue that may impact your experience on our platform. Our engineering team is actively investigating the issue, and we are working to resolve it as quickly as possible. We apologize for any inconvenience this may cause and appreciate your patience. We will keep you updated on our progress and notify you as soon as the platform is performing at optimal levels.
Our engineering team has identified the root cause and applied a fix. We are actively monitoring the results, and services are recovering.
Our internal monitoring and testing confirm that the platform and all services are fully restored and operating normally. We apologize for any disruption this issue may have caused and appreciate your patience while our team worked on a fix. If you continue to experience any issues, please don't hesitate to reach out to our support team.
We are experiencing intermittent site responsiveness issues, causing slow page loads and feature performance. Our engineering team is currently investigating and working on a solution. We will provide updates here as soon as they become available. We apologize for the inconvenience and appreciate your patience.
Our engineering team has identified the root cause and applied a fix. We are actively monitoring the results and expect services to recover over the next 30 minutes.
Our internal monitoring and testing confirm that the platform and all services are fully restored and operating normally. We apologize for any disruption this issue may have caused and appreciate your patience while our team worked on a fix. If you continue to experience any issues, please don't hesitate to reach out to our support team.
On Tuesday morning, 08/04/2026, a performance update caused temporary and intermittent slowness across parts of the platform, particularly during batch check-in and check-out. Our team isolated the cause, reversed the update, and fully restored normal performance by early afternoon. No data was impacted. We have paused these types of updates during the busy back to school season and are updating our internal monitoring to ensure faster detection moving forward.
We are currently experiencing an outage from our phone vendor. To contact Support please put in a ticket via the in application messenger, or email us at [email protected]. We will provide an update here as soon as the phones are working again.
Resolved: Phone Services Restored The issues affecting our phone systems have been fully resolved, and all lines are back up and running. We apologize for the inconvenience and appreciate your patience while we worked to restore functionality.
We are investigating reports of an issue affecting the use of certain characters in the ticket comments fields. Workaround: You can type up notes in another field or text editor, such as Word or Notepad, and copy the text into the comments field. The rest of the platform remains operational. We are working to identify the problem and will update you soon.
We are pleased to share that a resolution has been implemented, and our team is currently monitoring the system to ensure continued stability. If you are still seeing any irregularities, please refresh your session by closing and restarting your browser.
The underlying root cause has been addressed and our monitoring confirms that all systems are functioning as expected. If you continue to see any local cached issues, we recommend a final browser restart.
We are aware of an issue affecting the loading of tickets. Our Engineering Team is currently investigating the cause and working on a solution. We will provide updates here as soon as they are available. Apologies for the inconvenience and thanks for your patience as we work to resolve this issue.
We are pleased to confirm that the issue has been resolved, and full functionality is restored. Our team is actively monitoring the system to ensure continued stability. Should you experience any further issues, please contact our support team immediately. We apologize for any inconvenience this caused and thank you for your patience.
We are pleased to report that the service issue reported earlier today has been resolved. Following a thorough review, our internal monitoring and testing have confirmed that all systems are fully operational. At approximately 1:30 PM ET, our support team was made aware of an issue preventing certain pages from loading within the platform. Our Engineering team, which was already investigating, confirmed that full system functionality was restored at 2:10 PM ET. We sincerely apologize for the disruption and appreciate your patience as we verified system health. If you are still experiencing any issues, please contact our support team directly for further investigation.
At separate times on October 27th and 30th, IncidentIQ experienced brief periods of degraded platform performance. During these windows, users experienced slow page loads, timeouts, and overall sluggish response times across several areas of the application. The issues were remediated by adjusting, but not fundamentally changing, cloud database configurations. Initial investigation by the engineering team identified issues with a patch deployed by one of our upstream cloud vendors. While working with the vendor to confirm the cause and roll back the change, iiQ engineers increased capacity and deployed additional resources to ensure performance was not impacted between October 30th and November 7th. On November 7, 2025 the provider’s engineers rolled back the patch overnight. No similar incidences of these symptoms have occurred since the rollback, and we expect the rollback will fully resolve this class of issues. We know that platform reliability is critical for your team's operations. Thank you for your patience as we resolved this with the vendor.
We are aware of intermittent issues affecting page loads and login issues. Our Engineering Team is currently investigating the cause and working on a solution. We will provide updates here as soon as they are available. Apologies for the inconvenience and thanks for your patience as we work to resolve this issue.
We are pleased to report that all systems are stable and operating normally. Our team will continue to closely monitor performance to ensure continued stability. Thank you for your patience.
Following a review, our internal monitoring and testing show that all systems are fully operational. We appreciate your patience as we verified system health. If you are still experiencing any issues, please contact our support team directly so we can investigate further.
At separate times on October 27th and 30th, IncidentIQ experienced brief periods of degraded platform performance. During these windows, users experienced slow page loads, timeouts, and overall sluggish response times across several areas of the application. The issues were remediated by adjusting, but not fundamentally changing, cloud database configurations. Initial investigation by the engineering team identified issues with a patch deployed by one of our upstream cloud vendors. While working with the vendor to confirm the cause and roll back the change, iiQ engineers increased capacity and deployed additional resources to ensure performance was not impacted between October 30th and November 7th. On November 7, 2025 the provider’s engineers rolled back the patch overnight. No similar incidences of these symptoms have occurred since the rollback, and we expect the rollback will fully resolve this class of issues.
We are experiencing some intermittent site responsiveness issues that are causing slow page loading times. Our Engineering Team is currently investigating the cause and working on a solution. We will provide updates here as soon as they are available. Apologies for the inconvenience and thanks for your patience.
A fix has been implemented and we are monitoring the results.
Apologies for the inconvenience and thank you for your patience This incident has been resolved. If you experience any additional problems please reach out to Support.
We are aware of an intermittent issue affecting the loading of tickets and assets views. Our Engineering Team is currently investigating the cause and working on a solution. We will provide updates here as soon as they are available. Apologies for the inconvenience and thanks for your patience as we work to resolve this issue.
We have deployed a fix to address the intermittent issue with tickets and assets views loading and are currently monitoring the situation. We will provide additional updates as they become available. In the meantime, please contact our support team if you have any additional questions or concerns during this time.
Our internal monitoring and testing indicates that the platform and all services are functioning as expected. If you have any additional questions or concerns, please reach out to support. We apologize for any disruption this may have caused, and thank you for your understanding during this time.
We are aware of an intermittent issue affecting the loading of assets on certain pages. Our Engineering Team is currently investigating the cause and working on a solution. We will provide updates here as soon as they are available. Apologies for the inconvenience and thanks for your patience as we work to resolve this issue.
We have deployed a fix to address the intermittent asset loading issue and are currently monitoring the situation. We will provide additional updates as they become available. In the meantime, please contact our support team if you have any additional questions or concerns during this time.
We are aware of intermittent issues with check-in/out and verifications. Our Engineering Team is currently working on a solution. We will provide updates here as soon as they are available. Apologies for the inconvenience and thanks for your patience as we work to resolve this issue.
We have deployed a fix to address the issues with check-in/out and verifications and are currently monitoring the situation. We will provide additional updates as they become available. In the meantime, please contact our support team if you have any additional questions or concerns during this time.
Our internal monitoring and testing indicates that the platform and all services are functioning as expected. If you have any additional questions or concerns, please reach out to support. We apologize for any disruption this may have caused, and thank you for your understanding during this time.
We've identified intermittent issues with fee tracker app, as well as, displaying and filtering on certain elements in assets explorer. There is no loss or impact to the underlying data. Thanks for your patience as our Engineering team works to resolve the issue as soon as possible.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We are aware of an issue intermittently preventing users from accessing the Incident IQ site or seeing blank pages load. Our engineering team is aware of this and is working to restore full system functionality as soon as possible. We will provide further updates as they become available.
We have deployed a fix to address the login and page-loading issues and are currently monitoring the situation. We will provide additional updates as they become available. In the meantime, please contact our support team if you have any additional questions or concerns during this time.
This incident has been resolved. We deeply regret any inconvenience this may have caused and appreciate your understanding as we worked to resolve the issue. If you have any additional questions or concerns please contact our support team for further assistance.
We are currently aware of an issue causing delays in job processing at the moment. Our team is working on identifying and resolving the underlying cause as soon as possible. We will continue to provide updates here as information becomes available.
Our team has identified the underlying cause of this issue, and a fix has been deployed. The system is currently processing all delayed jobs, and we hope to have full functionality restored shortly.
All delayed jobs have been processed, and the system is once again performing as expected. We appreciate your patience as our team worked to resolve this issue. If you have any additional questions or concerns, please contact our support team for further assistance.
Our phone provider is currently experiencing an outage affecting our ability to receive inbound calls. Please use the in-system support ticketing or email ([email protected]) to reach out for assistance. We will provide updates as functionality is restored, and we appreciate your understanding during this time.
Our phone support is back online. We will continue to monitor the situation for the time being. If you continue to experience any issues during this time, please reach out through the in-system support ticketing or email ([email protected]).
After monitoring the situation, everything appears stable and functioning as expected. If you have any additional questions or concerns, please reach out to support. We apologize for any disruption this may have caused, and thank you for your understanding during this time.
We are aware of an issue preventing users from accessing the Incident IQ site or causing blank loading pages. Our engineering team is aware of this and is working to restore system functionality as soon as possible. We will provide further updates as they become available.
We have deployed a fix to address the slowness and page-loading issues and are currently monitoring the situation. We will provide additional updates as they become available. In the meantime, please contact our support team if you have any additional questions or concerns during this time.
This incident has been resolved. Our Engineering team is completing an investigation into the cause and will provide an update here as soon as the postmortem is complete. We deeply regret any inconvenience this may have caused and appreciate your understanding as we worked to resolve the issue. If you have any additional questions or concerns please contact our support team for further assistance.
We are experiencing some intermittent site responsiveness issues that are causing slow page loading times and intermittent login problems. Our Engineering Team is currently investigating the cause and working on a solution. We will provide updates here as soon as they are available. Apologies for the inconvenience and thanks for your patience.
We have identified the issue and implemented a resolution. We are continuing to monitor the platform for stability and responsiveness. Additional updates regarding the cause of the issue and remediation steps will be posted here as soon as they are available.
During our continued platform monitoring, we have identified additional intermittent loading and responsiveness issues. Our Engineering team is focused 100% on returning the platform to optimal operational status. We will post updates here as they become available.
We are experiencing some intermittent site responsiveness issues that are causing slow page loading times and intermittent login problems. Our Engineering Team is currently investigating the cause and working on a solution. We will provide updates here as soon as they are available. Apologies for the inconvenience and thanks for your patience.
A fix has been implemented and we are monitoring the results.
This incident has been resolved and will be updated with a postmortem as soon as our investigation is complete. We understand how essential our platform is for your schools’ day-to-day activities, and we are committed to ensuring its reliability. We deeply regret any inconvenience this may have caused and appreciate your understanding as we worked to resolve the issue.
On Monday December 2, Incident IQ experienced a service interruption that caused degradation of services for several customers. **Timeline of events:** * 9:39 AM ET - Began receiving customer reports of slowness/degradation of services for portions of the platform to include login authentication. * 11:41 AM ET - All services and platform responsiveness levels restored **Cause of Incident:** * As part of normal operating procedures, Incident IQ scales platform capacity to meet the needs of our current customer and predictive demands. * Incident IQ uses the Microsoft Azure platform to automatically scale infrastructure as part of normal operating procedures. The process to scale encountered errors on Monday December 2nd and took five times longer than normal to complete. * The combination of infrastructure not scaling on normal schedules and the larger load than the available infrastructure caused the outage. **Remediation:** * Services were ultimately restored by a combination of manually adding infrastructure resources and the automated scaling job completion * Incident IQ is working alongside Microsoft and have identified that unforeseeable abnormally high transactions on the SQL environment during scaling caused the process to run longer. * We are continuing to work with Microsoft to prevent a recurrence of this issue. The reliability of our platform remains of the utmost importance to us. We understand the impact these moments have on our customers. The remediations put in place to prevent a recurrence of this particular incident, as well as the processes we have in place to continuously improve the platform, provide us with a level of certainty that we are able to stay ahead of unexpected surges in traffic. As before, we do sincerely apologize for this disruption and want to thank you for your patience and partnership as we worked through this issue.
We want to inform you that we are currently experiencing a performance issue that may impact your experience on our platform. You may notice delays in loading ticket pages, performing searches, taking actions, or creating new tickets. Our engineering team is actively investigating the issue, and we are working to resolve it as quickly as possible. We apologize for any inconvenience this may cause and appreciate your patience. We will keep you updated on our progress and notify you as soon as the platform is performing at optimal levels. Thank you for your understanding and partnership.
The engineering team has reported that the platform is now stable, and they are actively investigating the root cause of the event to prevent any recurrence. I’ll keep you posted on any further findings or actions. Thank you for your patience and understanding.
This incident has been resolved.
We are aware of an issue preventing users from accessing the Incident IQ site or causing blank loading pages. Our engineering team is aware of this and is working to restore system functionality as soon as possible. We will provide further updates as they become available.
We have identified the issue and have deployed a resolution to restore access to affected sites are currently monitoring the situation. If you have any additional questions or concerns during this time please contact our support team.
We encountered a platform-wide issue between 11:31 am - 12:10 pm EDT which degraded performance and in some cases prevented login access. Since the remediation deployment this afternoon, we have been monitoring all platform metrics and continuing with extensive testing. We're sorry for any delays or confusion this issue has caused and appreciate everyone's patience as we worked to resolve this issue. If you have any additional questions or concerns please contact our support team for further assistance.
We're aware of an issue preventing users from closing tickets. Our engineering team is working on resolving the issue. We will continue to provide updates as they become available.
We identified an issue where only adding a resolution to actions that were not visible to requestors would prevent users from closing tickets if resolution actions were required. A fix has been deployed and this issue has been resolved. We're sorry for any delays this may have caused in closing tickets out this morning. If you have any further questions or concerns please reach out to our support team for assistance.
We are aware of an issue preventing users from accessing the Incident IQ site or causing blank loading pages. Our engineering team is working to restore system functionality as soon as possible. We will provide further updates as they become available.
We are continuing to investigate this issue.
We have restored system functionality and will continue to monitor the situation to ensure stability. We will continue to provide updates as they become available, but if you have any additional questions or concerns please contact our support team.
On September 9th, at approximately 9:49 AM ET, we identified an issue affecting critical infrastructure, which resulted in a temporary service outage. Our team immediately took action, and by 11:04 AM ET, all services were fully restored. There was no data loss during this time. Reason for Outage (RFO): The outage was caused by an unexpected issue in our infrastructure, temporarily impacting system availability. Our engineering team promptly resolved the problem to restore full functionality. Next Steps: To ensure the continued reliability of our platform, we have taken the following steps: Enhanced Monitoring: We have added additional monitoring and alert systems to detect and resolve issues before they impact your service. Infrastructure Optimization: Our systems have been further optimized and scaled to handle even greater demands, providing more stability for critical operations. We understand how essential our platform is for your schools’ day-to-day activities, and we are committed to ensuring its reliability. We deeply regret any inconvenience this may have caused and appreciate your understanding and trust. Incident Resolved: 11:04 AM ET
On September 9th, at approximately 9:49 AM ET, we identified an issue affecting critical infrastructure, which resulted in a temporary service outage. Our team immediately took action, and by 11:04 AM ET, all services were fully restored. There was no data loss during this time. Reason for Outage \(RFO\): The outage was caused by an unexpected issue in our infrastructure, temporarily impacting system availability. Our engineering team promptly resolved the problem to restore full functionality. Next Steps: To ensure the continued reliability of our platform, we have taken the following steps: * Enhanced Monitoring: We have added additional monitoring and alert systems to detect and resolve issues before they impact your service. * Infrastructure Optimization: Our systems have been further optimized and scaled to handle even greater demands, providing more stability for critical operations. We understand how essential our platform is for your schools’ day-to-day activities, and we are committed to ensuring its reliability. We deeply regret any inconvenience this may have caused and appreciate your understanding and trust.
