53 Sharpencx incidents · апрель 2024 г. — official updates, affected components, duration and resolution details.
В настоящее время мы расследуем проблему с SharpenQ в зоне изоляции IZ1, вызывающей ухудшение производительности. Длинная длина соединения вызова и один способ передачи звука.
Реализовано исправление, и в настоящее время мы отслеживаем производительность SharpenQ.
Этот инцидент был урегулирован.
Автоматический перевод официального обновления инцидента.
В настоящее время мы расследуем проблему, затрагивающую входящие звонки в среде IZ1. Некоторые клиенты могут испытывать неудавшиеся или пропущенные входящие звонки в течение этого времени. Наша команда активно работает над выявлением первопричины. Мы опубликуем обновление, как только у нас будет больше информации.
Мы определили проблему, затрагивающую входящие вызовы в Classic Logic на IZ1, и применили исправление. Звонки должны снова работать, и мы продолжаем следить, чтобы подтвердить полную стабильность.
Входящие звонки, направляемые в Classic Logic на IZ1, видели мертвую тишину, начинающуюся около 5:42 вечера по восточному времени. Наша команда проследила это до проблемы с производительностью базы данных, затрагивающей голосовую обработку, и исправила ее к 6:56 вечера по восточному времени. Звонки вернулись к норме.
Автоматический перевод официального обновления инцидента.
В настоящее время мы знаем о проблеме в нашей зоне изоляции IZ1, влияющей на производительность Sharpen Q, включая задержки в подключении к вызову, влияние качества звука и задержки с подключением исходящих вызовов. Наша команда активно исследует и работает над решением.
Мы считаем, что мы определили проблему, влияющую на производительность SharpenQ в IZ1, и внедряем исправление.
Было реализовано исправление, и мы наблюдаем улучшение производительности Sharpen Q для качества вызовов и задержек исходящих вызовов. Мы продолжаем следить за текущей стабильностью.
Мы по-прежнему видим улучшение стабильности/восстановления платформы с тех пор, как наши инженерные команды внедрили исправление. Возможности исходящего вызова, производительность очереди, качество вызова и т. Д. Все они восстановлены до базовых уровней функциональности. Если вы продолжаете видеть влияние определенных очередей, конечных пользователей или конкретных действий в пакете приложений Sharpen - пожалуйста, свяжитесь с нашей командой по уходу через существующий случай поддержки.
Автоматический перевод официального обновления инцидента.
We are aware of an incident preventing outbound calls from completeting in Sharpen Q. We are working with all avaiilable resources to resolve.
Outbound Dialing in SharpenQ has been resolved and we are continuing to monitor for stability.
This incident has been resolved.
Our engineering teams are aware of an infrastructure issue that is impacting call routing, WebRTC connectivity for users, and various page navigation efforts (Insights, Admin pages, Logic configuration editors, etc). We have pin-pointed the fault zone and have all necessary resources engaged to restore services as quickly as possible.
Our engineering teams are actively working through corrective actions, and further status updates will be provided as a resolution timeline is established.
The corrective action has been deployed in our infrastructure, and all identified service fault-zones have been addressed. Insights reporting, call routing, and WebRTC connection are now responsive and behaving in the expected fashion. We're actively monitoring with our engineering team. If you're continuing to see impact, please reach out to the Care Team via your existing support case!
Validation testing performed across the application suite has confirmed recovery to all impacted systems. In the event you are still encountering lingering effects tied to interaction routing, WebRTC connectivity problems, or an inability to access a specific configuration page within the app - please reach out to our Care Team for further assistance!
We're currently investigating an issue for IZ1 customers, where Inbound routes pointed to a Classic Logic configuration are experiencing unexpected disconnects. We're actively able to reproduce the problem and are working to restore service as soon as possible.
An attempted remediation effort was made to address calls dropping in Classic Logic. While that did have the expected outcome, a second new set of symptoms were introduced. This includes delays in calls routing to agents in SharpenQ, and delays when agents attempt to connect to the interaction. Our teams are continuing to investigate with all available resources.
We've identified the underlying issue and are working with our internal teams to get a fix deployed across our Production environment. We're targeting a comprehensive roll-out for our infrastructure in the next 30 minutes. Further updates will follow as soon as corrective actions have been fully implemented.
The corrective changes have been made across our infrastructure, and we're actively monitoring the system/running health-checks. The expectation is that call routing through Classic Logic, call audio/quality, and interaction handling behavior should be considerably improved for impacted users.
Service health-checks and sustained monitoring of our call routing systems have shown recovery to the platform. Validation testing across Classic Logic configurations or inbound routes that send calls directly to queue are routing in a timely fashion. In the event you have users still encountering connection issues, please reach out to the Care Team with examples of the misbehavior.
We are currently investigating reports of degraded call quality affecting some customers, intermittently. The following behaviors have been reported beginning around 12:00 PM ET: - Delays (up to 25+ seconds) between a call connecting and the interaction appearing in the agent interface - Dead air on inbound calls prior to ringing and after agent answer - Calls populating in queue but experiencing delays before being assigned to an agent - Absence of hold music for callers waiting in queue - Inability to place calls on hold Our team is actively investigating the cause. We will provide an update as soon as additional information is available. Affected: Voice interactions
We are continuing to investigate the issue and have all necessary resources engage to resolve the issue as soon as possible.
Thank you for your continued patience. We are continuing to investigate the issue with all necessary resources involved. The problem is localized, but thus far our remediation efforts have not sufficiently remediated impact. We are continuing to explore all options to resolve the issue.
We are continuing to investigate this issue.
Thank you for your continued patience. We continue to aggressively address the intermittent service degradation impacting call connection times (agent-caller delay) and interface responsiveness. We will continue to pursue resolution as soon as possible.
We are continuing to investigate the issue. We're receiving additional reports of some users experiencing intermittent 504 Gateway timeout errors. We are including this symptom in our investigation as it likely shares the same root cause.
We are continuing to investigate this issue. We have introduced 1 fix and are working on another which we expect will improve performance.
Our initial fix is showing significant effectiveness, drastically improving the timeliness of Sharpen Q voice interaction related interactions. Call delivery, answer times, and interaction call controls should mostly be restored at this point. We've identified an additional fix which will bring further improvements shortly.
We are continuing working through our second fix and will update this page when it is complete.
We are continuing to monitor and test to validate improvement following our recent 2 fixes.
Data, Monitoring, and testing has confirmed recent fixes have resolved this issue.
This issue was resolved at 9:25 PM ET 2026-02-19. Our status page mechanism encountered an issue at the time of us marking resolution. This is retroactive conclusion of the incident.
Users of the Sharpen platform in IZ0 are likely experiencing a variety of symptoms mostly impacting Sharpen Q and Insights. Our teams are engaged working to resolve the issue as soon as possible.
Sharpen Q has been restored. We are now working on restoring Insights defects and expect recovery shortly.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We are currently investigating a failure to load Insights Dashboards
A fix has been implemented and we are monitoring the results. A refresh of the Insights dashboard will be necessary to restore functionality.
This incident has been resolved.
We are currently aware of the following symptoms in our IZ0 isolation zone - Inbound voice calls delivered through Logic+ result in dead air - Inbound emails do not enter queue In our IZ1 isolation zone Inbound emails do not enter queue We have all necessary resources engaged and are working to restore service as soon as possible.
We have identified the fault for both symptoms and are working through remediation steps now. We are targeting resolution within the next 30 minutes.
Inbound voice through Logic+ in IZ0 is now resolved. We are continuing working through the fix for inbound email failure.
We are continuing our efforts to resolve the inbound email fault. The fault is still properly understood, and we continue working through diligent corrective actions to restore service as soon as possible. Emails which have not been delivered during this impact will not be lost, and we expect them to queue up when this fault is resolved.
We are continuing working on a fix for the email issue and have all necessary resources involved to resolve as soon as possible. Thank you for your patience.
We have identified a new option for recovery and are working through implementation steps now.
We have attempted a fix, which yielded positive progress, but we are still working on a complete fix. Thank you for your patience as we continue our investigation.
We have identified a new path for resolution and are orchestrating a fix now.
We are continuing our investigation. The root cause remains understood, but our attempted changes, allowing the proper communication between our Email service and Sharpen Q, have not yet allowed for the services to properly regain expected communication. We will continue to work diligently until this is resolved.
We have engaged additional resources from our Cloud Service provider to accelerate resolution.
We have an alternative fix ready for deployment and are initiating shortly.
We have identified a new path for resolution and are working to build a new fix. We are targeting a fix within the hour.
We are continuing to work on a fix for this issue.
We have introduced a fix and are seeing some recovery. We are validating expected email functionality now.
We have achieved partial recovery. If emails are forwarded to the Sharpen queue email address, functionality is restored. However, if the sharpen queue email address is included within a distribution group, you are likely still seeing failures. This symptom was confirmed using Microsoft Exchange distribution groups. If you have the ability to forward directly to email addresses, you will experience recovery. The team continues to work toward a full fix regardless.
The issue has been identified and a fix is being implemented.
A fix has been implemented to our IZ0 isolation zone. We are working on propagating it to our IZ1 isolation zone now.
This incident has been resolved. Both IZ0 and IZ1 are working as expected.
Starting at approximately 8PM Eastern, users in the Queue application are experiencing a "Could not place call" error in the UI. We're aware of the problem and actively engaged with our engineering teams to restore functionality as quickly as possible. Further updates will be provided as soon as we have recovery actions identified.
Outbound calling was restored at 9:26PM Eastern, and all monitoring/validation testing since that time has consistently shown recovered call processing behavior. In the event you're still experiencing an issue, please reach out to the Care Team!
As part of the wide-ranging impact that various technical vendors are experiencing (as part of this Amazon Web Services disruption), the Sharpen team is tracking a variety of issues across our platform that involve: - Delayed interaction routing/connectivity behavior - Inability to place outbound calls - Web Chat functionality inconsistencies - SharpenDB querying or data-return issues - Missing or unresponsive interaction control buttons in the Queue interface Our teams are actively working to update the application, in order to address service dependencies or availability issues throughout the platform. We'll continue to update this tracking item as more zones of impact are identified.
We are continuing our efforts to work through these issues - focusing on ways to avoid impacted AWS end-points/services. As certain features are restored to an operational status, we'll continue to update this tracking item.
This incident has been resolved as of 6:53 PM Eastern time upon the conclusion of the AWS incident. Our teams continue to monitor for any additional impact.
Due to an outage with our cloud service provider, the SharpenDB end-point is experiencing degraded availability and is not able to consistently return data. Navigating to the Sharpen Admin --> config tab --> SharpenDB container may show a "Zero Tables Found" error, and various DB lookups (initiated through Classic Logic, Logic+, or through Query API) may not return any results. We've engaged with our cloud service provider and are working on restoring functionality for SharpenDB as quickly as possible. Updates will follow as additional details or restoration timelines are identified.
Our cloud service provider (AWS) has isolated the source of the network connectivity/end-point misbehavior, and are actively working towards resolution. We don't yet have a recovery timeline to share, but will provide further status updates when stability returns to effected services.
Our Development Team has performed an update to circumvent some of the dependencies/communication issues being experienced with our cloud services provider (AWS). We're performing validation testing for SharpenDB's access and querying capabilities, and are seeing a restoration in availability for that portion of the platform. Please test at your discretion, as we continue to make updates and improve platform stability.
This unique tracking items is being closed out, in preference of centrally tracking known areas of impact through a general tracking item (that ties back to the identified service interruption with cloud service provider AWS). Further updates on SharpenDB performance, API end-point engagement, and any other newly identified fault-zones will be communicated using our general tracking item hosted on the Status Page.
Due to a outage with a cloud service provider, the Sharpen Service desk is experiencing reachability issues. The SharpenCare team is still reachable via phone or chat, however web tickets cannot be viewed or submitted. We are monitoring updates from our service provider for resolution.
This incident has been resolved with the resolution on the AWS outage at 6:53 PM Eastern Time. Our team has validated the restoration of this service.
We are currently investigating an issue where CX CURL requests in Logic Classic result in dead air.
We are still investigating the issue where VCX/CX routes in Logic Classic result in dead air. Our engineering team is continuing to work toward identifying the root cause.
We are still investigating the issue, and our teams are fully swarmed working toward a resolution as quickly as possible.
Initial corrective actions have been taken, and we've seen our CX/VCX hosts return to a healthy status. Our focus has shifted towards restoring functionality to the portion of the system that allows Inbound Routing or timed project triggers to engage with a deployed CX/VCX project. Additional updates will be provided as we make progress towards comprehensive recovery.
To clarify our last update: while CX/VCX hosts are reporting healthy, the outage is not fully resolved. Inbound Routing and timed project triggers that connect to deployed CX/VCX projects are still impacted. Our team is actively working to restore this functionality, and we’ll continue to share updates as progress is made.
We've identified the underlying issue that is preventing different portions of the system (Classic logic, Inbound Routes, etc) from being able to engage with CX/VCX projects. We have been able to partially restore service to some applications, and are continuing to work towards comprehensive recovery. NOTE: Some projects are operational at this time while others CX/VCX deployments continue to see impact. We're aware of this behavior and are continuing efforts to correct all projects.
We appreciate your continued patience. Our team remains engaged and is working to resolve the impact. For many customers we have implemented workarounds or alternative solutions that have restored customer experiences. We are still pursuing all options of service restoration. If you are impacted and have unique needs to overcome this disruption, please reach out to us at [email protected]
We're continuing our efforts to resolve CX/VCX routing issues across the IZ1 infrastructure. While we've had partial success, certain projects remain unreachable and currently result in a dead-air experience for customers. Mitigation efforts have been underway with the Care Team, involving updating routing conditions (away from CX/VCX) to minimize impact. In addition to Inbound routing or Classic Logic call-flows, this misbehavior extends to in-queue operations like timed-bots, "Max Wait", and "Exit No Agents" conditions that point to a Callback/Voicemail related VCX customization. The short-term recommendation is to disable timed-bots, and instead leverage the in-queue callback or voicemail offering. Further communication will follow, once we've established recovery for CX/VCX project usage. In the event assistance is required to update queue configurations, please reach out to the Care Team via [email protected]
At this stage we have restored service to the majority of impacted flows and are working directly with customers who are experiencing any continued impact. If you are experiencing a problem with IVR routing or in queue messaging regarding the use of VCX/CX, please reach out to us at support.sharpencx.com so we can assist.
Starting at approximately 9:30PM Eastern, calls routing through our IZ0 infrastructure that engage a transfer action (whether that be a queue-based "Max Wait" or "Exit No Agents" condition or an agent-initiated transfer to a different queue) are experiencing an unexpected termination. Test calls hitting one of these transfer conditions will hear a series of beeps, followed by the unexpected disconnect.
We've identified a routing issue within our infrastructure that's uniquely tied to transfer functionality (both queue and agent-initiated) and are working towards a fix. All available resources have been engaged at this time, and we're actively working towards recovery as quickly as possible. Further updates will be made to the status page when we've reached a point of recovery.
Our teams are currently working towards rolling out the hot-fix/update across Sharpen's infrastructure. As soon as that effort has been completed, we'll initiate validation testing and confirm recovery of the system. Further updates will be posted to the status page as soon as that task finishes.
The corrective hot-fix has been deployed throughout our infrastructure, and initial validation testing has confirmed a restoration in queue-based and agent-initiated transfer functionality. Please verify with end-users if they're seeing a restoration in functionality. If ongoing impact is experienced, please reach out to the Care Team for further troubleshooting/analysis - with example users/extension info/QCMIDs that can be reviewed.
Continued validation testing and monitoring efforts have shown comprehensive recovery across the Sharpen Q system. This incident is being transitioned into a monitoring status. In the event that new misbehaviors or lingering symtoms/issues are encountered, please reach out to the Care Team.
Inbound numbers that route through a Logic+ are resulting in dead-air. We are currently investigating and working towards a resolution
We identified the root cause impacting inbound calls routed through Logic+ and have applied corrective measures. Calls are now completing successfully. We will continue to closely monitor the service to ensure full stability.
The issue with inbound calls routed through Logic+ has been fully resolved. All services are operating normally.
Starting shortly after 09:00AM Eastern, Inbound numbers that route though a Logic+ configuration are intermittently resulting in a dead-air or delayed playback misbehavior.
We're aware of an ongoing/active issue, that is causing Inbound calls to experience dead-air or extremely delayed prompt playback - specifically when the number processes through a Logic+ configuration. We're investigating some infrastructure alerts that were triggered, and have all necessary team members engaged to troubleshoot the issue. We'll post further status updates as additional details become available.
We've isolated the underlying issue within our infrastructure, and have taken initial steps to reroute traffic away from the misbehaving pathway causing the intermittent disconnects, delayed prompt playback, or dead-air. Our internal teams are actively preparing a code-change to completely address the problem, and we expect to have that pushed out to our Production systems in the next few minutes.
Our IZ1 infrastructure update has been deployed to the Production system, and initial validation efforts have shown full recovery to previously impacted call-flows routing though Logic+. In the event that you're still experiencing symptoms tied to dead-air, latent prompt playback, or unexpected disconnects through a Logic+ bot, please reach out to the Care Team with examples of the misbehavior and we'll troubleshoot further!
We are currently investigating an issue impacting the inability to place outbound calls in our IZ1 infrastructure.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We are aware of a potential service disruption impacting our IZ1 Infrastructure that may result in inbound calls not reaching queues. We have all necessary resources engaged to restore service. Additional details will be provided soon.
We have identified the cause and are working on a corrective fix.
We have corrected the issue impacting IZ1 application availability and are continuing to monitor.
This incident has been resolved. Any agents experiencing lingering impact may need to log out of app.iz1.sharpen.cx and log back in.