53 Sharpencx incidents · Απρίλιος 2024 — official updates, affected components, duration and resolution details.
Αυτή τη στιγμή ερευνούμε ένα θέμα με το SharpenQ στη ζώνη απομόνωσης IZ1 προκαλώντας υποβάθμιση της απόδοσης. Μακρά μήκη σύνδεσης κλήσεων και ένας τρόπος ήχου μπορεί να είναι έμπειρος.
Ένα fix έχει εφαρμοστεί και αυτή τη στιγμή παρακολουθούμε τις επιδόσεις του SharpenQ.
Το περιστατικό λύθηκε.
Αυτόματη μετάφραση από την επίσημη ενημέρωση του συμβάντος.
Ερευνούμε ένα θέμα που επηρεάζει εισερχόμενες κλήσεις στο περιβάλλον IZ1. Μερικοί πελάτες μπορεί να έχουν αποτύχει ή πέσει εισερχόμενες κλήσεις κατά τη διάρκεια αυτής της περιόδου. Η ομάδα μας εργάζεται ενεργά για να εντοπίσει τη βασική αιτία. Θα δημοσιεύσουμε μια ενημέρωση μόλις έχουμε περισσότερες πληροφορίες.
Αναγνωρίσαμε το θέμα που επηρεάζει τις εισερχόμενες κλήσεις στην Κλασική Λογική στον ΙΖ1 και εφαρμόσαμε μια διόρθωση. Οι κλήσεις θα πρέπει να λειτουργούν ξανά, και συνεχίζουμε να παρακολουθούμε για να επιβεβαιώσουμε την πλήρη σταθερότητα.
Οι εισερχόμενες κλήσεις δρομολογήθηκαν στην Κλασική Λογική στον IZ1 είδαν νεκρές σιωπές να ξεκινούν γύρω στις 5:42 μμ ET. Η ομάδα μας το εντόπισε σε ένα θέμα απόδοσης της βάσης δεδομένων που επηρέασε την επεξεργασία φωνής και το έφτιαξε μέχρι τις 6:56 μμ ET. Τα τηλεφωνήματα επέστρεψαν στο φυσιολογικό τώρα.
Αυτόματη μετάφραση από την επίσημη ενημέρωση του συμβάντος.
Επί του παρόντος γνωρίζουμε ένα θέμα στη ζώνη απομόνωσης IZ1 μας που επηρεάζει την απόδοση Sharpen Q, συμπεριλαμβανομένων των καθυστερήσεων στη σύνδεση κλήσεων, την ποιότητα του ήχου αντίκτυπο, και τις καθυστερήσεις με Outbound κλήσεις σύνδεση. Η ομάδα μας ερευνά ενεργά και εργάζεται προς μια λύση.
Πιστεύουμε ότι έχουμε εντοπίσει το θέμα που επηρεάζει την απόδοση SharpenQ στο IZ1 και εφαρμόζουμε μια ρύθμιση.
Μια διόρθωση έχει εφαρμοστεί και παρατηρούμε Sharpen Q βελτίωση της απόδοσης για την ποιότητα κλήσης και outbound καθυστερήσεις κλήσεων. Συνεχίζουμε να παρακολουθούμε για συνεχή σταθερότητα.
Συνεχίσαμε να βλέπουμε τη σταθερότητα/ανάκτηση της πλατφόρμας να βελτιώνεται από τότε που οι μηχανικές ομάδες μας εφάρμοσαν μια σταθερή δράση. Οι εξερχόμενες δυνατότητες κλήσης, η απόδοση της ουράς, η ποιότητα της κλήσης, κλπ έχουν όλα αποκατασταθεί στα βασικά επίπεδα λειτουργικότητας. Σε περίπτωση που συνεχίζετε να βλέπετε αντίκτυπο με συγκεκριμένες ουρές, τελικούς χρήστες, ή συγκεκριμένες ενέργειες εντός της σουίτας εφαρμογών Sharpen - παρακαλούμε επικοινωνήστε με την Ομάδα Φροντίδας μας μέσω της υπάρχουσας περίπτωσης υποστήριξής σας.
Αυτόματη μετάφραση από την επίσημη ενημέρωση του συμβάντος.
We are aware of an incident preventing outbound calls from completeting in Sharpen Q. We are working with all avaiilable resources to resolve.
Outbound Dialing in SharpenQ has been resolved and we are continuing to monitor for stability.
This incident has been resolved.
Our engineering teams are aware of an infrastructure issue that is impacting call routing, WebRTC connectivity for users, and various page navigation efforts (Insights, Admin pages, Logic configuration editors, etc). We have pin-pointed the fault zone and have all necessary resources engaged to restore services as quickly as possible.
Our engineering teams are actively working through corrective actions, and further status updates will be provided as a resolution timeline is established.
The corrective action has been deployed in our infrastructure, and all identified service fault-zones have been addressed. Insights reporting, call routing, and WebRTC connection are now responsive and behaving in the expected fashion. We're actively monitoring with our engineering team. If you're continuing to see impact, please reach out to the Care Team via your existing support case!
Validation testing performed across the application suite has confirmed recovery to all impacted systems. In the event you are still encountering lingering effects tied to interaction routing, WebRTC connectivity problems, or an inability to access a specific configuration page within the app - please reach out to our Care Team for further assistance!
We're currently investigating an issue for IZ1 customers, where Inbound routes pointed to a Classic Logic configuration are experiencing unexpected disconnects. We're actively able to reproduce the problem and are working to restore service as soon as possible.
An attempted remediation effort was made to address calls dropping in Classic Logic. While that did have the expected outcome, a second new set of symptoms were introduced. This includes delays in calls routing to agents in SharpenQ, and delays when agents attempt to connect to the interaction. Our teams are continuing to investigate with all available resources.
We've identified the underlying issue and are working with our internal teams to get a fix deployed across our Production environment. We're targeting a comprehensive roll-out for our infrastructure in the next 30 minutes. Further updates will follow as soon as corrective actions have been fully implemented.
The corrective changes have been made across our infrastructure, and we're actively monitoring the system/running health-checks. The expectation is that call routing through Classic Logic, call audio/quality, and interaction handling behavior should be considerably improved for impacted users.
Service health-checks and sustained monitoring of our call routing systems have shown recovery to the platform. Validation testing across Classic Logic configurations or inbound routes that send calls directly to queue are routing in a timely fashion. In the event you have users still encountering connection issues, please reach out to the Care Team with examples of the misbehavior.
We are currently investigating reports of degraded call quality affecting some customers, intermittently. The following behaviors have been reported beginning around 12:00 PM ET: - Delays (up to 25+ seconds) between a call connecting and the interaction appearing in the agent interface - Dead air on inbound calls prior to ringing and after agent answer - Calls populating in queue but experiencing delays before being assigned to an agent - Absence of hold music for callers waiting in queue - Inability to place calls on hold Our team is actively investigating the cause. We will provide an update as soon as additional information is available. Affected: Voice interactions
We are continuing to investigate the issue and have all necessary resources engage to resolve the issue as soon as possible.
Thank you for your continued patience. We are continuing to investigate the issue with all necessary resources involved. The problem is localized, but thus far our remediation efforts have not sufficiently remediated impact. We are continuing to explore all options to resolve the issue.
We are continuing to investigate this issue.
Thank you for your continued patience. We continue to aggressively address the intermittent service degradation impacting call connection times (agent-caller delay) and interface responsiveness. We will continue to pursue resolution as soon as possible.
We are continuing to investigate the issue. We're receiving additional reports of some users experiencing intermittent 504 Gateway timeout errors. We are including this symptom in our investigation as it likely shares the same root cause.
We are continuing to investigate this issue. We have introduced 1 fix and are working on another which we expect will improve performance.
Our initial fix is showing significant effectiveness, drastically improving the timeliness of Sharpen Q voice interaction related interactions. Call delivery, answer times, and interaction call controls should mostly be restored at this point. We've identified an additional fix which will bring further improvements shortly.
We are continuing working through our second fix and will update this page when it is complete.
We are continuing to monitor and test to validate improvement following our recent 2 fixes.
Data, Monitoring, and testing has confirmed recent fixes have resolved this issue.
This issue was resolved at 9:25 PM ET 2026-02-19. Our status page mechanism encountered an issue at the time of us marking resolution. This is retroactive conclusion of the incident.
Users of the Sharpen platform in IZ0 are likely experiencing a variety of symptoms mostly impacting Sharpen Q and Insights. Our teams are engaged working to resolve the issue as soon as possible.
Sharpen Q has been restored. We are now working on restoring Insights defects and expect recovery shortly.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We are currently investigating a failure to load Insights Dashboards
A fix has been implemented and we are monitoring the results. A refresh of the Insights dashboard will be necessary to restore functionality.
This incident has been resolved.
We are currently aware of the following symptoms in our IZ0 isolation zone - Inbound voice calls delivered through Logic+ result in dead air - Inbound emails do not enter queue In our IZ1 isolation zone Inbound emails do not enter queue We have all necessary resources engaged and are working to restore service as soon as possible.
We have identified the fault for both symptoms and are working through remediation steps now. We are targeting resolution within the next 30 minutes.
Inbound voice through Logic+ in IZ0 is now resolved. We are continuing working through the fix for inbound email failure.
We are continuing our efforts to resolve the inbound email fault. The fault is still properly understood, and we continue working through diligent corrective actions to restore service as soon as possible. Emails which have not been delivered during this impact will not be lost, and we expect them to queue up when this fault is resolved.
We are continuing working on a fix for the email issue and have all necessary resources involved to resolve as soon as possible. Thank you for your patience.
We have identified a new option for recovery and are working through implementation steps now.
We have attempted a fix, which yielded positive progress, but we are still working on a complete fix. Thank you for your patience as we continue our investigation.
We have identified a new path for resolution and are orchestrating a fix now.
We are continuing our investigation. The root cause remains understood, but our attempted changes, allowing the proper communication between our Email service and Sharpen Q, have not yet allowed for the services to properly regain expected communication. We will continue to work diligently until this is resolved.
We have engaged additional resources from our Cloud Service provider to accelerate resolution.
We have an alternative fix ready for deployment and are initiating shortly.
We have identified a new path for resolution and are working to build a new fix. We are targeting a fix within the hour.
We are continuing to work on a fix for this issue.
We have introduced a fix and are seeing some recovery. We are validating expected email functionality now.
We have achieved partial recovery. If emails are forwarded to the Sharpen queue email address, functionality is restored. However, if the sharpen queue email address is included within a distribution group, you are likely still seeing failures. This symptom was confirmed using Microsoft Exchange distribution groups. If you have the ability to forward directly to email addresses, you will experience recovery. The team continues to work toward a full fix regardless.
The issue has been identified and a fix is being implemented.
A fix has been implemented to our IZ0 isolation zone. We are working on propagating it to our IZ1 isolation zone now.
This incident has been resolved. Both IZ0 and IZ1 are working as expected.
Starting at approximately 8PM Eastern, users in the Queue application are experiencing a "Could not place call" error in the UI. We're aware of the problem and actively engaged with our engineering teams to restore functionality as quickly as possible. Further updates will be provided as soon as we have recovery actions identified.
Outbound calling was restored at 9:26PM Eastern, and all monitoring/validation testing since that time has consistently shown recovered call processing behavior. In the event you're still experiencing an issue, please reach out to the Care Team!
As part of the wide-ranging impact that various technical vendors are experiencing (as part of this Amazon Web Services disruption), the Sharpen team is tracking a variety of issues across our platform that involve: - Delayed interaction routing/connectivity behavior - Inability to place outbound calls - Web Chat functionality inconsistencies - SharpenDB querying or data-return issues - Missing or unresponsive interaction control buttons in the Queue interface Our teams are actively working to update the application, in order to address service dependencies or availability issues throughout the platform. We'll continue to update this tracking item as more zones of impact are identified.
We are continuing our efforts to work through these issues - focusing on ways to avoid impacted AWS end-points/services. As certain features are restored to an operational status, we'll continue to update this tracking item.
This incident has been resolved as of 6:53 PM Eastern time upon the conclusion of the AWS incident. Our teams continue to monitor for any additional impact.
Due to an outage with our cloud service provider, the SharpenDB end-point is experiencing degraded availability and is not able to consistently return data. Navigating to the Sharpen Admin --> config tab --> SharpenDB container may show a "Zero Tables Found" error, and various DB lookups (initiated through Classic Logic, Logic+, or through Query API) may not return any results. We've engaged with our cloud service provider and are working on restoring functionality for SharpenDB as quickly as possible. Updates will follow as additional details or restoration timelines are identified.
Our cloud service provider (AWS) has isolated the source of the network connectivity/end-point misbehavior, and are actively working towards resolution. We don't yet have a recovery timeline to share, but will provide further status updates when stability returns to effected services.
Our Development Team has performed an update to circumvent some of the dependencies/communication issues being experienced with our cloud services provider (AWS). We're performing validation testing for SharpenDB's access and querying capabilities, and are seeing a restoration in availability for that portion of the platform. Please test at your discretion, as we continue to make updates and improve platform stability.
This unique tracking items is being closed out, in preference of centrally tracking known areas of impact through a general tracking item (that ties back to the identified service interruption with cloud service provider AWS). Further updates on SharpenDB performance, API end-point engagement, and any other newly identified fault-zones will be communicated using our general tracking item hosted on the Status Page.
Due to a outage with a cloud service provider, the Sharpen Service desk is experiencing reachability issues. The SharpenCare team is still reachable via phone or chat, however web tickets cannot be viewed or submitted. We are monitoring updates from our service provider for resolution.
This incident has been resolved with the resolution on the AWS outage at 6:53 PM Eastern Time. Our team has validated the restoration of this service.
We are currently investigating an issue where CX CURL requests in Logic Classic result in dead air.
We are still investigating the issue where VCX/CX routes in Logic Classic result in dead air. Our engineering team is continuing to work toward identifying the root cause.
We are still investigating the issue, and our teams are fully swarmed working toward a resolution as quickly as possible.
Initial corrective actions have been taken, and we've seen our CX/VCX hosts return to a healthy status. Our focus has shifted towards restoring functionality to the portion of the system that allows Inbound Routing or timed project triggers to engage with a deployed CX/VCX project. Additional updates will be provided as we make progress towards comprehensive recovery.
To clarify our last update: while CX/VCX hosts are reporting healthy, the outage is not fully resolved. Inbound Routing and timed project triggers that connect to deployed CX/VCX projects are still impacted. Our team is actively working to restore this functionality, and we’ll continue to share updates as progress is made.
We've identified the underlying issue that is preventing different portions of the system (Classic logic, Inbound Routes, etc) from being able to engage with CX/VCX projects. We have been able to partially restore service to some applications, and are continuing to work towards comprehensive recovery. NOTE: Some projects are operational at this time while others CX/VCX deployments continue to see impact. We're aware of this behavior and are continuing efforts to correct all projects.
We appreciate your continued patience. Our team remains engaged and is working to resolve the impact. For many customers we have implemented workarounds or alternative solutions that have restored customer experiences. We are still pursuing all options of service restoration. If you are impacted and have unique needs to overcome this disruption, please reach out to us at [email protected]
We're continuing our efforts to resolve CX/VCX routing issues across the IZ1 infrastructure. While we've had partial success, certain projects remain unreachable and currently result in a dead-air experience for customers. Mitigation efforts have been underway with the Care Team, involving updating routing conditions (away from CX/VCX) to minimize impact. In addition to Inbound routing or Classic Logic call-flows, this misbehavior extends to in-queue operations like timed-bots, "Max Wait", and "Exit No Agents" conditions that point to a Callback/Voicemail related VCX customization. The short-term recommendation is to disable timed-bots, and instead leverage the in-queue callback or voicemail offering. Further communication will follow, once we've established recovery for CX/VCX project usage. In the event assistance is required to update queue configurations, please reach out to the Care Team via [email protected]
At this stage we have restored service to the majority of impacted flows and are working directly with customers who are experiencing any continued impact. If you are experiencing a problem with IVR routing or in queue messaging regarding the use of VCX/CX, please reach out to us at support.sharpencx.com so we can assist.
Starting at approximately 9:30PM Eastern, calls routing through our IZ0 infrastructure that engage a transfer action (whether that be a queue-based "Max Wait" or "Exit No Agents" condition or an agent-initiated transfer to a different queue) are experiencing an unexpected termination. Test calls hitting one of these transfer conditions will hear a series of beeps, followed by the unexpected disconnect.
We've identified a routing issue within our infrastructure that's uniquely tied to transfer functionality (both queue and agent-initiated) and are working towards a fix. All available resources have been engaged at this time, and we're actively working towards recovery as quickly as possible. Further updates will be made to the status page when we've reached a point of recovery.
Our teams are currently working towards rolling out the hot-fix/update across Sharpen's infrastructure. As soon as that effort has been completed, we'll initiate validation testing and confirm recovery of the system. Further updates will be posted to the status page as soon as that task finishes.
The corrective hot-fix has been deployed throughout our infrastructure, and initial validation testing has confirmed a restoration in queue-based and agent-initiated transfer functionality. Please verify with end-users if they're seeing a restoration in functionality. If ongoing impact is experienced, please reach out to the Care Team for further troubleshooting/analysis - with example users/extension info/QCMIDs that can be reviewed.
Continued validation testing and monitoring efforts have shown comprehensive recovery across the Sharpen Q system. This incident is being transitioned into a monitoring status. In the event that new misbehaviors or lingering symtoms/issues are encountered, please reach out to the Care Team.
Inbound numbers that route through a Logic+ are resulting in dead-air. We are currently investigating and working towards a resolution
We identified the root cause impacting inbound calls routed through Logic+ and have applied corrective measures. Calls are now completing successfully. We will continue to closely monitor the service to ensure full stability.
The issue with inbound calls routed through Logic+ has been fully resolved. All services are operating normally.
Starting shortly after 09:00AM Eastern, Inbound numbers that route though a Logic+ configuration are intermittently resulting in a dead-air or delayed playback misbehavior.
We're aware of an ongoing/active issue, that is causing Inbound calls to experience dead-air or extremely delayed prompt playback - specifically when the number processes through a Logic+ configuration. We're investigating some infrastructure alerts that were triggered, and have all necessary team members engaged to troubleshoot the issue. We'll post further status updates as additional details become available.
We've isolated the underlying issue within our infrastructure, and have taken initial steps to reroute traffic away from the misbehaving pathway causing the intermittent disconnects, delayed prompt playback, or dead-air. Our internal teams are actively preparing a code-change to completely address the problem, and we expect to have that pushed out to our Production systems in the next few minutes.
Our IZ1 infrastructure update has been deployed to the Production system, and initial validation efforts have shown full recovery to previously impacted call-flows routing though Logic+. In the event that you're still experiencing symptoms tied to dead-air, latent prompt playback, or unexpected disconnects through a Logic+ bot, please reach out to the Care Team with examples of the misbehavior and we'll troubleshoot further!
We are currently investigating an issue impacting the inability to place outbound calls in our IZ1 infrastructure.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We are aware of a potential service disruption impacting our IZ1 Infrastructure that may result in inbound calls not reaching queues. We have all necessary resources engaged to restore service. Additional details will be provided soon.
We have identified the cause and are working on a corrective fix.
We have corrected the issue impacting IZ1 application availability and are continuing to monitor.
This incident has been resolved. Any agents experiencing lingering impact may need to log out of app.iz1.sharpen.cx and log back in.