Störungsverlauf von Sisense

Unfortunately, an ongoing incident with an upstream provider (Red Hat / Quay.io) has degraded again, affecting container registry push/pull operations (https://status.redhat.com/incidents/jn6r256zc62c). As a precaution, customers may experience intermittent slowness or delays with some Sisense services that depend on this provider. We are actively monitoring the situation and assessing any potential impact to our platform.

A fix has been implemented and we are monitoring the results.

We are continuing to monitor for any further issues.

This incident has been resolved.

We are aware of an ongoing incident with an upstream provider (Red Hat / Quay.io) that is reporting degraded performance on container registry push/pull operations https://status.redhat.com/incidents/jn6r256zc62c. As a precaution, customers may experience intermittent slowness or delays with some Sisense services that depend on this provider. We are actively monitoring the situation and assessing any potential impact to our platform.

RedHat have restored manifest pushes and are monitoring the environment for stability.

This incident has been resolved.

Please be aware that on May 11, starting at 10:30 AM EEST, we experienced a service outage affecting the AI Assistant. During this window, users across all accounts may have experienced non-responsive behavior when attempting to create charts or submit general queries to the Assistant.

Following a technical investigation, the issue was traced back to the Shared Gateway Service, which was preventing requests from reaching the AI processing engine.

Our engineering team applied mitigation steps and re-applied configuration to clear the gateway bottleneck and restore the flow of traffic.

2:00 PM EEST The fix was verified, and the service was confirmed as fully operational. Root Cause: The outage was corrupted configuration in the AI Shared Gateway Service, affecting its ability to handle Assistant-specific traffic. We sincerely apologize for the impact this had on your workflow. We are currently reviewing our gateway capacity and alerting thresholds to prevent a recurrence of this specific issue.

Please be aware that AWS (Amazon Web Services) is currently reporting significant hardware and power issues in the US-EAST-1 (N. Virginia) region. For the most up-to-date information on their infrastructure recovery, please follow the official AWS Health Dashboard https://health.aws.amazon.com/health/status . Unfortunately, our Sisense servers are also severely impacted by this event. As a result, users may experience total loss of access, dashboard loading failures, or significant latency. Our engineering team is actively monitoring the AWS recovery process and is working to restore Sisense services as quickly as hardware availability allows. We apologize for this disruption and will provide further updates here as soon as we have more information.

The issue has been identified and a fix is being implemented.

A fix has been implemented and we are monitoring the results.

We no longer see the consequences of the AWS incident after they updated the AWS incident page with "Resolved" status. For the latest on the underlying AWS infrastructure, please refer to: https://health.aws.amazon.com/health/status

AWS has officially resolved the thermal and power event in the US-EAST-1 region. All underlying infrastructure supporting our Sisense servers has been stabilized and is performing at pre-event levels. Our engineering team has verified that our services are fully operational. We will continue to perform routine checks to ensure long-term stability, but we consider this incident fully resolved. Thank you for your patience while we navigated this external disruption. https://health.aws.amazon.com/health/status

We’re currently observing a partial outage affecting Sisense managed instances. Our team is actively investigating the issue and working to identify the root cause. We’ll share updates as soon as we have more information. Thank you for your patience.

The issue has been identified and we are actively working on a fix.

A fix has been implemented and we are monitoring the results.

The incident has been resolved.

We’ve identified an issue affecting specific EKS clusters that may impact the build process. Our team is actively working on a resolution.

The issue has been resolved, and we are continuing to monitor to ensure stability.

This incident has been resolved.

Due to issues with the SSL certificate, some clients might experience an error while trying to log in. Our engineering team is aware of the situation and is investigating the matter in order to come up with a workaround or a fix.

We are continuing to investigate this issue.

A fix was deployed, and the engineering team is continuously monitoring the state of the platform to make sure that the performance is back to normal.

The platform's performance has been restored to the normal level.

Cloudflare is currently experiencing an issue. As a result, some requests may fail or errors may be displayed. We are monitoring the situation and will provide updates as it progresses.

Cloudflare has implemented a fix for the issue, and we are currently monitoring the results.

This incident has been resolved.

We are aware of an incorrect banner message indicating that your trial will expire in 7 days. Please disregard this notification. Our Engineering team is actively working on a fix.

This incident has been resolved.

We are aware of the ongoing widespread disruption affecting the Cloudflare Global Network. We are actively monitoring this situation to assess any potential impact on our services. At this time, we have not detected any major service degradation or significant impact on our platform, however customer that use own domain through CloudFlare service might experience issues with access and 500 errors. We will continue to monitor the Cloudflare incident closely and provide updates as the situation evolves.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

We are currently experiencing service degradation that affects the creation of new environments and the upgrade of existing environments.

Status Summary: No Impact to Running Environments: Your currently deployed applications and services are operational and unaffected by this issue. Affected Operations: Issues with creating new environments and initiating upgrades may fail due to dependency errors. Root Cause: This degradation is a result of an upstream Azure platform problem, specifically involving Azure Front Door. This issue is preventing the successful download of required Helm binaries, as tracked in Helm GitHub Issue #13303. Reference the Upstream Status: For more information on the cause, please refer to the Azure Status Page. Mitigation: Our team is actively monitoring the Azure status and will automatically resume all affected operations as soon as the upstream service is restored. We will provide an update when normal service is resumed.

This incident has been resolved.

We are currently aware of ongoing AWS issues impacting the us-east-1 region. As a result, some of our managed services in this region may experience degraded performance or intermittent problems. Our team is actively monitoring the situation and will provide updates as more information becomes available.

We can confirm global services and features that rely on us-east-1 have recovered. We’ll keep monitoring closely and share more updates as needed.

Incident was resolved - AWS experienced an outage in the us-east-1 region earlier today, which caused minor issues with some ALBs and VPN connections for customers hosted there. All services are now fully operational.

Our live chat service for Siense CDT is currently unavailable. Please reach us at [email protected] for assistance. We apologize for any inconvenience and will restore chat functionality as soon as possible.

The CDT Live Chat Support has been restored. The live chat service is now fully operational. You can once again reach our support team directly through live chat or via email at [email protected]. Thank you for your patience while we resolved this issue.

This incident has been resolved.

We’re experiencing an issue with our ticketing system that may cause slower response times than usual. Our Support team is working to fix this as quickly as possible.

The issue with the ticketing system that was causing slower response times has been resolved. Support response times have returned to normal, and no further impact is expected.

This incident has been resolved.

We are currently experiencing degraded availability of certain AI-powered features, including Natural Language Queries (NLQ), in the us-east-1 region. This is related to ongoing network performance issues reported by our security provider (Cloudflare), as noted on their status page: https://www.cloudflarestatus.com/incidents/d9n3g1vnxdd2 All core Sisense services remain fully operational. Only AI-related functionality is impacted, and customers may encounter intermittent errors or unavailability when using these features. Our engineering team is actively monitoring the situation and working with Cloudflare to track their progress toward resolution. We will continue to provide updates and confirm once AI feature availability has been fully restored.

We are continuing to work on a fix for this issue.

This incident has been resolved.

Some customers experienced intermittent issues when accessing Sisense APIs and the Support Portal. The disruption was caused by a dynamic adjustment within our traffic bot management rules, which automatically adapt based on observed traffic patterns. In this instance, the adjustment inadvertently impacted certain legitimate API requests. Once the issue was identified, our security engineering team promptly reverted the adjustment, and services were quickly restored. All environments are fully operational, and we are actively monitoring system performance to ensure continued stability. We are working closely with our traffic management provider to fine-tune protections so that future adjustments do not affect customer traffic. We sincerely apologize for the disruption and greatly appreciate your patience and understanding as we resolved the issue.

We are currently investigating an issue affecting some of our embedded capabilities via iframe, where widgets may fail to load and appear blank on limited number of environment. Our engineering team is actively working to identify the root cause and implement a fix. At this time, there is no estimated time of resolution (ETA). Thank you for your patience and understanding.

The issue has been identified and fix is going to be rollout soon.

Patch is rolling out to all impacted instances.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

Some CDT clients may experience a temporary interuption of the SQL query processing. The engineering team was notified and is already working on identifying the cause of the issue to apply an appropriate fix and restore SQL query processing.

The engineering team was able to restore the SQL query processing service functionality. The fix was applied, we continue to actively monitor the situation and performance.

The issue was resolved, and the platform performance is restored to normal condition

The admin section is currently inaccessible in a limited number of lower environments. Our team is actively working on a resolution and expects to have it fixed by the end of the day.

This incident has been resolved.

Some clients might experience issues with accessing the password-protected shared dashboard links. The engineering team has identified the cause of the issue and is working to develop a fix. Please feel free to contact [email protected] for any additional questions

A fix to address the access issues for shared dashboard links was deployed. The engineering team continues to monitor the platform to make sure that all services are operational. For any assistance please feel free to reach out to [email protected]

This incident has been resolved.