52 Mode Analytics incidents · Mart 2024 — official updates, affected components, duration and resolution details.
09:00 ve 11:30 UTC arasında üçüncü bir parti servis sağlayıcı ile ilgili sorunları gözlemledik. Etki, sonuçları, şema tarayıcılarında masaların otomatikleştirilmesi ve rapor duplikasyon gibi eylemlerin bildirimleri için bildirimlerin (report duplication hala arka planda tamamlanacaktı). Bu konu şimdi çözüldü. Lütfen herhangi bir sorunuz varsa destek için dışarı çık.
Resmî olay güncellemesinden otomatik olarak çevrilmiştir.
Yaklaşık 3:55 AM ve 4:15 AM PDT arasında, kullanıcılar bize-batı-2 (Oregon) bölgesinde bir AWS ağ konusu nedeniyle hizmet süreleri yaşayabilir. AWS geçici olarak bölgeden gelen internet bağlantısını kesintiye uğrattı, dış hizmetlerle iletişim kurma yeteneğimizi etkiledi. Araştırmamıza dayanarak, konu altyapımıza dışsaldı ve tüm hizmetler, olayın AWS’nin kararını takip eden tamamen restore edilmiştir.
Resmî olay güncellemesinden otomatik olarak çevrilmiştir.
We are currently investigating an issue where users may experience errors in authenticating to Mode through Microsoft SSO. Please reach out to Mode Support ([email protected]) if you or any of your users are experiencing this issue.
The issue has been identified and a fix is being implemented.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
As of March 7th there have been failures when syncing new data to the Discovery Database. As a result the latest data seen in the Discovery Database will be for March 6th as subsequent syncs have failed. Our engineers are investigating to identify the root cause of the issue and we will share updates accordingly.
Our engineers have identified the issue and are currently applying a fix for the syncing pipeline of the Discovery Database.
The freshness of data has been validated and we are monitoring the Discovery Database pipeline through the next daily sync.
This incident has been resolved.
Intercom messenger is experiencing an outage and users are currently unable to load the in-app chat widget in Mode. If you need to contact Support during this time, please email us at: [email protected], or by reaching out to your Account Team to raise any priority issues, and they will alert us directly. We will resume operations for existing and new tickets in Intercom as soon as possible. To stay up-to-date: https://www.intercomstatus.com/
Intercom messenger has fully recovered, and the Support team has resumed normal operations.
Intercom messenger is experiencing an outage and the Support Team is intermittently unable to access Intercom at this time. Please reach out to your Account Team to raise any priority issues, and they will alert us directly. We will resume operations for existing and new tickets in Intercom as soon as possible. To stay up-to-date: https://www.intercomstatus.com/
Intercom messenger has fully recovered, and the Support team has resumed normal operations.
We have identified an internal issue affecting the Discovery Database sync, resulting in a delay of up to 24 hours. Some customers may see missing or outdated data during this period. We are actively working to resolve it and will provide an update once resolved.
The Discovery Database sync delay has been resolved. The sync completed successfully and data is now up to date.
At approximately 12:00 AM Eastern on December 11, we received the first report that Python visuals were not appearing in reports. While the visuals displayed correctly in the notebook, the corresponding report elements appeared empty. Our engineering team identified the cause of the issue and resolved it. All notebook visuals should now render as expected in reports. If you continue to experience any issues, please reach out to Support.
GitHub is reporting failures and degraded availability for Git operations, which may interrupt Mode's Github integration. Some syncs may fail, stall, or not reflect recent changes. We're monitoring GH's incident and will provide updates as it progresses. You can also follow the GitHub updates at https://www.githubstatus.com/
GitHub has reported this incident as resolved and all systems are operational. You can review GitHub updates at https://www.githubstatus.com/
We became aware of an issue affecting scheduled reports executions beginning around 11:00 PM EST on November 17th. The issue affected hourly, 15-minute, and weekly scheduled jobs. Monthly and 30-minute schedules were not impacted. Engineering has resolved the issue, and normal scheduling has resumed. We are continuing to monitor to ensure full stability. Further updates will be provided. Edit: Scheduled executions began failing at 2:00 PM EST on November 17 and were fully resolved by 6:30 AM EST on November 18. As expected after recovery, some delayed or duplicate scheduled runs were triggered as jobs caught up. Normal scheduling has now resumed as of Nov 20.
We're investigating an issue causing reports that use Notebook generated datasets to fail. Affected reports may show missing datasets in the Data pane, charts that infinitely load, or errors stating that the underlying dataset was deleted. Our team has identified the cause, and a fix is currently being implemented to restore functionality.
A fix has been deployed, and functionality for reports using Notebook generated datasets has been restored. If you continue to experience any issues, please refresh your report and try again. If you still encounter any issues, please reach out to our support team for assistance.
We're aware of a widespread AWS outage in the US-EAST-1 region that may have caused errors and delays across Mode. The issue has since been resolved by AWS, and Mode services are operating normally. We will continue to monitor the situation. You can also follow the AWS updates at https://health.aws.amazon.com/health/status that began on Oct 20 12:11 AM PDT
AWS had another incident ~30 mins ago. We are monitoring the situation closely. Users may experience degraded performance and/or errors.
The earlier AWS outage impacting Mode services has been resolved. All systems are operational, though some background jobs and scheduled runs may take additional time to clear as backlogs process. Please reach out to Support if you have any questions.
Intercom messenger is experiencing degraded performance and the Support Team is intermittently unable to access Intercom at this time. Please reach out to your Account Team to raise any priority issues, and they will alert us directly. We will resume operations for existing and new tickets in Intercom as soon as possible.
We are continuing to investigate this issue.
Intercom access has recovered, and the Support team has resumed normal operations.
We have observed issues with a third party service provider beginning around 16:00 UTC. The impact would have contributed to Notebook cells failing to render their results, auto pinning of tables in the schema browser, and notifications for actions such as report duplication (report duplication will still complete in the background). We are investigating the issue with our third party provider and will share updates as they become available.
The issue is resolved. Please reach out to Support if you have any questions.
This morning's daily Discovery Database update was not successful. We are investigating and will provide additional information as soon as possible. Previous data should still be available.
We have applied a fix. We will be able to manually trigger the DDB sync in about an hour.
Our team has started the Discovery Database sync. It should complete within an hour.
The issue is resolved. Please reach out to Support if you have any questions.
Emails were blocked by ~1.5 hours. Emails were blocked from 4:22-5:57PM Pacific. The issue is resolved and emails have resumed.
An AWS partial outage is impacting the Audit Logs API endpoint. We are aware of and monitoring the situation. You can follow the AWS updates at https://health.aws.amazon.com/health/status under the "Increased Queue Processing Time" incident that began on June 13th, at 10:12 AM PDT.
We are observing expected response codes and response times for the Audit Logs API endpoint. If you experience any other disruptions please feel free to reach out to Mode Support via in-app chat or by email ([email protected]). Thank you.
The Support Team is intermittently unable to access Intercom at this time. Please reach out to your Account Team to raise any priority issues, and they will alert us directly. We will resume operations for existing and new tickets in Intercom as soon as possible.
Intercom access is recovering and the Support team will resume operations as normal.
The dataset failed to load. When attempting to use the reusable dataset, it results in a blank screen. The issue has been identified, and our team is actively working on the fix.
A fix has been successfully deployed, and we are actively monitoring.
This incident has been resolved.
We are aware that some scheduled emails over the weekend may be delayed. We have identified the cause and are working to resolve the issue.
A fix for the issue has been implemented by our engineers. Scheduled emails that were not previously received should be arriving in the respective inboxes.
The incident has been resolved. Please reach out to Support, via chat or by emailing [email protected], if you are still encountering any issues in receiving scheduled emails. Thank you for your patience as we worked to resolve this.