58 Fullstory incidents · Νοέμβριος 2023 — official updates, affected components, duration and resolution details.
Στις 23 Σεπτεμβρίου έγινε μια αλλαγή στον τρόπο με τον οποίο επιλύονται οι διευθύνσεις URL για δεσμευμένα περιουσιακά στοιχεία· αυτό οδήγησε ακούσια σε διακοπή της αναπαραγωγής για συνεδρίες στο Data Center της ΕΕ1. Αυτό διορθώθηκε επιστρέφοντας πίσω αυτή την αλλαγή στις 9:45 πμ UTC.
Αυτόματη μετάφραση από την επίσημη ενημέρωση του συμβάντος.
Εφαρμόσαμε μια ρύθμιση για ένα θέμα που προκάλεσε καθυστερήσεις σε κάποια διαθεσιμότητα δεδομένων συνεδρίας, συνέβαλε στην υποβαθμισμένη απόδοση αναζήτησης και ταμπλό, και ενδεχομένως καθυστερημένες ειδοποιήσεις ειδοποίησης στο Fullstory της εφαρμογής. Παρακολουθούμε ενεργά τα αποτελέσματα για να διασφαλίσουμε πλήρη σταθερότητα.
Συνεχίζουμε να παρακολουθούμε για τυχόν περαιτέρω ζητήματα.
Αυτό το περιστατικό έχει πλέον επιλυθεί. Εάν αντιμετωπίζετε οποιαδήποτε θέματα μετά τις 6:15 μμ (ET), παρακαλούμε στείλτε μας email στο [email protected].
Αυτόματη μετάφραση από την επίσημη ενημέρωση του συμβάντος.
Γνωρίζουμε ένα θέμα που αναφέρθηκε από τον πάροχο υποδομής cloud (Google Cloud Platform) που επηρεάζει πολλαπλές υπηρεσίες Fullstory στο κέντρο δεδομένων NA1. Οι επηρεαζόμενες υπηρεσίες περιλαμβάνουν τη σύλληψη δεδομένων (web and native mobile), την εφαρμογή Fullstory web, API, Webhooks & Alerts, οπουδήποτε: Αποθήκη, Οδηγοί και Έρευνες, StoryAI, και Workforce. Η λήψη δεδομένων μπορεί να διακοπεί κατά τη διάρκεια αυτής της περιόδου. Είμαστε ενεργά απασχολημένοι με τον πάροχο μας, παρακολουθούμε στενά την κατάσταση και θα παρέχουμε ενημερώσεις καθώς θα είναι διαθέσιμες.
Βλέπουμε ανάκαμψη για το θέμα που αναφέρθηκε από τον πάροχο υποδομής cloud μας (Google Cloud Platform) και παρακολουθούμε ενεργά για να διασφαλίσουμε την πλήρη σταθερότητα που κινείται προς τα εμπρός. Μπορείτε επίσης να ακολουθήσετε την κατάσταση του παρόχου απευθείας εδώ: https://status.cloud.google.com/incidents/J5ia5t9p3g9Q5Wi7r8Ev
Το θέμα που αναφέρθηκε από τον πάροχο υποδομής cloud μας (Google Cloud Platform) έχει πλέον επιλυθεί και οι υπηρεσίες Fullstory λειτουργούν πλήρως από την 1η Σεπτεμβρίου 2026 στις 4:32 μμ (ET). Όλα τα υποβληθέντα δεδομένα υποβάλλονται σε επεξεργασία και θα είναι διαθέσιμα σύντομα. Εάν έχετε οποιεσδήποτε ερωτήσεις ή αντιμετωπίζετε οποιαδήποτε συνεχή ζητήματα, παρακαλούμε επικοινωνήστε με το [email protected].
Αυτόματη μετάφραση από την επίσημη ενημέρωση του συμβάντος.
Στις 10:40AM ET, οι χρήστες Silver Schema στο NA1 άρχισαν να βιώνουν καθυστερήσεις συγχρονισμού αποθήκης με όλους τους χρήστες που επηρεάστηκαν γύρω στις 11:10am ET. Ακόμα ερευνούμε αυτό το θέμα.
Έχουμε εφαρμόσει μια ρύθμιση για το θέμα που ξεκίνησε γύρω στις 10:40AM ET, η οποία προκαλούσε καθυστερήσεις για τους πελάτες της αποθήκης Silver Schema στην περιοχή NA1. Το σύστημα πλησιάζει. Η ομάδα μας λαμβάνει μέτρα για να διασφαλίσει ότι όλα τα δεδομένα υποβάλλονται σε επεξεργασία με ακρίβεια, και θα δημοσιεύσουμε άλλη ενημέρωση μόλις όλη η επεξεργασία επανέλθει στο κανονικό.
Το θέμα που προκάλεσε καθυστερήσεις στους πελάτες της αποθήκης Silver Schema στην περιοχή NA1 έχει επιλυθεί και η υπηρεσία είναι πίσω στην πλήρη λειτουργία από τις 18 Αυγούστου 2026 στις 4:23 μμ ET. Εάν έχετε απορίες ή αντιμετωπίζετε οποιαδήποτε συνεχή ζητήματα, παρακαλώ μην διστάσετε να επικοινωνήσετε με την ομάδα μας στο [email protected].
Αυτόματη μετάφραση από την επίσημη ενημέρωση του συμβάντος.
From 7:58pm UTC on the 6th of August, users in NA1 & EU1 organisations with Guides & Surveys enabled were unable to create or edit tours and smart tips. There was also some disruption of end-user content delivery, though this was mitigated by edge caching serving cached content. The issue was resolved by 8:30am UTC on the 7th of August.
Γνωρίζουμε ένα θέμα με το BigQuery που ξεκίνησε γύρω στις 23 Ιουλίου 2026 στις 7:45 π.μ. (ET) το οποίο έχει αντίκτυπο στο Fullstory Anywhere: Renection connections utilizing BigQuery. Παρακολουθούμε στενά την κατάσταση και θα παρέχουμε ενημερώσεις καθώς θα είναι διαθέσιμες.
Για να αποσαφηνιστεί το πεδίο εφαρμογής αυτού του περιστατικού: Το Fullstory χρησιμοποιεί το BigQuery εσωτερικά για να μετατρέψει δεδομένα για όλους τους πελάτες Anywhere: Αποθήκη στο κέντρο δεδομένων NA1, ανεξάρτητα από τον τύπο τελικού προορισμού. Ως αποτέλεσμα, αυτή η υποβάθμιση του BigQuery μπορεί να καθυστέρησε τον συγχρονισμό ενός μικρού αριθμού πινάκων για όλους τους πελάτες της Αποθήκης στο NA1 — συμπεριλαμβανομένων των BigQuery, Redshift, Databricks και Snowflake προορισμούς — όχι μόνο οι πελάτες που συγχρονίζονται απευθείας στο BigQuery. Συνεχίζουμε να παρακολουθούμε και θα παρέχουμε μια περαιτέρω ενημέρωση καθώς οι συγχρονισμοί ανακάμπτουν.
Το περιστατικό λύθηκε. Η υποκείμενη υποβάθμιση BigQuery έχει μετριαστεί, και το στρώμα μετασχηματισμού δεδομένων που χρησιμοποιεί το Fullstory για οπουδήποτε: Αποθήκη στο NA1 έχει επιστρέψει στην κανονική λειτουργία. Κάθε συγχρονισμός που καθυστέρησε κατά τη διάρκεια αυτής της περιόδου έχουν πλέον προηγηθεί, και επηρεάζονται πίνακες είναι ενημερωμένοι σε όλους τους προορισμούς (BigQuery, Redshift, Databricks, και Snowflake). Το ζήτημα προέκυψε από περίπου 7:45 π.μ. ET έως 11:20 π.μ. ET στις 23 Ιουλίου 2026. Ζητάμε συγγνώμη για την αναστάτωση και σας ευχαριστούμε για την υπομονή σας. Στείλε μας email στο [email protected] αν συνεχίσεις να βλέπεις θέματα.
Αυτόματη μετάφραση από την επίσημη ενημέρωση του συμβάντος.
Το μοντέλο πίσω από τα χαρακτηριστικά του StoryAI που είναι άμεσα ενσωματωμένα μέσα σε πίνακες, τμήματα, και Replay συνεδρία βλέπουν υποβαθμισμένη διαθεσιμότητα λόγω μιας διακοπής Google. Συνεργαζόμαστε στενά με την ομάδα τους για την επίλυση αυτού του ζητήματος. Άλλα χαρακτηριστικά AI μέσα Fullstory δεν επηρεάζονται αυτή τη στιγμή.
Αυτό το περιστατικό έχει επιλυθεί
Αυτόματη μετάφραση από την επίσημη ενημέρωση του συμβάντος.
Between July 6, 2026 at 8:47 AM EDT and July 7, 2026 at 12:00 PM EDT, incorrect filter configurations were intermittently applied to the default canned metrics (Users Today, Sessions Today, Error Rate, and Avg Page Load Time), which could have caused them to display unexpected filter values or report zero results due to no matching data. If you have any questions or are experiencing any continued issues with canned metrics, please reach out to [email protected].
Beginning at approximately 04:00 CEST (22:00 EDT Jun 10), end users on sites where Guides & Surveys is installed may have experienced guides, tours, surveys, and smart tips not displaying correctly, and interaction data was delayed. This only is impacting customers in our NA data center. This issue was caused by resource exhaustion in the service responsible for delivering and recording end-user activity, not in the FullStory application itself. The FullStory app, Guides & Surveys editor, and theme/targeting configuration were not affected. No data was lost. Content delivery may have been delayed, and whether queued interactions are replayed will depend on whether the end user's session re-qualifies against targeting rules. We have identified the root cause of this issue, implemented a fix, and are actively monitoring the service. We will provide a further update once the service has been fully restored.
This incident has been resolved. Guides, tours, surveys, and smart tips are now delivering normally for end users in the NA data center. We apologize for the disruption and are implementing additional safeguards to prevent recurrence. If you are still experiencing issues, please contact support at [email protected].
We are investigating an issue where Metric Alerts are triggering at thresholds that do not match the underlying metric data for customers in both our EU and NA data centers. Customers may be receiving alert notifications that appear to fire incorrectly or more frequently than expected. Our engineering team has been notified and is actively investigating. We will provide an update as soon as we have more information. We apologize for the disruption and appreciate your patience.
As of 9:36 AM ET on June 11, 2026, this issue is resolved. A feature flag introduced as part of our multi-metric support rollout caused alert evaluations to diverge from actual metric values, resulting in Metric Alerts firing incorrectly even when the underlying metric had not breached its threshold. Start time of this issue: Multi-Metric Beta customers: June 4, 2026 at 10:27 AM ET All other customers using Metrics with division: June 10, 2026 at 4:25 PM ET If you continue to receive alerts you believe are incorrect, please reach out to us at [email protected]. We apologize for the disruption.
We are monitoring an issue that began on June 6 at 10:00am ET causing data processing delays in our EU1 data center. New sessions are appearing normally, but analytics for recent sessions and data sent from Fullstory to connected warehouses may be delayed. A fix has been implemented today and we are monitoring the system's recovery. We will provide another update as soon as possible.
This issue is now resolved. If you notice any further delays in analytics indexing or data warehouse syncing reach out to [email protected].
On April 6th, 2026, we introduced a limit on the number of pages that can be recorded in a single session to ensure optimal performance. An unexpected side effect caused our systems to continue collecting excess session data without properly saving it, creating a processing backlog. To resolve this, we quickly paused the limit and significantly increased our system's processing capacity to clear the backlog. Unfortunately, because of the size of the delay, a portion of session data was permanently deleted before it could be successfully archived. Due to ongoing metric degradation as a result of that incident, we experienced a similar build up in our processing backlog on May 18th 2026. While we were able to capture the events in these sessions, the affected sessions will appear in a blank state. This issue impacted a small subset of sessions in some customer accounts in our NA data center from May 7th, 04:00 UTC to May 19th, 07:00 UTC when the issue was ultimately resolved.
On April 6th, 2026, we introduced a limit on the number of pages that can be recorded in a single session to ensure optimal performance. An unexpected side effect caused our systems to continue collecting excess session data without properly saving it, creating a backlog in our database. To resolve this, we quickly paused the limit and significantly increased our system's processing capacity to clear the backlog. Unfortunately, because of the size of the delay, a portion of session data was permanently deleted before it could be successfully archived. While we were able to capture the events in these sessions, the affected sessions will appear in a blank state. This issue impacted a small subset of sessions in some customer accounts in our NA data center from April 6th, 11:00 UTC to April 14th, 14:00 UTC when the issue was ultimately resolved.
Some customers using Fullstory Anywhere: Warehouse may have observed a significant decrease in the number of events being transported to their data warehouse between February 24, 2026 to March 13, 2026. No data was lost during this time and all data has been backfilled to the impacted customers' warehouses. This incident is now resolved.
2026/02/24 Data transport to data warehouse significantly delayed for some customers This postmortem details the customer impact, the root cause of what happened, how we addressed the problem, and how we will prevent it from happening in the future. # Customer Impact Some customers using Fullstory Anywhere: Warehouse may have observed a significant decrease in the number of events being transported to their data warehouse between Feb 24, 2026 to Mar 13, 2026. The impact here is limited to object storage destinations such as Amazon S3, Azure Blob Storage, and Google Cloud Storage. # Root Cause The Fullstory platform has two internal pipelines that generate event data and notify downstream systems. There is a streaming pipeline for real time event processing, and a batch pipeline for everything else. Object storage data destinations receive notifications from the latter whenever it processes session and page entities that are deemed inactive i.e. no longer receiving event data. On Feb 24, 2026 the team made a change to the streaming pipeline to track additional metadata about page entities that were being processed. Unfortunately this change had an unintended effect on the batch pipeline which caused notifications to not be fired upon successful processing for a subset of events. Thus event data that should have been transported to object storage to instead remain buffered indefinitely. # Resolution On Mar 11, 2026 the team investigated the issue and pinpointed it to a code change that was deployed on Feb 24, 2026. A fix was deployed on the same day as the issue was found. From there events from newly processed page entities began flowing to affected customer’s data warehouses again. Once the fix was in place the team focused on impact analysis and remediation. A backfill process was kicked off on Mar 12, 2026 to transport event data that did not originally make it to affected customer’s data warehouses. As of Mar 13, 2026 the backfill is complete and any gaps in event data have been filled. No further action is required from those impacted. # Process Changes and Prevention Our team has completed a thorough review of the incident, fixed the relevant bug, and implemented a backfill process for remediation here. We have also identified several areas of improvement with our metrics and monitoring, as well as system changes to prevent this kind of unintended cross-pipeline impact. We deeply regret this incident and invite any Fullstory customer who was materially affected to contact [[email protected]](mailto:[email protected]). We stand by ready to fully address all of your concerns.
We are currently experiencing delays in data processing in our EU1 data center. New sessions may not appear until several minutes later. Additionally, data passed to connected warehouses from Fullstory will be delayed. We are investigating the issue.
Our team has implemented a fix and the system is now catching up. We are addressing delays that began at the following times: - Analytics: 2026-02-27 20:00 UTC - Warehouse: 2026-02-25 03:00 UTC We are taking steps to ensure all data is processed accurately and will post another update once processing is back to normal.
The delay in processing for new sessions has been resolved and this system is now operating normally. We are continuing to work through the delays affecting data passed to connected warehouses from Fullstory. We will provide another update once all processing is back to normal.
This incident in the EU1 region has been resolved. All data processing delays have been cleared, and all systems are now operating normally.
We are currently monitoring an issue that is causing delays for warehouse customers located in our EU1 region. Customers may experience slower than normal processing times. We have identified the root cause and will continue to monitor the situation. We will provide another update as soon as we have more information.
We have implemented a fix for the issue that began around 18:40 UTC, which was causing delays for our warehouse customers in the EU1 region. The system is now catching up. Our team is taking steps to ensure all data is processed accurately, and we will post another update once all processing is back to normal.
Correction: The issue began at approximately 18:40 ET, not 18:40 UTC as previously stated. We have resolved the issue that was causing delays for our warehouse customers in the EU1 region, and all systems are now fully operational. Our team will continue to ensure all data is processed correctly. We appreciate your patience and apologize for any inconvenience this may have caused.
We experienced a service disruption in our EU1 region that impacted playback functionality from 8:45 AM to 9:37 AM UTC. We apologize for any inconvenience this may have caused.
From 1:04 PM to 1:26 PM EST, Fullstory experienced elevated error rates for session capture and some areas of the app. This also caused slight processing delays that lasted until about 2:00 PM EST. There was minimal to no data loss, and most failed requests were successful on retry.
From January 22, 2026 at 12:54 PM ET through January 23, 2026 at 11:26 AM ET, the Get Session API was unintentionally unavailable. Any requests to Get Session API may have incorrectly responded with HTTP 403 for valid calls. Fullstory customers that attempted to send an API request during the outage window will need to process these requests again. If you believe your account was impacted or have any questions, please reach out to [email protected].
Certain organizations are experiencing elevated server API error rates, specifically for the create event batch API.
This incident has been resolved.
