Ιστορικό διακοπών του Scalyr

1. 6 Ιουλίου 2026 στις 8:24 μ.μ. UTCresolved

   Χρονολόγιο γεγονότων Ιουλ 6, 06:58 UTC: Γνωρίζουμε τις καθυστερήσεις στην κατάποση δεδομένων που επηρεάζουν ορισμένους πελάτες. Μπορεί να αντιμετωπίσετε καθυστέρηση στην εμφάνιση νέων δεδομένων. Ιουλ 6, 10:23 UTC: Το υποκείμενο ζήτημα έχει εντοπιστεί και οι μηχανικές ομάδες μας εργάζονται ενεργά για την εφαρμογή μιας ρύθμισης. Θα παρέχουμε ενημερώσεις καθώς θα σημειώνουμε πρόοδο και θα εκτιμούμε τη συνεχή υπομονή σας καθώς θα επιλύουμε το περιστατικό. Ιουλ 6, 15:50 UTC: Οι υπηρεσίες έχουν αποκατασταθεί, και εργαζόμαστε για να επικυρώσουμε την υγεία των συστημάτων σας. Ενώ η κατάποση δεδομένων έχει επαναληφθεί, υπάρχει ένα προσωρινό ιστορικό των δεδομένων που εξακολουθούν να υποβάλλονται σε επεξεργασία. Καθώς τα συστήματα φτάνουν, μπορεί να δείτε μια αύξηση σε καθυστερημένες ειδοποιήσεις. Αυτή είναι η αναμενόμενη συμπεριφορά.

Αυτόματη μετάφραση από την επίσημη ενημέρωση του συμβάντος.

1. 10 Ιουνίου 2026 στις 4:51 μ.μ. UTCidentified

   Starting June 9 at 20:55 UTC, ingestion delays affected accounts on app.us1.dataset.com and app.eu1.dataset.com. The issue was identified, and a hotfix was deployed on June 10 at approximately 08:00 UTC. Ingestion has returned to normal in the EU environment. We are currently processing the remaining backlog in the US environment to restore normal performance.

2. 11 Ιουνίου 2026 στις 4:37 μ.μ. UTCresolved

   As of June 11 at 8:11 UTC, the backlog events for accounts on app.us1.dataset.com have been completely processed, and the incident has been resolved.

1. 8 Μαΐου 2026 στις 10:31 μ.μ. UTCinvestigating

   We are currently investigating this issue.

2. 8 Μαΐου 2026 στις 10:50 μ.μ. UTCidentified

   We have identified an issue with a backend cluster that is currently disrupting event ingestion. Our engineering team is performing a system failover to restore normal operations.

3. 9 Μαΐου 2026 στις 12:07 π.μ. UTCmonitoring

   Rebalancing our ingestion servers has stabilized the incoming data volume. Engineering is adding capacity to accelerate the processing of the data backlog. The impacted cluster has resumed ingestion at normal levels. All affected data has been queued since the beginning of the incident, so there is no data loss.

4. 9 Μαΐου 2026 στις 1:30 π.μ. UTCresolved

   This incident has been resolved.

1. 23 Απριλίου 2026 στις 7:45 μ.μ. UTCmonitoring

   Apologies for the corrections. To clarify the scope of the incident mentioned in our previous post, the following environments were impacted: app.eu.scalyr.com, app.us1.dataset.com, and app.eu1.dataset.com.

2. 23 Απριλίου 2026 στις 7:47 μ.μ. UTCresolved

   This incident has been resolved.

1. 23 Απριλίου 2026 στις 7:34 μ.μ. UTCresolved

   Between 4:05 PM and 7:16 PM UTC, some DataSet customers may have experienced intermittent issues logging into the UI via SSO. This incident was limited strictly to the login interface; data ingestion and query capabilities were not impacted and remained fully operational throughout the event. Our engineering team has successfully mitigated the issue via a system rollback, and all SSO login services have been fully restored.

1. 9 Σεπτεμβρίου 2025 στις 5:03 μ.μ. UTCinvestigating

   We are currently investigating an issue affecting both search and ingestion functionality on app.scalyr.com.

2. 9 Σεπτεμβρίου 2025 στις 6:02 μ.μ. UTCidentified

   The issue has been identified and our Engineering teams are working to implement a fix. We will continue to provide updates on our progress.

3. 9 Σεπτεμβρίου 2025 στις 8:29 μ.μ. UTCresolved

   All services have been fully restored and the incident is now resolved. We have validated that all systems are functioning normally. Thank you for your patience throughout this incident.

1. 22 Ιανουαρίου 2025 στις 12:00 π.μ. UTCidentified

   Ingestions were temporarily interrupted at approximately 8:30 PM GMT due to an authorization issue on the cluster. Recovery is currently underway, and users can expect the events to be backfilled slowly to their accounts

2. 22 Ιανουαρίου 2025 στις 2:14 π.μ. UTCresolved

   This incident has been resolved.

1. 16 Ιανουαρίου 2025 στις 6:39 μ.μ. UTCinvestigating

   We are currently investigating this issue.

2. 16 Ιανουαρίου 2025 στις 7:10 μ.μ. UTCinvestigating

   We are in the process of rolling back the change and restarting the affected servers. Some accounts may observe a gradual recovery of their logs.

3. 16 Ιανουαρίου 2025 στις 7:48 μ.μ. UTCidentified

   A misconfiguration deployment caused server resource allocation issues. A rollback and server restart have been initiated to address the problem.

4. 16 Ιανουαρίου 2025 στις 10:01 μ.μ. UTCmonitoring

   Recovery is currently in progress. The estimated time for full ingestion restoration is approximately 60 to 90 minutes from now.

5. 17 Ιανουαρίου 2025 στις 12:22 π.μ. UTCresolved

   The backlog data caused by the incident has been fully recovered in all regions

1. 24 Οκτωβρίου 2024 στις 4:54 μ.μ. UTCinvestigating

   We are currently investigating the issue.

2. 24 Οκτωβρίου 2024 στις 5:13 μ.μ. UTCidentified

   A misconfiguration deployed this morning prevented the servers from scaling up correctly. We are currently in the process of manually scaling up the servers to manage the ingestion volume effectively.

3. 24 Οκτωβρίου 2024 στις 6:11 μ.μ. UTCidentified

   The aggressive scaling out of servers led to a 500 error when loading the page due to hitting the database connection limit. We are currently in the process of scaling the servers back in, and the error should be resolved shortly.

4. 24 Οκτωβρίου 2024 στις 7:11 μ.μ. UTCmonitoring

   The UI should now be loading as expected. We have increased the database connection limit to accommodate more concurrent connections. The queue is in the process of recovering and is gradually processing the backlog of events.

5. 24 Οκτωβρίου 2024 στις 11:49 μ.μ. UTCresolved

   This incident has been resolved.

1. 26 Σεπτεμβρίου 2024 στις 7:36 μ.μ. UTCidentified

   Ingestion on app.us1.dataset.com returns 429 and 500 status codes since 2 pm GMT. A significant increase in ingestion has overloaded the cluster. The issue has been identified and we're working on a fix.

2. 26 Σεπτεμβρίου 2024 στις 11:38 μ.μ. UTCresolved

   The ingestion pipeline and supporting systems are fully functional. While there is no suspected data loss it may take up to 24 hours to work through the backlog of data that has built up during the incident to be fully processed.

1. 20 Σεπτεμβρίου 2024 στις 7:21 μ.μ. UTCinvestigating

   We are currently investigating this issue.

2. 20 Σεπτεμβρίου 2024 στις 7:34 μ.μ. UTCidentified

   The issue has been identified and a fix is being implemented.

3. 20 Σεπτεμβρίου 2024 στις 8:33 μ.μ. UTCresolved

   This incident has been resolved.

1. 12 Σεπτεμβρίου 2024 στις 6:29 μ.μ. UTCresolved

   Ingest for users on app.eu.scalyr.com was effectively halted at 3pm GMT. The issue was fully mitigated at 5:45pm GMT.

1. 10 Σεπτεμβρίου 2024 στις 5:41 μ.μ. UTCidentified

   The correct build of the query servers failed to initialize, leading to performance issues with query execution. The root cause has been identified and the problem has been mitigated for users on app.scalyr.com. The recovery is still in progress for users on app.dataset.com

2. 10 Σεπτεμβρίου 2024 στις 6:12 μ.μ. UTCmonitoring

   All query servers were restarted and query performance is back to normal

3. 10 Σεπτεμβρίου 2024 στις 6:23 μ.μ. UTCresolved

   This incident has been resolved.

1. 19 Αυγούστου 2024 στις 2:05 π.μ. UTCinvestigating

   We are currently investigating this issue.

2. 19 Αυγούστου 2024 στις 2:45 π.μ. UTCidentified

   The issue has been identified and a fix is being implemented.

3. 19 Αυγούστου 2024 στις 3:48 π.μ. UTCresolved

   Ingestion traffic is now back to normal. The issue has been mitigated

1. 12 Ιουλίου 2024 στις 6:57 μ.μ. UTCinvestigating

   The issue is currently under investigation

2. 12 Ιουλίου 2024 στις 7:15 μ.μ. UTCmonitoring

   Queries are now functional on app.eu.scalyr.com after restarting the machines with the OOM issue.

3. 12 Ιουλίου 2024 στις 7:39 μ.μ. UTCresolved

   This incident has been resolved.

1. 2 Μαΐου 2024 στις 4:59 μ.μ. UTCinvestigating

   We are investigating an ingestion issue on app.us1.dataset.com that began around 1450 GMT

2. 2 Μαΐου 2024 στις 6:52 μ.μ. UTCmonitoring

   A fix has been implemented and we are monitoring the results.

3. 2 Μαΐου 2024 στις 11:28 μ.μ. UTCresolved

   Queue server ingestion delay has been resolved.

1. 1 Μαΐου 2024 στις 10:08 μ.μ. UTCresolved

   Failure to update the certification causes ingestion issues on the EU cluster. It has been reverted and the ingestion has resumed since 21:25 GMT

1. 11 Απριλίου 2024 στις 8:18 π.μ. UTCinvestigating

   The issue is currently under investigation

2. 11 Απριλίου 2024 στις 8:25 π.μ. UTCidentified

   The issue has been identified. A new endpoint in the last deployment broke the web application. A revert is in progress

3. 11 Απριλίου 2024 στις 8:55 π.μ. UTCidentified

   The issue was affecting some users on app.scalyr.com and app.eu.scalyr.com. The service on those two endpoints have been restored. We are still in the process of restoring services on app.dataset.com and app.eu.dataset.com

4. 11 Απριλίου 2024 στις 9:39 π.μ. UTCmonitoring

   All sites are now restored and the services are now back online

5. 11 Απριλίου 2024 στις 10:00 π.μ. UTCresolved

   This incident has been resolved

1. 10 Απριλίου 2024 στις 9:48 μ.μ. UTCidentified

   Some users of app.scalyr.com, app.us1.dataset.com, and app.eu1.dataset.com are receiving intermittent 429 errors on search and dashboards. We have identified the issue and are working to remediate it.

2. 10 Απριλίου 2024 στις 10:42 μ.μ. UTCidentified

   A fix has been implemented and validation in progress.

3. 10 Απριλίου 2024 στις 11:30 μ.μ. UTCmonitoring

   A fix has been implemented and we are monitoring the results.

4. 11 Απριλίου 2024 στις 12:03 π.μ. UTCresolved

   This incident has been resolved.

1. 8 Μαρτίου 2024 στις 6:27 μ.μ. UTCresolved

   Users of app.us1.dataset.com experienced slow queries and errors between 16:50 UTC and 17:50 UTC. We have applied a fix to address the immediate issue and are implementing a code change as a long-term solution.