Ιστορικό διακοπών του Insided

1. 21 Οκτωβρίου 2026 στις 11:53 π.μ. UTCidentified

   Ως προληπτικό μέτρο, έχουμε προσωρινά απενεργοποιήσει τη λειτουργία αποστολής στην κοινότητα. Κατά τη διάρκεια αυτής της περιόδου, τα μέλη δεν θα μπορούν να ανεβάζουν αρχεία, συμπεριλαμβανομένων των inline εικόνων, των avatars προφίλ και των συνημμένων. Όλες οι άλλες λειτουργίες της κοινότητας, συμπεριλαμβανομένης της ανάγνωσης, της απόσπασης και της απάντησης, δεν επηρεάζονται. Δεν έχουμε δει καμία επίδραση στα δεδομένα των πελατών ή τη διαθεσιμότητα της κοινότητας. Περιμένουμε να αποκαταστήσουμε τα uploads μόλις ολοκληρωθεί η δουλειά μας και θα ενημερώσουμε αυτή τη σελίδα μόλις έχουμε περισσότερες πληροφορίες.

2. 21 Οκτωβρίου 2026 στις 12:40 μ.μ. UTCidentified

   Συνεχίζουμε να παρακολουθούμε στενά την πλατφόρμα. Η έρευνά μας δεν βρήκε αποδείξεις ότι το θέμα ασφαλείας που αντιμετωπίζουμε ήταν αντικείμενο εκμετάλλευσης. Οι αποστολές αρχείων παραμένουν προσωρινά απενεργοποιημένες ως προφύλαξη, και εργαζόμαστε για την αποκατάστασή τους το συντομότερο δυνατόν. Θα κρατήσουμε αυτή τη σελίδα ενημερωμένη καθώς η κατάσταση εξελίσσεται.

3. 21 Οκτωβρίου 2026 στις 1:00 μ.μ. UTCidentified

   Ευπάθεια PHP (PHP2Shell): αποστολή προσωρινά απενεργοποιημένη Το Ginsight Customer Communities χρησιμοποιεί το PHP. Στις 20 Οκτωβρίου, δύο κρίσιμα τρωτά σημεία στο ίδιο το PHP (CVE-2026-109618 και CVE-2026-109619, γνωστό ως PHP2Shell) αποκαλύφθηκαν πριν από τον επίσημο καθορισμό. Η ομάδα ασφαλείας του PHP απελευθερώνει το fix σήμερα, 21 Οκτωβρίου, στις 16:00 CEST (14:00 UTC). Τι έχουμε κάνει: Απενεργοποίηση όλων των ανεβάσεων αρχείων σε όλες τις κοινότητες, συμπεριλαμβανομένων των inline εικόνων, avatars προφίλ και post συνημμένων. Τοποθετήστε πρόσθετα προστατευτικά τείχη προστασίας έναντι του τύπου της αίτησης που χρησιμοποιείται σε αυτές τις επιθέσεις. Εξετάσαμε τα αρχεία καταγραφής μας για σημάδια εκμετάλλευσης. Μέχρι στιγμής, δεν έχουμε βρει αποδείξεις ότι η ευπάθεια αξιοποιήθηκε στην πλατφόρμα μας. Τι θα συμβεί μετά: Θα εφαρμόσουμε την ενημέρωση για την ασφάλεια του PHP σε όλες τις υποδομές του EMEA και των ΗΠΑ μόλις κυκλοφορήσει. Μετά την επιδιόρθωση, θα τρέξουμε μια πλήρη συμβιβαστική αναθεώρηση και θα μοιραστούμε το αποτέλεσμα εδώ. Θα αποκαταστήσουμε τα uploads μόλις το έμπλαστρο εφαρμοστεί και ελεγχθεί. Ανάγνωση, απόσπαση και απάντηση εργασίας ως κανονική. Η επόμενη ενημέρωση μας θα είναι στις 17:00 CEST, ή νωρίτερα αν αλλάξει κάτι.

4. 21 Οκτωβρίου 2026 στις 2:07 μ.μ. UTCidentified

   Συνεχίζουμε να εργαζόμαστε πάνω σε μια λύση για αυτό το ζήτημα.

5. 21 Οκτωβρίου 2026 στις 2:08 μ.μ. UTCidentified

   Ενημέρωση: Η ομάδα ασφαλείας PHP έχει κυκλοφορήσει τη διόρθωση. Τώρα το επικυρώνουμε και το δοκιμάζουμε ενάντια στην πλατφόρμα μας. Θα το αναπτύξουμε σε όλη την υποδομή του ΕΜΕΑ και των ΗΠΑ μόλις ολοκληρωθούν οι δοκιμές. Οι αποστολές αρχείων παραμένουν προσωρινά απενεργοποιημένες μέχρι να αναπτυχθεί και να επαληθευτεί η διόρθωση. Η παρακολούθησή μας εξακολουθεί να μην αποδεικνύει ότι η ευπάθεια αξιοποιήθηκε στην πλατφόρμα μας. Το διάβασμα, η απόσπαση και η απάντηση εξακολουθούν να λειτουργούν ως φυσιολογικές. Η επόμενη ενημέρωση μας θα είναι στις 5 μμ CEST, ή νωρίτερα αν αλλάξει κάτι.

6. 21 Οκτωβρίου 2026 στις 3:30 μ.μ. UTCidentified

   Ενημέρωση: Δοκιμάζουμε την αποδεσμευμένη επιδιόρθωση για να διασφαλίσουμε ότι δεν θα εισάγουμε καμία παλινδρόμηση. Οι αποστολές αρχείων παραμένουν προσωρινά απενεργοποιημένες μέχρι να αναπτυχθεί και να επαληθευτεί η διόρθωση. Η παρακολούθησή μας εξακολουθεί να μην αποδεικνύει ότι η ευπάθεια αξιοποιήθηκε στην πλατφόρμα μας. Το διάβασμα, η απόσπαση και η απάντηση εξακολουθούν να λειτουργούν ως φυσιολογικές. Η επόμενη ενημέρωση θα γίνει στις 6 μμ CEST, ή νωρίτερα αν αλλάξει κάτι.

7. 21 Οκτωβρίου 2026 στις 4:16 μ.μ. UTCidentified

   Ενημέρωση: Δοκιμάζουμε την αποδεσμευμένη επιδιόρθωση για να διασφαλίσουμε ότι δεν θα εισάγουμε καμία παλινδρόμηση. Οι αποστολές αρχείων παραμένουν προσωρινά απενεργοποιημένες μέχρι να αναπτυχθεί και να επαληθευτεί η διόρθωση. Η παρακολούθησή μας εξακολουθεί να μην αποδεικνύει ότι η ευπάθεια αξιοποιήθηκε στην πλατφόρμα μας. Το διάβασμα, η απόσπαση και η απάντηση εξακολουθούν να λειτουργούν ως φυσιολογικές. Η επόμενη ενημέρωση θα γίνει στις 7 μμ CEST, ή νωρίτερα αν αλλάξει κάτι.

8. 21 Οκτωβρίου 2026 στις 6:18 μ.μ. UTCidentified

   Update: We are completing the testing in staging before deploying to production. File uploads remain temporarily disabled until the fix is deployed and verified. Our monitoring continues to show no evidence that the vulnerability was exploited on our platform. Reading, posting and replying continue to work as normal. Our next update will be by 8:30PM CEST, or sooner if anything changes.

9. 21 Οκτωβρίου 2026 στις 6:53 μ.μ. UTCmonitoring

   Update: The PHP security fix has been deployed to production and we are continuing to monitor closely. We have found no evidence that the vulnerability was exploited on our platform. File uploads remain temporarily disabled out of an abundance of caution. Reading, posting and replying work as normal. Next update by 08:00AM CEST, or sooner if anything changes.

10. 22 Οκτωβρίου 2026 στις 6:12 π.μ. UTCmonitoring

    Update: We are continuing to monitor and have found no evidence that the vulnerability was exploited on our platform. File uploads remain temporarily disabled out of an abundance of caution. Reading, posting and replying work as normal. Next update by 10:00AM CEST, or sooner if anything changes.

11. 22 Οκτωβρίου 2026 στις 8:17 π.μ. UTCmonitoring

    Update: We are continuing to monitor and have found no evidence that the vulnerability was exploited on our platform. File uploads remain temporarily disabled out of an abundance of caution. Reading, posting and replying work as normal. Next update by 12:00PM CEST, or sooner if anything changes.

12. 22 Οκτωβρίου 2026 στις 11:19 π.μ. UTCmonitoring

    Update: We are continuing to monitor and have found no evidence that the vulnerability was exploited on our platform. File uploads remain temporarily disabled out of an abundance of caution. Reading, posting and replying work as normal. Next update by 3:00PM CEST, or sooner if anything changes.

13. 22 Οκτωβρίου 2026 στις 2:18 μ.μ. UTCmonitoring

    Update: We are continuing to monitor and have found no evidence that the vulnerability was exploited on our platform. File uploads remain temporarily disabled out of an abundance of caution. Reading, posting and replying work as normal. Next update by 6:00PM CEST, or sooner if anything changes.

14. 22 Οκτωβρίου 2026 στις 6:43 μ.μ. UTCmonitoring

    Update: We are continuing to monitor and have found no evidence that the vulnerability was exploited on our platform. File uploads that render in the page remain temporarily disabled out of an abundance of caution. Reading, posting and replying work as normal. Attachments of files pdf, zip, txt is again possible. Next update by 10:00PM CEST, or sooner if anything changes.

15. 22 Οκτωβρίου 2026 στις 8:02 μ.μ. UTCmonitoring

    Update: We are continuing to monitor and have found no evidence that the vulnerability was exploited on our platform. File uploads that render in the page remain temporarily disabled out of an abundance of caution. Reading, posting and replying work as normal. Attachments of files pdf, zip, txt is again possible. Next update by 10:00AM CEST, or sooner if anything changes.

16. 23 Οκτωβρίου 2026 στις 11:13 π.μ. UTCmonitoring

    Update: We are continuing to monitor and have found no evidence that the vulnerability was exploited on our platform. Image file uploads that render in the page remain temporarily disabled out of an abundance of caution. Reading, posting and replying work as normal. We will have an update on restoring image file uploads by end of day. Attachments of files pdf, zip, txt is again possible. Next update by 6:00PM CEST, or sooner if anything changes.

17. 23 Οκτωβρίου 2026 στις 7:44 μ.μ. UTCmonitoring

    Update: We are continuing to monitor and have found no evidence that the vulnerability was exploited on our platform. Image file uploads that render in the page remain temporarily disabled out of an abundance of caution. Reading, posting and replying work as normal. We have started reworking the concerned services to ensure we can restore image uploads as fast as possible. We will have a firm ETA by 25 Sunday 12pm CET Attachments of files pdf, zip, txt is again possible. Next update by 12:00PM CEST, or sooner if anything changes.

18. 26 Οκτωβρίου 2026 στις 4:39 π.μ. UTCmonitoring

    Uploads restored: Uploads are available again in all Gainsight Customer Communities. No action is needed. We will continue to monitor closely.

Αυτόματη μετάφραση από την επίσημη ενημέρωση του συμβάντος.

1. 29 Σεπτεμβρίου 2026 στις 11:05 π.μ. UTCidentified

   Το θέμα έχει εντοπιστεί και υλοποιείται μια λύση.

2. 29 Σεπτεμβρίου 2026 στις 11:28 π.μ. UTCmonitoring

   Εφαρμόστηκε μια ρύθμιση και παρακολουθούμε τα αποτελέσματα.

3. 29 Σεπτεμβρίου 2026 στις 12:35 μ.μ. UTCresolved

   Αυτό το περιστατικό επιλύθηκε - ζητάμε ειλικρινά συγγνώμη για αυτή τη διαταραχή στην πλατφόρμα και θα ανεβάσουμε εδώ ένα έγγραφο RCA όταν ολοκληρωθεί.

Αυτόματη μετάφραση από την επίσημη ενημέρωση του συμβάντος.

1. 24 Αυγούστου 2026 στις 7:35 π.μ. UTCinvestigating

   Αυτή τη στιγμή ερευνούμε αυτό το θέμα - η επισκόπηση των χρηστών για τις κοινότητες των ΗΠΑ παίρνει περισσότερο από το συνηθισμένο για να εμφανίσει νέους χρήστες, αυτή η βραδύτητα συμβαίνει επίσης στα widgets αρχική σελίδα ελέγχου.

2. 25 Αυγούστου 2026 στις 1:21 μ.μ. UTCresolved

   Το περιστατικό λύθηκε.

Αυτόματη μετάφραση από την επίσημη ενημέρωση του συμβάντος.

1. 4 Αυγούστου 2026 στις 7:57 π.μ. UTCinvestigating

   Αυτή τη στιγμή διερευνούμε ένα θέμα με θέματα που δεν φορτώνουν σωστά, θα ενημερώσουμε αυτή τη σελίδα το συντομότερο δυνατόν με περισσότερες πληροφορίες κάποτε γνωστές.

2. 4 Αυγούστου 2026 στις 8:08 π.μ. UTCmonitoring

   A fix has been implemented and we are monitoring the results.

3. 4 Αυγούστου 2026 στις 11:27 π.μ. UTCresolved

   This issue has been resolved - time of the incident (CEST) 9:33am - 10:00am; a 27 minute period in which most topics were returning a 404 error page instead of the topic. It was caused by an unexpected side affect of a downstream change to one of our platform security mechanisms, we apologise for this temporary disruption.

Αυτόματη μετάφραση από την επίσημη ενημέρωση του συμβάντος.

1. 8 Ιουλίου 2026 στις 9:45 π.μ. UTCidentified

   Intermittent timeouts and slowness are identified in our EU communities. We identified the issue and are mitigating it. We will follow up and update this page as soon as we have more information.

2. 8 Ιουλίου 2026 στις 11:54 π.μ. UTCresolved

   We have traced the root cause of the recent downtime to an AWS hardware failure. We apologize for the inconvenience this caused. Our team is already putting structural measures in place to ensure that similar hardware issues will not impact our platform going forward.

1. 2 Ιουλίου 2026 στις 5:28 μ.μ. UTCinvestigating

   We are currently investigating an issue affecting AI Summary features, which may result in unexpected or incomplete responses. Search results are not impacted.

2. 2 Ιουλίου 2026 στις 7:34 μ.μ. UTCresolved

   The issue affecting AI Summary features has been successfully resolved, and normal functionality has been fully restored. Thank you for your patience while we worked to fix this.

1. 29 Μαΐου 2026 στις 7:21 μ.μ. UTCinvestigating

   We are currently investigating reports that the Events module is not working or failing to load for users within our US communities. Our engineering team is looking into the root cause, and we will provide an update within the next 30 minutes. We apologize for the inconvenience and appreciate your patience while we work to resolve this.

2. 29 Μαΐου 2026 στις 7:45 μ.μ. UTCresolved

   The issue affecting the Events module in US communities has been fully resolved, and normal functionality has been restored. We appreciate your patience and apologize for any disruption this caused to your workflow. If you continue to experience any issues, please reach out to our support team at [email protected]

1. 21 Νοεμβρίου 2025 στις 12:47 μ.μ. UTCinvestigating

   We are investigating reports of Salesforce Connection failures and we will update here as soon as we have more details.

2. 21 Νοεμβρίου 2025 στις 2:01 μ.μ. UTCidentified

   This issue has been tied to the temporary revocation of Salesforce authentication credentials by Salesforce - we are working on getting these reinstated asap, no community data should be lost and events can be replayed again on our side to fill in the gaps once the authentication credentials have been reinstated.

3. 22 Νοεμβρίου 2025 στις 2:57 π.μ. UTCidentified

   Work is ongoing to get Salesforce authentication reinstated. No community data should be lost and events can be replayed again on our side to fill in the gaps once the authentication credentials have been reinstated.

4. 22 Νοεμβρίου 2025 στις 7:36 μ.μ. UTCidentified

   The Salesforce and Gainsight incident remains ongoing. Our teams are actively working to resolve the issue and restore functionality as quickly and safely as possible. For the latest updates and FAQs, please visit the main status page: https://status.gainsight.com/incidents/gvng0kly8vwf

5. 22 Νοεμβρίου 2025 στις 7:48 μ.μ. UTCidentified

   We are continuing to work on a fix for this issue.

6. 2 Δεκεμβρίου 2025 στις 5:42 μ.μ. UTCidentified

   The Salesforce and Gainsight incident remains ongoing. Rest assured our teams are working around the clock to resolve the issue and restore functionality as quickly and safely as possible. In the meantime, we are preparing a workaround and steps to re-enable services. For the latest updates and FAQs, please visit the main status page: https://status.gainsight.com/incidents/gvng0kly8vwf

7. 13 Δεκεμβρίου 2025 στις 10:43 μ.μ. UTCmonitoring

   The Gainsight CC<>Salesforce integration has now been re-enabled and customers can proceed with re-authenticating their connection. To restore full functionality, please follow the step-by-step reconnection guide available here: https://communities.gainsight.com/customer-communities-cc-17/communities-salesforce-integration-restoration-guide-29962

8. 18 Μαρτίου 2026 στις 4:35 μ.μ. UTCresolved

   This incident has been resolved.

1. 17 Νοεμβρίου 2025 στις 9:07 π.μ. UTCinvestigating

   We are currently investigating an issue on some communities where content cannot be created or edited; either via the control administration area or sometimes from within the community destination pages.

2. 17 Νοεμβρίου 2025 στις 12:03 μ.μ. UTCidentified

   The issue has been identified and a fix is being implemented.

3. 17 Νοεμβρίου 2025 στις 12:20 μ.μ. UTCmonitoring

   A fix has been implemented and we are monitoring the results.

4. 17 Νοεμβρίου 2025 στις 1:49 μ.μ. UTCresolved

   This issue has been resolved - we traced the source of the issue to some database indexes that were lost when a subsequent database update wiped out these indexes. We estimate that only around 10% of customers had this issue happen to their community and the problem was certainly not widespread. Any left over issues please contact [email protected] and we apologies for any inconvenience caused by this.

1. 10 Νοεμβρίου 2025 στις 6:38 μ.μ. UTCinvestigating

   We’re investigating an issue where custom community pages may not load correctly.

2. 10 Νοεμβρίου 2025 στις 8:15 μ.μ. UTCidentified

   Our engineering team has identified the root cause and is actively working on a fix.

3. 10 Νοεμβρίου 2025 στις 9:08 μ.μ. UTCidentified

   We have applied the initial fix which resolves this issue for the majority of customers. We are actively working on a fix that continues to impact custom pages for less than 1% of CC customers.

4. 10 Νοεμβρίου 2025 στις 10:19 μ.μ. UTCidentified

   We are continuing to work on a fix for this issue.

5. 10 Νοεμβρίου 2025 στις 11:54 μ.μ. UTCresolved

   This incident has been resolved.

1. 6 Οκτωβρίου 2025 στις 3:18 μ.μ. UTCidentified

   Embedded media is currently not working for some video providers (e.g., Vimeo, Twitter, Vidyard, etc.). YouTube embeds remain unaffected and continue to work as expected. We’ve identified the root cause of the issue, and our engineering team is actively working on a fix. We’ll share an update as soon as the resolution is deployed.

2. 6 Οκτωβρίου 2025 στις 7:25 μ.μ. UTCmonitoring

   The fix was deployed, we are now monitoring this issue.

3. 6 Οκτωβρίου 2025 στις 7:55 μ.μ. UTCresolved

   This incident has been resolved.

1. 26 Σεπτεμβρίου 2025 στις 8:32 π.μ. UTCinvestigating

   Embedded media (e.g., Vimeo videos) is currently not working across communities. Users may see “Media not available” when trying to play embedded videos. Our engineering team is actively investigating the issue and working on a fix.

2. 26 Σεπτεμβρίου 2025 στις 10:12 π.μ. UTCresolved

   The incident affecting embedded media has been resolved. Embedded videos should now display correctly across all affected communities.

1. 3 Σεπτεμβρίου 2025 στις 4:49 μ.μ. UTCidentified

   We identified a degraded performance especially on the homepages of our US communities. The investigation is ongoing into what is the root cause to this.

2. 3 Σεπτεμβρίου 2025 στις 5:43 μ.μ. UTCidentified

   The root cause was identified, and the performance is back to expected performance. We will continue monitoring this incident.

3. 3 Σεπτεμβρίου 2025 στις 5:44 μ.μ. UTCidentified

   We are continuing to work on a fix for this issue.

4. 3 Σεπτεμβρίου 2025 στις 5:44 μ.μ. UTCmonitoring

   A fix has been implemented and we are monitoring the results.

5. 3 Σεπτεμβρίου 2025 στις 8:56 μ.μ. UTCresolved

   This incident has been resolved.

1. 29 Αυγούστου 2025 στις 8:48 π.μ. UTCidentified

   We have identified an issue with LinkedIn and Facebook login functionality - it can be temporarily mitigated by adding a new 'Authorized redirect URL for your app', adding a http version of the url e.g: http://[yourcommunityurl.com]/sso/return. We are working on a full fix for this now though and will update this page with more information once available. Please note that this issue will not affect normal SSO logins / registrations which continue to function normally.

2. 29 Αυγούστου 2025 στις 10:36 π.μ. UTCmonitoring

   A fix has been implemented and we are monitoring the results.

3. 29 Αυγούστου 2025 στις 11:30 π.μ. UTCresolved

   This incident has been resolved and both Linkedin and Facebook logins are back to normal, we sincerely apologise for the disruption caused by this.

1. 6 Αυγούστου 2025 στις 2:05 μ.μ. UTCinvestigating

   We are investigating an issue where topics are not loading and and error page is shown, it seems to be affecting Articles and Product Updates.

2. 6 Αυγούστου 2025 στις 2:24 μ.μ. UTCidentified

   The issue has been identified and a fix is being implemented.

3. 6 Αυγούστου 2025 στις 3:33 μ.μ. UTCmonitoring

   A fix has been implemented and we are monitoring the results.

4. 6 Αυγούστου 2025 στις 5:28 μ.μ. UTCresolved

   This incident has been resolved.

1. 28 Ιουλίου 2025 στις 5:29 μ.μ. UTCidentified

   We’re currently experiencing an issue where community pages are not loading correctly. Our engineering team has identified the root cause and is actively working on a fix. We’re aiming to restore full functionality as soon as possible and will share updates here as progress is made. Thank you for your patience and understanding.

2. 28 Ιουλίου 2025 στις 9:24 μ.μ. UTCresolved

   This incident has been resolved.

1. 11 Αυγούστου 2025 στις 10:21 π.μ. UTCresolved

   Incident Summary: On 24 July 2025, we deployed a performance improvement designed to speed up page loading by changing when certain parts of our application load in the browser. As a result of this change, some page elements loaded later than before. Certain third party scripts used by customers depended on those elements being available straight away, so those scripts failed to run correctly. This caused integration errors and broken functionality for affected customers. Impact: Seven customers experienced issues with their integrations. In these cases, third party scripts could not access the expected page elements, which led to errors and features not working as intended. Timeline: - 24 July 2025 9:00 AM (CET) - Deployment completed with new loading behaviour as part of performance improvements. - 25 July 2025 - First customer reports of integration issues received. An internal investigation began to identify the cause. - 26 to 29 July 2025 - Investigation confirmed that lazy loading of certain elements meant they were not available when customer scripts ran. Compatibility checks were carried out to assess possible fixes and confirm the scope of the impact. - 30 July 2025 15:30 AM (CET) - Decision made to revert the change to restore the original loading behaviour. Reversion deployed to production. Monitoring confirmed resolution for all affected accounts. Impacted customers were contacted and confirmed the issue was resolved. Resolution: We rolled back the performance change so that all elements load as they did before. This restored the expected functionality for all affected customers. Mitigation and Next Steps: - Implement a compatibility aware rollout strategy for future performance changes. - Conduct an audit of existing third party scripts to identify other potential compatibility risks. - Include broader customer impact reviews before releasing performance optimisations. - Give customers and support teams more notice of significant changes that may affect integrations. Lessons Learned: There are a wide range of third party scripts in use across all customer instances, and changes to page performance can have unintended side effects. Future performance updates will be introduced in phased rollouts with additional compatibility checks, and customer impact will be reviewed more thoroughly before release.

1. 15 Ιουλίου 2025 στις 3:14 μ.μ. UTCinvestigating

   We are currently investing an issue with an elevated error rate (15-20%) affecting EU communities, we will update here once more information is available.

2. 15 Ιουλίου 2025 στις 3:24 μ.μ. UTCidentified

   The issue has been identified and a fix is being implemented.

3. 15 Ιουλίου 2025 στις 3:34 μ.μ. UTCresolved

   This issue has been resolved and the platform is stable again. Any more follow up issues please contact [email protected] - apologies for this short disruption.

1. 27 Ιουνίου 2025 στις 7:26 π.μ. UTCinvestigating

   We are currently investigating an issue with SSO logins leading to an error 500 page after login.

2. 27 Ιουνίου 2025 στις 7:42 π.μ. UTCidentified

   The issue has been identified and a fix is being implemented.

3. 27 Ιουνίου 2025 στις 8:08 π.μ. UTCmonitoring

   A fix has been implemented and we are monitoring the results.

4. 27 Ιουνίου 2025 στις 9:39 π.μ. UTCresolved

   This incident has been resolved - we sincerely apologise for the disruption caused by this, any more issues or questions please email [email protected].

1. 13 Ιουνίου 2025 στις 3:19 μ.μ. UTCinvestigating

   We are currently investigating this issue.

2. 13 Ιουνίου 2025 στις 3:45 μ.μ. UTCinvestigating

   We are continuing to investigate this issue - we have identified a couple of infrastructure issues thought to be causing this. We appreciate your continued patience,

3. 13 Ιουνίου 2025 στις 4:46 μ.μ. UTCinvestigating

   We are continuing to investigate the source of the problem, and have identified a few possibilities for it. As soon as we know more information we will post it here asap.

4. 13 Ιουνίου 2025 στις 5:37 μ.μ. UTCinvestigating

   We are continuing to investigate this issue.

5. 13 Ιουνίου 2025 στις 6:11 μ.μ. UTCinvestigating

   User experience is improving at this time but we are still investigating.

6. 13 Ιουνίου 2025 στις 6:44 μ.μ. UTCinvestigating

   Users may experience degraded performance in some areas.

7. 13 Ιουνίου 2025 στις 6:56 μ.μ. UTCidentified

   The issue has been identified, and a fix was deployed. Performance is steadily improving.

8. 13 Ιουνίου 2025 στις 7:05 μ.μ. UTCmonitoring

   We are monitoring the results of our fix, and performance is back to normal.

9. 13 Ιουνίου 2025 στις 8:03 μ.μ. UTCresolved

   This incident has been resolved. Details will be added once available.

10. 19 Ιουνίου 2025 στις 10:58 π.μ. UTCpostmortem

    **Redis Cache Degradation and Service Impact - June 13, 2025** ### **Overview** On **June 13, 2025**, from **17:09 to 21:13 CET**, some of our services in the **US region** experienced significant disruptions due to a cascading infrastructure failure. The issue began with degraded performance in our Redis caching layer, triggered by underlying **AWS infrastructure instability**, and ultimately led to elevated error rates and service outages across several systems. ‌ ### **What Happened** A sudden degradation in our Redis cache - likely caused by transient **network issues within AWS infrastructure** - resulted in dramatically slower response times and dropped connections. As Redis became unavailable, systems attempted to query the main database directly, which caused that database to hit its connection limits and become overloaded. This caused broader performance issues across multiple services. ### **Customer Impact** * **High error rates** \(HTTP 5XX\) across several services in the US region * **Slower API response times** and degraded system performance * **Stale or delayed data** due to failed or missed caching * **Up to 4 hours of partial or full service disruption** * **No customer data was lost**, but availability and performance were impacted ### **Timeline of Events \(CEST\)** * **17:09** - Error rates spike and initial alarms trigger * **17:13** - Incident response begins * **17:30-19:00** - Teams work to stabilise the system, including service rollbacks and traffic load reduction * **18:38** - AWS confirms a related infrastructure/network event affecting Redis connectivity * **19:30** - Redis instance upgraded to a higher-capacity configuration * **21:10** - Services fully recover * **21:13** - Incident officially resolved ### **What Caused the Issue?** Our Redis cache experienced performance degradation and stopped serving traffic effectively. Based on internal logs and external confirmations, this was **likely due to transient network and infrastructure issues within AWS**, which impacted bandwidth and connection handling for Redis. As a result, traffic was redirected to the main database, which then became overloaded, creating a cascading service failure. ### **How Was It Resolved?** * Redis was scaled up to handle more traffic and recover performance * Non-essential services were temporarily paused to reduce system load * Once Redis recovered, database pressure eased and systems began returning to normal ### **Could This Have Been Prevented?** Partially. Some safeguards helped limit impact \(such as CDN caching for guest users\), but earlier detection of Redis degradation and better isolation between cache and core systems could have reduced downtime. The unexpected AWS infrastructure event contributed significantly to the incident’s scope. ‌ ### **Key Lessons and Next Steps** * **Improve monitoring and alerting** for Redis resource exhaustion and network saturation * **Isolate caching infrastructure** by service or use case to prevent single points of failure * **Enhance fallback logic** to reduce stress on primary databases when caching layers fail * **Work closely with AWS** to better understand and mitigate infrastructure-related disruptions * **Explore more resilient multi-layer caching strategies** to handle future spikes gracefully We sincerely apologise for the disruption and appreciate your patience. We are taking meaningful steps to strengthen our infrastructure and improve our detection and response capabilities to prevent similar events in the future. For any more information or issues please email [[email protected]](mailto:[email protected]).