Ιστορικό διακοπών του Jumpcloud

1. 23 Οκτωβρίου 2026 στις 3:55 μ.μ. UTCinvestigating

   Αυτή τη στιγμή ερευνούμε αναφορές σφαλμάτων κατά τη φόρτωση του χρήστη JumpCloud ή της πύλης Admin. Θα ενημερώσουμε με πρόσθετες πληροφορίες μέσα σε 30 λεπτά.

2. 23 Οκτωβρίου 2026 στις 4:00 μ.μ. UTCmonitoring

   Εντοπίσαμε λάθη που σχετίζονται με μια υπηρεσία τρίτων. Τα ποσοστά σφάλματος έχουν μειωθεί και συνεχίζουμε να παρακολουθούμε.

3. 23 Οκτωβρίου 2026 στις 4:11 μ.μ. UTCmonitoring

   Συνεχίζουμε να παρακολουθούμε για τυχόν περαιτέρω ζητήματα.

4. 23 Οκτωβρίου 2026 στις 4:24 μ.μ. UTCmonitoring

   Μεταξύ περίπου 15:20 και 15:50 UTC στις 23 Οκτωβρίου 2026, ορισμένοι πελάτες δεν ήταν σε θέση να φορτώσουν αξιόπιστα το JumpCloud Admin Portal και το User Portal. Επηρεασμένοι χρήστες μπορεί να έχουν δει σελίδες σφαλμάτων ή αργά, ελλιπή φορτία σελίδων. Η αιτία ήταν μια διακοπή του δικτύου σε έναν από τους τρίτους φορείς παροχής περιεχομένου και ασφάλειας, που επηρέασε την κυκλοφορία μέσω ορισμένων από τις ανατολικές τοποθεσίες των ΗΠΑ. Μόλις σταθεροποιηθεί το δίκτυό τους, η πρόσβαση και στις δύο πύλες επανήλθε στο φυσιολογικό, και έχουμε επιβεβαιώσει ότι το θέμα έχει επιλυθεί πλήρως. Γνωρίζουμε πόσο εξαρτάστε από την αξιόπιστη πρόσβαση στο JumpCloud για να διαχειριστείτε τον οργανισμό σας, και λυπούμαστε για την αναστάτωση που προκάλεσε σε εσάς και τους χρήστες σας. Ακόμα και όταν μια διαταραχή ξεκινά με έναν πάροχο, το να σας κρατάμε συνδεδεμένους είναι δική μας ευθύνη. Ήδη διαθέτουμε πλεονάσματα για τέτοιου είδους γεγονότα, και το ενισχύουμε ενεργά έτσι ώστε η αποτυχία να γίνεται γρηγορότερα και πιο αυτόματα στο μέλλον. Διεξάγουμε μια πλήρη ανασκόπηση αυτού του περιστατικού και θα δημοσιεύσουμε μια Ανασκόπηση Post-Incident εδώ με ένα λεπτομερές χρονοδιάγραμμα, βασική αιτία, και τα βήματα που κάνουμε για να αποτρέψουμε μια επανάληψη.

5. 23 Οκτωβρίου 2026 στις 6:25 μ.μ. UTCresolved

   Το περιστατικό λύθηκε.

Αυτόματη μετάφραση από την επίσημη ενημέρωση του συμβάντος.

1. 22 Οκτωβρίου 2026 στις 4:21 μ.μ. UTCinvestigating

   Έχουμε εντοπίσει ένα θέμα που επηρεάζει την έκδοση πιστοποιητικών συσκευών και την ανανέωση για τους πελάτες μας στην περιοχή των ΗΠΑ, ξεκινώντας από περίπου 9:26 ΠΜ MT (15:26 UTC). Κατά τη διάρκεια αυτής της περιόδου, οι συσκευές που ζητούν νέο ή ανανεωμένο πιστοποιητικό συσκευής ενδέχεται να αδυνατούν να αποκτήσουν πιστοποιητικό. Αυτό επηρεάζει κυρίως τις νεοεγγεγραμμένες συσκευές και συσκευές των οποίων τα πιστοποιητικά πρόκειται να ανανεωθούν. Οι συσκευές με τα υφιστάμενα έγκυρα πιστοποιητικά δεν επηρεάζονται και συνεχίζουν να λειτουργούν κανονικά.

2. 22 Οκτωβρίου 2026 στις 4:44 μ.μ. UTCidentified

   Το θέμα έχει εντοπιστεί και υλοποιείται μια λύση.

3. 22 Οκτωβρίου 2026 στις 4:50 μ.μ. UTCmonitoring

   Εφαρμόστηκε μια ρύθμιση και παρακολουθούμε τα αποτελέσματα.

4. 22 Οκτωβρίου 2026 στις 5:00 μ.μ. UTCresolved

   Το περιστατικό λύθηκε.

Αυτόματη μετάφραση από την επίσημη ενημέρωση του συμβάντος.

1. 25 Σεπτεμβρίου 2026 στις 5:27 μ.μ. UTCinvestigating

   Αυτή τη στιγμή ερευνούμε αναφορές για υποβαθμισμένες επιδόσεις με ενημέρωση λεπτομερειών ή ενώσεων προς εντολές στην Πύλη Διοίκησης. Ερευνούμε την αιτία των ζητημάτων επί του παρόντος, και θα ενημερώσουμε το γεγονός κατάστασης μόλις επιλυθεί.

2. 25 Σεπτεμβρίου 2026 στις 5:29 μ.μ. UTCidentified

   Το θέμα έχει εντοπιστεί και υλοποιείται μια λύση.

3. 25 Σεπτεμβρίου 2026 στις 5:55 μ.μ. UTCmonitoring

   Εφαρμόστηκε μια ρύθμιση και παρακολουθούμε τα αποτελέσματα.

4. 25 Σεπτεμβρίου 2026 στις 6:33 μ.μ. UTCresolved

   Το περιστατικό λύθηκε.

Αυτόματη μετάφραση από την επίσημη ενημέρωση του συμβάντος.

1. 25 Σεπτεμβρίου 2026 στις 5:22 μ.μ. UTCinvestigating

   Αυτή τη στιγμή ερευνούμε αναφορές για υποβαθμισμένες επιδόσεις με ενημέρωση λεπτομερειών ή ενώσεων προς εντολές στην Πύλη Διοίκησης. Ερευνούμε την αιτία των ζητημάτων επί του παρόντος, και θα ενημερώσουμε το γεγονός κατάστασης μόλις επιλυθεί.

Αυτόματη μετάφραση από την επίσημη ενημέρωση του συμβάντος.

1. 14 Σεπτεμβρίου 2026 στις 1:33 μ.μ. UTCinvestigating

   Αυτή τη στιγμή ερευνούμε ένα θέμα όπου μετά από μια επαναφορά κωδικού πρόσβασης, κάποια RADIUS sign-ins (VPN/Wi‐Fi) μπορεί να αποτύχει με το νέο κωδικό πρόσβασης. Ωστόσο - επιρρεπείς χρήστες μπορούν ακόμα να συνδεθείτε με παλιό κωδικό πρόσβασης. Διερευνούμε την αιτία των ζητημάτων που βρίσκονται επί του παρόντος, και θα παρέχουμε ενημέρωση εντός μιας ώρας.

Αυτόματη μετάφραση από την επίσημη ενημέρωση του συμβάντος.

1. 14 Σεπτεμβρίου 2026 στις 1:25 μ.μ. UTCinvestigating

   Αυτή τη στιγμή ερευνούμε ένα θέμα όπου μετά από μια επαναφορά κωδικού πρόσβασης, κάποια RADIUS sign-ins (VPN/Wi‐Fi) μπορεί να αποτύχει με το νέο κωδικό πρόσβασης. Ωστόσο - επιρρεπείς χρήστες μπορούν ακόμα να συνδεθείτε με παλιό κωδικό πρόσβασης. Διερευνούμε την αιτία των ζητημάτων που βρίσκονται επί του παρόντος, και θα παρέχουμε ενημέρωση εντός μιας ώρας.

2. 14 Σεπτεμβρίου 2026 στις 1:58 μ.μ. UTCmonitoring

   Εφαρμόστηκε μια ρύθμιση και παρακολουθούμε τα αποτελέσματα.

3. 14 Σεπτεμβρίου 2026 στις 2:38 μ.μ. UTCresolved

   Αυτό το περιστατικό έχει πλέον επιλυθεί.

4. 21 Σεπτεμβρίου 2026 στις 8:49 μ.μ. UTCpostmortem

    # ** Αναφορά συστατικού ** **Ημερομηνία **: 21 Σεπ 2026 **Ημερομηνία συμβάντος: ** 14 Σεπ 2026 **Περιγραφή**: Ανάλυση ριζικών αιτίων για αποτυχίες ταυτοποίησης RADIUS που επηρεάζουν τους χρήστες της ΕΕ **Περιοχή: ** Ευρωπαϊκή Ένωση **Περίληψη: ** Στις 14 Σεπτεμβρίου 2026, το JumpCloud εντόπισε ένα ζήτημα που επηρεάζει την ταυτοποίηση του RADIUS για τους χρήστες στην περιοχή της Ευρωπαϊκής Ένωσης. Οι χρήστες που είχαν αλλάξει πρόσφατα τον κωδικό τους δεν ήταν σε θέση να συνδεθούν με το νέο τους κωδικό πρόσβασης, ενώ ο προηγούμενος κωδικός τους συνέχισε να λειτουργεί. Το θέμα προήλθε από ένα αντίγραφο της βάσης δεδομένων στην περιοχή της ΕΕ που σταμάτησε να λαμβάνει ενημερώσεις από την κύρια βάση δεδομένων μετά από μια προγραμματισμένη αναβάθμιση της υποδομής. Ως αποτέλεσμα, η πιστοποίηση RADIUS στην περιοχή της ΕΕ επιδόθηκε από ένα ξεπερασμένο αντίγραφο των δεδομένων καταλόγου, και οι πρόσφατες αλλαγές του λογαριασμού δεν αντικατοπτρίστηκαν εκεί. Το JumpCloud αποκατέστησε την υπηρεσία με την αναδρομολόγηση της ταυτοποίησης του EU RADIUS σε υγιή αντίγραφα βάσεων δεδομένων σε άλλες περιοχές, στη συνέχεια ανακατασκευάζοντας και ξανασυγχρονίζοντας το προσβεβλημένο αντίγραφο της ΕΕ. Η ταυτοποίηση στην περιοχή της ΕΕ αποκαταστάθηκε πλήρως την ίδια ημέρα. ** Τι συνέβη: ** Στο πλαίσιο μιας προγραμματισμένης αναβάθμισης της υποδομής, οι περιπτώσεις βάσεων δεδομένων που υποστηρίζουν την πιστοποίηση RADIUS αναβαθμίστηκαν. Στην περιοχή της ΕΕ, το αντίγραφο που διαβάστηκε στη βάση δεδομένων δεν επανέλαβε την αναπαραγωγή μετά την αναβάθμιση και σταμάτησε να λαμβάνει ενημερώσεις από την κύρια βάση δεδομένων από τις 10 Σεπτεμβρίου 2026. Μόνο το αντίγραφο της ΕΕ επηρεάστηκε. Επειδή η πιστοποίηση RADIUS στην περιοχή της ΕΕ επιδόθηκε από αυτό το αντίγραφο, στηρίχθηκε σε ένα αντίγραφο δεδομένων καταλόγου που ίσχυε μέχρι τις 10 Σεπτεμβρίου. Οποιαδήποτε αλλαγή λογαριασμού που έγινε μετά το σημείο αυτό δεν αντικατοπτρίστηκε στην περιοχή της ΕΕ μέχρι να επιλυθεί το ζήτημα. Το πιο ορατό σύμπτωμα ήταν οι αλλαγές κωδικού πρόσβασης - ένας νέος κωδικός πρόσβασης θα αποτύχει ενώ ο προηγούμενος κωδικός πρόσβασης εξακολουθεί να λειτουργεί. Ως εκ τούτου, κατά τη διάρκεια του θιγόμενου παραθύρου, οι αλλαγές του λογαριασμού στην περιοχή της ΕΕ ενδέχεται να μην έχουν τεθεί σε ισχύ εκεί, συμπεριλαμβανομένων των αλλαγών που αποσκοπούν στην αφαίρεση ή την αναστολή της πρόσβασης του χρήστη. Μόλις επιλύθηκε το ζήτημα και επανασυγχρονιστεί το αντίγραφο, εφαρμόστηκαν όλες οι εκκρεμείς αλλαγές και τα δεδομένα του καταλόγου της ΕΕ επέστρεψαν σε μια πλήρως τρέχουσα κατάσταση. ** Συμβολή Παράγοντας: ** Η καθυστέρηση στον εντοπισμό μεταξύ 10 Σεπτεμβρίου και 14 Σεπτεμβρίου συνέβη επειδή η πληγείσα περίπτωση της βάσης δεδομένων συνέχισε να αναφέρει μια υγιή κατάσταση στο επίπεδο της υποδομής, παρόλο που η αντιγραφή είχε σταματήσει. Η ειδοποίησή μας ήταν προσανατολισμένη γύρω από αυτές τις μετρήσεις διαθεσιμότητας σε επίπεδο περίπτωσης και δεν κάλυψε αυτή τη συγκεκριμένη κατάσταση αντιγραφής, έτσι η σταματημένη αντιγραφή δεν προκάλεσε έναν αυτόματο συναγερμό, καλύπτοντας το υποκείμενο ζήτημα. Το κλείσιμο αυτού του χάσματος ανίχνευσης είναι το επίκεντρο των διορθωτικών μέτρων που ακολουθούν. **Συγκριτική δράση: ** Το JumpCloud λαμβάνει τις ακόλουθες ενέργειες για να αποτρέψει την επανάληψη: 1. **Ανακύκλωση-έλεγχος της υγείας και προειδοποίηση. ** Προσθέτουμε αυτοματοποιημένη ειδοποίηση που ανιχνεύει όταν ένα αντίγραφο της βάσης δεδομένων σταματά την αντιγραφή, ανεξάρτητα από τη γενική του κατάσταση διαθεσιμότητας, οπότε αυτή η κατηγορία αποτυχίας συλλαμβάνεται αμέσως. Κατάσταση: Σε εξέλιξη. 2. **Αυστηρή επικύρωση ετοιμότητας για την επαλήθευση της κυκλοφορίας. ** Ενισχύουμε τους υγειονομικούς μας ελέγχους έτσι ώστε ένα αντίγραφο που δεν είναι πλήρως συγχρονισμένο να αφαιρείται αυτόματα από την εξυπηρέτηση της κυκλοφορίας πιστοποίησης, εξασφαλίζοντας ότι τα αιτήματα θα απαντηθούν μόνο από ενημερωμένα δεδομένα. Κατάσταση: Προγραμματισμένη. 3. ** Σταθερή rollout με επικύρωση αντιγραφής για αλλαγές υποδομής. ** Ενημερώνουμε τις διαδικασίες μας για αλλαγές σε περιπτώσεις βάσεων δεδομένων ώστε να εφαρμόζονται μία περιοχή τη φορά, με ρητή επαλήθευση ότι η αντιγραφή έχει ανακάμψει πλήρως πριν προχωρήσει στην επόμενη περιοχή. Κατάσταση: Σε εξέλιξη.

Αυτόματη μετάφραση από την επίσημη ενημέρωση του συμβάντος.

1. 3 Σεπτεμβρίου 2026 στις 6:07 π.μ. UTCinvestigating

   Το JumpCloud παρουσιάζει αυτή τη στιγμή καθυστερήσεις στην ανάθεση χρηστών και συστημάτων σε δυναμικές ομάδες. Οι διαχειριστές ενδέχεται να διαπιστώσουν καθυστερήσεις σε αυτές τις ενημερώσεις μελών του Ομίλου, και έτσι καθυστερήσεις στην παροχή συνδεδεμένων πόρων.

2. 3 Σεπτεμβρίου 2026 στις 6:14 π.μ. UTCidentified

   Το θέμα έχει εντοπιστεί και υλοποιείται μια λύση.

3. 3 Σεπτεμβρίου 2026 στις 8:56 π.μ. UTCidentified

   Συνεχίζουμε να εργαζόμαστε πάνω σε μια λύση για αυτό το ζήτημα.

4. 3 Σεπτεμβρίου 2026 στις 10:05 π.μ. UTCmonitoring

   Εφαρμόστηκε μια ρύθμιση και παρακολουθούμε τα αποτελέσματα.

5. 3 Σεπτεμβρίου 2026 στις 10:54 π.μ. UTCmonitoring

   Συνεχίζουμε να παρακολουθούμε για τυχόν περαιτέρω ζητήματα.

6. 3 Σεπτεμβρίου 2026 στις 12:22 μ.μ. UTCresolved

   Το περιστατικό λύθηκε.

Αυτόματη μετάφραση από την επίσημη ενημέρωση του συμβάντος.

1. 22 Ιουνίου 2026 στις 2:21 μ.μ. UTCinvestigating

   We are currently investigating delays or timeouts loading the user console. We are investigating the cause of the issues currently, and will provide an update within one hour.

2. 22 Ιουνίου 2026 στις 3:08 μ.μ. UTCidentified

   We have identified that a third party outage is impacting our service. More information can be found at https://www.cloudflarestatus.com/incidents/v3yl7jqmqj51

3. 22 Ιουνίου 2026 στις 4:39 μ.μ. UTCresolved

   The third party issue impacting our service has been resolved. Our internal monitoring is showing service states returned to normal.

4. 25 Ιουνίου 2026 στις 8:00 μ.μ. UTCpostmortem

    ‌ **Date**: Jun 25, 2026 **Date of Incident:** Jun 22, 2026 **Description**: RCA for Third-Party CDN Network Disruption — Cloudflare / Zayo Fiber Cut ‌ #### **Summary:** On June 22, 2026, beginning at approximately 13:25 UTC, some JumpCloud customers experienced degraded access to web-facing services including the Admin Console, User Portal, API endpoints, and SSO authentication flows. Customers connecting through North America or accessing services routed through North American infrastructure were most affected. The root cause was a fiber cut in Eastern North America, a physical break in underground or undersea cables carrying internet traffic. Cloudflare attributed the disruption to Zayo, a network transit provider, experiencing an outage on some of its network routes, which caused reachability issues for services routing through those paths. Because JumpCloud's traffic transits the Cloudflare network before reaching JumpCloud's origin infrastructure, degradation at the Cloudflare/Zayo layer directly impacted our customers' ability to reach JumpCloud services. By late morning, most affected platforms had stabilized, with Cloudflare indicating its fix was being actively rolled out. JumpCloud's backend services and data plane remained fully operational throughout the event. The issue was isolated to the traffic path between end users and JumpCloud's origin infrastructure. ‌ #### **What Happened:** JumpCloud utilizes Cloudflare as the security and networking layer in front of all its global infrastructure. All HTTP/HTTPS requests to JumpCloud's web services traverse the Cloudflare network before reaching JumpCloud's origin infrastructure. ‌ #### **Provider Root Cause:** Cloudflare engineers first detected elevated error rates and latency at approximately 13:25 UTC on June 22, 2026. By 14:37 UTC, they had traced the root cause to a fiber cut in Eastern North America. Cloudflare confirmed that Zayo, a network provider, was experiencing an outage on some of its network routes, causing sites and services routing through those paths to become unreachable. Cloudflare's status page indicated that customers connecting through North America or accessing services in Europe may have seen increased latencies and timeouts as Cloudflare engineers worked to mitigate the issue. Traffic engineering efforts successfully mitigated the majority of congestion and packet drops, with services reported as largely stable with only minor residual impact remaining. Cloudflare's scheduled Newark \(EWR\) datacenter maintenance overlapped in timing but was a separate event - it was not the root cause of the fiber cut outage. ‌ #### **Corrective Actions \(Target End-of-July 2026\)**: While this incident was caused by a third-party provider we recognize the impact it had on our customers. JumpCloud is committed to reducing our exposure to single-provider failures and improving our resilience posture. The following corrective actions are actively underway: * **Multi-CDN Redundancy:** Cloudflare remains part of our architecture. We are adding a parallel, independently operated CDN path with weighted routing and security parity across both, so traffic can shift automatically if one path degrades. When complete, JumpCloud's edge will be capable of routing around a complete outage of any single CDN provider without customer action or service interruption. This work is in active delivery within our Platform Engineering organization. * **Resilient DNS Management:** We are modernizing how we manage DNS with phased, automated changes and rollback capabilities, reducing the risk that DNS operations themselves become a source of disruption. * **Validated Failover:** We are running controlled failover exercises including a production-scale test before relying on this redundancy in production. * **Enhanced Third-Party Monitoring & Automated Alerting:** We are expanding our monitoring to include automated detection of third-party provider degradation, enabling faster incident declaration and more rapid customer communication. This includes synthetic monitoring from multiple geographic regions that can differentiate between JumpCloud-origin issues and upstream provider issues ‌ #### **Provider Remediation:** Cloudflare has communicated the following remediation actions: * **Transit Path Restoration:** Cloudflare is coordinating with Zayo and fiber providers to repair the affected Eastern North American routes and restore full transit capacity. * **Traffic Engineering Response:** Cloudflare's traffic engineering teams implemented manual rerouting to redistribute load across unaffected network paths during the incident window. * **Ongoing Investigation:** Cloudflare has indicated a full post-incident review is underway to assess automated failover capabilities for large-scale transit provider failures. ‌ JumpCloud takes the reliability and availability of our platform seriously. We understand that our customers depend on JumpCloud for critical identity and device management operations, and any disruption, regardless of its origin, impacts their business. We are actively investing in infrastructure resilience to ensure that our dependency on any single third-party provider does not create an unacceptable risk to service availability.

1. 8 Μαΐου 2026 στις 6:41 π.μ. UTCinvestigating

   We are currently investigating delays or timeouts loading the devices page in the admin console. We are investigating the cause of the issues currently, and will provide an update within one hour.

2. 8 Μαΐου 2026 στις 6:44 π.μ. UTCidentified

   The issue has been identified and the team is working on the fix.

3. 8 Μαΐου 2026 στις 6:51 π.μ. UTCmonitoring

   A fix has been implemented. We are currently monitoring the results.

4. 8 Μαΐου 2026 στις 8:28 π.μ. UTCresolved

   This incident has been resolved.

1. 28 Απριλίου 2026 στις 8:01 π.μ. UTCinvestigating

   We are currently aware of reports with agent Installation failing, this is affecting MacOS, Windows and Linux. We are investigating the cause of the issues currently, and will provide an update within one hour.

2. 28 Απριλίου 2026 στις 8:57 π.μ. UTCidentified

   The issue has been identified and a fix will be implemented soon.

3. 28 Απριλίου 2026 στις 9:00 π.μ. UTCmonitoring

   A fix has been implemented, and Agent installations should now be functioning as expected. Our team is actively monitoring the situation to ensure continued stability.

4. 28 Απριλίου 2026 στις 9:55 π.μ. UTCresolved

   The incident has been resolved.

1. 2 Απριλίου 2026 στις 5:08 μ.μ. UTCinvestigating

   We are currently investigating a delay in LDAP directory synchronizations. While updates are processing, changes to user attributes and passwords may take longer than expected to propagate. The team is actively working to increase processing capacity and accelerate synchronizations.

2. 2 Απριλίου 2026 στις 5:16 μ.μ. UTCidentified

   The issue has been identified and a fix is being implemented.

3. 2 Απριλίου 2026 στις 5:23 μ.μ. UTCmonitoring

   A fix has been implemented and we are monitoring the results.

4. 2 Απριλίου 2026 στις 5:55 μ.μ. UTCresolved

   This incident has been resolved.

1. 31 Μαρτίου 2026 στις 12:32 π.μ. UTCidentified

   JumpCloud is currently experiencing dispatch delays for core Directory services. Administrators may experience delays updating associations for users, groups, policies, directories, and commands. We have identified the cause of the issue and are actively implementing a fix.

2. 31 Μαρτίου 2026 στις 2:38 π.μ. UTCidentified

   We have made progress in reducing the backlog affecting Directory services and are continuing to work through the remaining queue. Administrators may still experience delays when updating associations for users, groups, policies, directories, and commands. The team is actively implementing additional changes to increase processing capacity and accelerate resolution. We will provide further updates in one hour.

3. 31 Μαρτίου 2026 στις 3:42 π.μ. UTCidentified

   The backlog affecting Directory services has been significantly reduced and continues to decrease. Administrators may still experience delays when updating associations for users, groups, policies, directories, and commands. The team continues to work on additional changes to accelerate processing. We will provide further updates in an hour.

4. 31 Μαρτίου 2026 στις 5:00 π.μ. UTCmonitoring

   The backlog affecting Directory services continues to decrease and we are approaching resolution. Administrators may still experience some delays when updating associations. The team is continuing to monitor, and our next update will be to confirm full resolution.

5. 31 Μαρτίου 2026 στις 6:24 π.μ. UTCresolved

   This incident has been resolved.

6. 7 Απριλίου 2026 στις 2:56 μ.μ. UTCpostmortem

    **Date**: Apr 7, 2026 **Date of Incident:** Mar 30, 2026 **Description**: RCA for Directory Association Processing Delays ‌ **Summary:** Starting March 30th at approximately 15:40 MDT, JumpCloud customers experienced significant delays in directory-related updates. This included latency in password changes, user-to-group associations, and outbound provisioning reflecting in downstream systems. The root cause was identified as a specific code deployment in our Devices service that inadvertently flooded a background processing queue with unpartitioned messages, causing a bottleneck that prevented updates from processing in real-time. The issue was fully resolved by 00:25 MDT on March 31, 2026. ‌ **What Happened:** The incident was caused by a change in how the JumpCloud agent retrieves software application configurations. 1. **Traffic Spike:** The new code shifted the "source of truth" for these configurations to a new database. If a device polled the system and did not find its record in the new database, the code automatically enqueued a "track collect" request to sync the data. 2. **Unexpected Volume:** We anticipated a "lazy backfill" \(where records are created over time\), but underestimated the number of devices that had no existing software bindings. This resulted in an immediate, massive spike of nearly 280,000 messages. 3. **The Bottleneck \(Partitioning\):** Crucially, these specific messages were enqueued without a "Partition ID." In our high-scale FIFO \(First-In-First-Out\) queue architecture, messages without a partition ID are processed one-by-one rather than in parallel. This effectively "serialized" the queue, preventing us from scaling up workers to process the backlog faster and causing the observed latency. ‌ **Resolution and Recovery**: Once the offending code was rolled back, the "tap" was turned off, and no further unpartitioned messages were added to the queue. Because the bottleneck was caused by the lack of partitioning, simply scaling horizontally could not speed up the processing of the existing backlog. The team monitored the queue throughput and determined that the safest and fastest path to recovery was allowing the worker to process the existing messages sequentially rather than risking further disruption by attempting to manually manipulate the production queue. ‌ **Corrective Actions**: To ensure this type of bottleneck does not occur again, we have committed to the following: * Improving pre-production testing to better simulate the scale and conditions that can occur in production queue processing * Reviewing other areas of the platform where similar patterns could produce unexpected request spikes * Enhancing monitoring and alerting thresholds to enable faster detection and response when queue backlogs begin to form * Strengthening our deployment validation process to more thoroughly account for background data migrations before releasing dependent code changes

1. 12 Μαρτίου 2026 στις 4:54 μ.μ. UTCinvestigating

   We are currently investigating an issue with delays in syncing user, commands, and policy information with the JumpCloud Devices Agent backend service. We are investigating the cause of the issues currently, and will provide an update within one hour. Agent / Device logins are operating as expected.

2. 12 Μαρτίου 2026 στις 7:24 μ.μ. UTCinvestigating

   We are continuing to investigate an issue with delays in syncing user, commands, and policy information with the JumpCloud Devices Agent backend service. Local Agent / Device logins without MFA or leveraging TOTP are operating as expected.

3. 12 Μαρτίου 2026 στις 8:26 μ.μ. UTCidentified

   The issue has been identified and we have implemented a fix. We are starting to see some recovery with many agents checking in. We will update as our agent traffic normalizes.

4. 12 Μαρτίου 2026 στις 9:08 μ.μ. UTCmonitoring

   Agent traffic is reaching normal levels, and the majority of customers are seeing full service restoration. Our engineering teams continue to actively monitor backend stability and traffic patterns. We expect to move to 'Resolved' status shortly as final systems stabilize.

5. 12 Μαρτίου 2026 στις 9:48 μ.μ. UTCresolved

   This incident has been resolved.

6. 17 Μαρτίου 2026 στις 9:25 μ.μ. UTCpostmortem

    ‌ **Date**: Mar 17, 2026 **Date of Incident:** Mar 12, 2026 **Description**: RCA for Agent Backend \(HAProxy\) System Degradation ‌ **Summary:** On March 12, 2026, from 10:05 AM to 2:45 PM MDT, JumpCloud experienced a significant service degradation affecting Agent-related activities. During this window, agent updates, including syncing users, passwords, policies and other agent data, as well as new agent installations were unavailable. This was caused by a "thundering herd" event triggered by a backend traffic-shaping change. We have since identified the root causes and implemented infrastructure changes to prevent a recurrence. ‌ **What Happened?** At 10:00 AM MDT, our engineering team enabled a feature flag \(a "circuit breaker"\) designed to protect our System Insights API from high load by returning `503 Service Unavailable` responses for certain non-critical requests. While the flag performed its intended function, it had an unforeseen secondary effect on the JumpCloud Agent’s connection logic. Because the agents could not reuse existing connections for these specific failed requests, hundreds of thousands of agents in our main production environment attempted to establish new mTLS \(mutual TLS\) connections simultaneously. This created a "Thundering Herd" event that saturated our HAProxy ingress layer, exhausting CPU resources and causing a cascade of connection failures. ‌ **Root Cause:** The prolonged nature of this incident was the result of three distinct, overlapping bottlenecks that our team had to isolate and resolve one by one: 1. **CPU-Intensive SSL Handshaking:** Establishing an mTLS connection is a CPU-intensive process. The sheer volume of simultaneous connection attempts pushed our HAProxy pods to their resource limits. This caused the pods to become unresponsive, leading to "Out of Memory" \(OOM\) kills and failed health probes. 2. **Health Check Death Spiral:** Our internal health checks initially relied on a Layer 7 SSL validation. Because the CPU was 100% occupied with agent reconnections, the pods couldn't respond to their own health checks in time. This caused the system to erroneously mark healthy pods as "down”, removing them from the rotation and further overwhelming the remaining pods. 3. **Load Balancer Handshake Saturation:** As we attempted to scale our infrastructure, the Application Load Balancer \(ALB\) encountered a throughput bottleneck specifically related to the rate of new connection establishments. The surge of agents attempting to negotiate new SSL handshakes at the same time exceeded the ALB's burst capacity, temporarily preventing even healthy backend pods from receiving and processing traffic. ‌ **Why It Took Time to Resolve:** While reverting the flag was the correct first step, the agents were already in an aggressive retry loop that continued even after the 503 errors stopped.  We had to experiment with several configurations \(adjusting health check intervals and timeout windows\) to find a balance that allowed pods to stay "alive" long enough to process the backlog. Stability was achieved only once we implemented Concurrency Control. By lowering the maximum allowed concurrent connections per pod, we stopped the CPU from over-committing to handshakes, allowing the system to reliably process a controlled flow of traffic until the global queue cleared. ‌ **Corrective Actions / Risk Mitigation:** **1.\) Edge Infrastructure Hardening** We are standardized on a new high-availability configuration for our HAProxy ingress layer. * **Concurrency Governance**: We have implemented a strict maxconn limit per pod. This acts as a "pressure valve," ensuring that the CPU remains available to process existing requests rather than becoming saturated by new connection attempts. * **Dynamic Capacity Management via Autoscaling**: We are implementing Horizontal Pod Autoscaling \(HPA\) for our HAProxy ingress layer, calibrated to trigger based on both CPU utilization and active connection counts.  This ensures we can absorb sudden traffic fluctuations and also maintain a controlled flow of requests to our backend services. **2.\) Agent Connectivity Optimization** We are updating the JumpCloud Agent’s communication layer to be more "network-aware" during degraded states: * **Enhanced Connection Pooling**: We are reconfiguring the agent's HTTP transport logic to maximize the reuse of existing idle connections. This significantly reduces the "Connection Tax" on our backend during high-traffic events. * **Streamlined Resource Handling**: We are implementing stricter protocols for draining and closing HTTP response bodies, ensuring that pooled connections are returned to the rotation immediately and reliably. **3.\) Adaptive Retry Logic \(Jitter\)** To further break up "synchronized" traffic spikes: * **Introduction of Jitte**r: While our agents currently use exponential backoff for poll requests, we are adding randomized "jitter" to our retry intervals. This spreads reconnection attempts across a wider window, preventing large blocks of agents from hitting the service at the exact same millisecond. * **Standardizing Resilient Retry Logic:** We are transitioning the Agent’s default HTTP client to a unified **exponential backoff** model for all request types. * **Controlled Rollou**t: This update will be managed via a staged rollout to monitor for any unforeseen side effects on fleet-wide connectivity patterns.

1. 21 Νοεμβρίου 2025 στις 12:34 μ.μ. UTCinvestigating

   We had reports of delays or timeouts when accessing the Admin and User Portal consoles but now recovering. This issue has been traced to an ongoing network problem within the Asia region reported by Cloudflare. We are closely monitoring the situation and will provide an update as soon as new information becomes available.

2. 21 Νοεμβρίου 2025 στις 12:44 μ.μ. UTCmonitoring

   Per the latest update from CloudFlare, a fix has been implemented and results are being monitored. https://www.cloudflarestatus.com/incidents/5p8c3l54b5cl

3. 21 Νοεμβρίου 2025 στις 2:30 μ.μ. UTCresolved

   This incident has been resolved.

1. 19 Νοεμβρίου 2025 στις 6:46 π.μ. UTCinvestigating

   We're currently investigating reports of intermittent access issues with JumpCloud's Console Service for administrator at https://console.jumpcloud.com/. We are investigating the cause of the issues currently, and will update the status event once resolved.

2. 19 Νοεμβρίου 2025 στις 7:00 π.μ. UTCmonitoring

   A fix has been implemented and we are monitoring the results.

3. 19 Νοεμβρίου 2025 στις 7:16 π.μ. UTCresolved

   Issue is resolved and closed.

4. 21 Νοεμβρίου 2025 στις 6:25 μ.μ. UTCpostmortem

    ‌ **Date**: Nov 21, 2025 **Date of Incident:** Nov 19, 2025 **Description**: RCA for Admin Portal Login Errors ‌ **Summary:** On November 19, 2025, starting at approximately 04:30 UTC, between 1-5% of requests experienced intermittent failures to successfully authenticate to the Admin Console, lasting until roughly 06:30 UTC.  Users attempting to authenticate received an “unexpected” error message during this window, but subsequent retries may have been successful. **Root Cause:** This issue was triggered during a standard infrastructure update and traffic shift intended to move services to a new, updated cluster.  The core issue was a combination of an infrastructure configuration mismatch and gaps in our detection and validation processes. 1. Configuration Drift: The new infrastructure cluster \(Green Cluster\), intended to host the service, was missing a single but essential configuration value used by the control plane’s service mesh. This value had been recently applied to the existing cluster \(old cluster\) but was inadvertently excluded when the new cluster's baseline configuration was created and branched.  When production traffic began routing to the new cluster, the missing configuration caused some access components to fail, leading to the login errors. 2. Detection Gaps: The application logged the configuration failure as a _Warning_ message, rather than a critical error. This meant our automated monitoring system did not trigger an immediate alert or rollback when the issue first occurred. The team quickly isolated the issue to the new Green Cluster, and an emergency process was initiated to immediately revert all production traffic back to the stable old cluster. ‌ **Corrective Actions / Risk Mitigation:** 1. Automated Configuration Diff Check - Implementing an automated process to continuously compare and ensure 100% configuration parity between old and new production clusters during all transition phases. 2. Clear Rule Enforcement - Reinforcing and automating the process to ensure all configuration changes are applied consistently across all active and future clusters. 3. Multi-Layer Error Monitoring - Implementing error rate monitoring at every layer of the network and application stack to ensure no failure goes undetected.

1. 11 Νοεμβρίου 2025 στις 8:17 π.μ. UTCinvestigating

   We are currently investigating reports of intermittent issues with launching Remote Assist tool. We are investigating the cause of the issue currently, and will provide an update within one hour.

2. 11 Νοεμβρίου 2025 στις 8:40 π.μ. UTCidentified

   The issue has been identified and a fix is being implemented.

3. 11 Νοεμβρίου 2025 στις 9:12 π.μ. UTCmonitoring

   A fix has been implemented and we are monitoring the results.

4. 11 Νοεμβρίου 2025 στις 9:41 π.μ. UTCresolved

   This incident has been resolved.

1. 6 Νοεμβρίου 2025 στις 12:28 μ.μ. UTCinvestigating

   We are currently investigating an issue with the authentication on SSO OIDC applications. We have already identified the cause and will provide an update within one hour.

2. 6 Νοεμβρίου 2025 στις 12:33 μ.μ. UTCidentified

   The issue has been identified and a fix is being implemented.

3. 6 Νοεμβρίου 2025 στις 1:10 μ.μ. UTCmonitoring

   A fix has been implemented and we are monitoring the results.

4. 6 Νοεμβρίου 2025 στις 1:47 μ.μ. UTCresolved

   This incident has been resolved.

5. 13 Νοεμβρίου 2025 στις 11:33 μ.μ. UTCpostmortem

    ‌ **Date**: Nov 13, 2025 **Date of Incident:** Nov 6, 2025 **Description**: RCA for SSO/OIDC Service Degradation ‌ **Summary:** On November 6, 2025, starting at approximately 12:00 UTC, customers experienced failures to launch any application relying on JumpCloud's OIDC-based Single Sign-On \(SSO\), lasting for roughly one hour. ‌ **Root Cause:** The outage was caused by a combination of two errors during a scheduled compliance procedure: 1. Faulty Password Generation: Our automated system for rotating database passwords created a new credential that contained unsafe special characters. 2. Missing Special Character Logic: The entrypoint script for our core SSO service was missing logic to handle these special characters before using the password to construct a database connection string. When the SSO service attempted to restart and use the newly rotated password, the presence of the unsafe characters caused the connection string to be misinterpreted as invalid, leading to a parsing failure and service degradation. This issue stemmed from a latent configuration bug that was masked by prior rotation processes. Previously, database passwords were rotated manually using an older system \(`random_password` IAC resource\) which was explicitly configured to only generate alphanumeric characters. These characters are inherently safe in a URL context, so the underlying bug in the SSO service's connection logic was never exposed.  When the credential management was successfully migrated to the new, more robust rotation process, the new function began generating highly complex passwords, including special characters, for the first time. This immediately triggered the latent parsing flaw in the SSO service’s entrypoint script. ‌ **Corrective Actions / Risk Mitigation:** 1. Hardening code logic - All services that construct database connection strings will be audited and updated to explicitly encode the password component eliminating character misinterpretation. 2. Enhanced rotation alerting -  New monitoring and alerting dashboards are in place to track the health and success schedule of all automated credential rotation jobs, providing an immediate alert if a rotation creates an invalid credential. 3. Update password generation logic - The automated credential rotation function has been updated to explicitly generate passwords that are safe, avoiding complex, reserved characters.

1. 4 Νοεμβρίου 2025 στις 11:40 π.μ. UTCinvestigating

   We are seeing issues with SSO authentication. We are investigating this currently and will update within 1 hour

2. 4 Νοεμβρίου 2025 στις 11:47 π.μ. UTCinvestigating

   We are continuing to investigate this issue.

3. 4 Νοεμβρίου 2025 στις 12:20 μ.μ. UTCidentified

   We have identified an issue that is causing intermittent login issues with the JumpCloud User Portal and Admin Portal. We have also identified issues with JumpCloud MFA, LDAP, RADIUS, and authentication with SSO. We are working on implementing a fix and will provide another update as soon as possible.

4. 4 Νοεμβρίου 2025 στις 12:59 μ.μ. UTCidentified

   We continue to see intermittent issues with accessing the JumpCloud User and Admin Portal, MFA, LDAP, RADIUS, and SSO. During this time access attempts to LDAP and RADIUS are only impacted if a user is authenticating with MFA. Our team is working on implementing a fix and will provide another update as quickly as possible.

5. 4 Νοεμβρίου 2025 στις 1:17 μ.μ. UTCmonitoring

   We have implemented a fix and users should now be able to access the User and Admin Console, MFA, LDAP, RADIUS, and SSO without issue.  We will continue to monitor the results of the fix.

6. 4 Νοεμβρίου 2025 στις 2:19 μ.μ. UTCresolved

   Services have been fully restored and this incident has been resolved. We will provide a formal postmortem as a follow up.

7. 7 Νοεμβρίου 2025 στις 5:47 μ.μ. UTCpostmortem

    ‌ **Date**: Nov 7, 2025 **Date of Incident:** Nov 4, 2025 **Description**: RCA for Auth Database Degradation ‌ **Summary:** On November 4, 2025, a number of customers experienced intermittent failures, timeouts and increased latency when attempting to authenticate to multiple JumpCloud Services, including consoles, LDAP, RADIUS and SAML, or use Multi-Factor Authentication. ‌ **Root Cause:** The incident was triggered by an issue in the deployment process involving a database schema change and a subsequent application code release. During this deployment, a planned database change unintentionally removed several database indexes required by the existing application code. The sequence of failure was as follows: 1. Deployment Order Error: The database schema change \(which removed necessary indexes\) was applied to the production database before the new application code \(which did not require those indexes\) was deployed. 2. Performance Collapse: The existing, high-volume authentication code \(used for functions like TOTP and push authentication\) was forced to run against the now-inefficient database structure. Queries that normally took milliseconds suddenly took several seconds. 3. Connection Exhaustion: These slow queries held database connections open for extended periods, quickly overwhelming the database server's available connection pool. 4. Full Outage: With no available connections, the main authentication API could not communicate with the database, leading to 100% CPU utilization on the database server and triggering the intermittent timeouts and failures experienced by our customers. ‌ **Why Testing Did Not Catch This:** The issue was not identified during testing in our Development or Staging environments due to insufficient Load Simulation. The resource consumption issues and connection exhaustion only manifest under the extreme pressure of peak production traffic volume. The simulated load profiles in our lower environments were not sufficient to expose this specific failure mode. ‌ **Corrective Actions / Risk Mitigation:** 1. Mandatory schema change review - All database schema changes must now undergo an additional level of review to explicitly assess index dependencies and impact. 2. New deployment phasing - We are implementing new tools and checks to enforce that application code dependent on a schema change is deployed before a database change is executed. 3. Enhance alerting -  We are implementing new monitors and alerts specifically for the Auth-API's database connection pool health and CPU utilization. 4. Enhanced load testing - We are revisiting the load profiles used in our staging environments looking for opportunities to more accurately simulate peak production traffic.

1. 20 Οκτωβρίου 2025 στις 3:44 μ.μ. UTCinvestigating

   Due to an issue with our Cloud Provider, JumpCloud is experiencing intermittent issues with our PAM service and some Support tools which may affect case creation. We are investigating this with our provider and will update this incident as we know more.

2. 20 Οκτωβρίου 2025 στις 3:50 μ.μ. UTCidentified

   We are monitoring our services and continue to work with our provider to restore full functionality.

3. 20 Οκτωβρίου 2025 στις 5:43 μ.μ. UTCmonitoring

   We've seen an increase in recovery and are monitoring

4. 20 Οκτωβρίου 2025 στις 9:46 μ.μ. UTCresolved

   This incident has been resolved.

5. 27 Οκτωβρίου 2025 στις 7:21 μ.μ. UTCpostmortem

    **Date**: Oct 27, 2025 **Date of Incident:** Oct 20, 2025 **Description**: RCA for Service Degradation Linked to AWS US-EAST-1 Regional Disruption ‌ **Summary:** On October 20, 2025, between approximately 07:00 UTC and 10:00 UTC, the JumpCloud platform experienced significant performance degradation. This primarily affected the responsiveness of our core APIs, administrative console access, user console access, single sign-on \(SSO\) functions for customers, failed trial creation, user updates, and the potential loss of some Directory Insights events.  Residual performance degradation, affecting services like our Privileged Access Management \(PAM\) feature and specific Support Portal APIs, persisted until approximately 17:00 UTC, at which point all services were fully restored. ‌ The degradation was not caused by a failure within the JumpCloud platform code or infrastructure configuration, but was a direct consequence of a severe, cascading failure within the Amazon Web Services \(AWS\) US-EAST-1 region. Although many of our services are deployed across multiple Availability Zones \(AZs\) within the region for resilience, the nature of the AWS issue - impacting fundamental regional services - compromised inter-AZ communication preventing some of our standard failover mechanisms from operating successfully. Services returned to full operational status after AWS reported stability with the impacted foundational services. ‌ **Root Cause:** Based on the [post-incident analysis of AWS,](https://aws.amazon.com/message/101925/) the service disruption was not a single event but a sequence of cascading failures across three fundamental AWS services. ‌ 1. DynamoDB Failure Due to Latent DNS Race Condition \(Initial Trigger\) 2. EC2 Launch Congestion and Network Propagation Delays \(Sustained Impact\) 3. Network Load Balancer \(NLB\) Health Check Instability \(Final Phase\) ‌ All affected service teams were paged by our monitoring and alerting systems, and our incident management team was engaged to coordinate efforts and assess areas where we could throttle traffic to stabilize remaining capacity. ‌ ‌ **Corrective Actions / Risk Mitigation:** 1. While our current architecture mitigates some single-Availability Zone \(AZ\) failures, our primary focus is to eliminate single-region dependency entirely. JumpCloud is actively engaged in strategic engineering initiatives to further strengthen the platform's foundation. We are conducting a thorough review of cross-region dependencies and replication strategies to enhance our service's resilience against widespread environmental disruptions, always striving to meet the highest standards of availability.

1. 20 Οκτωβρίου 2025 στις 7:52 π.μ. UTCinvestigating

   Due to an issue with our Cloud Provider, JumpCloud is experiencing intermittent issues with multiple services. We are investigating this with our provider and will update this incident as we know more.

2. 20 Οκτωβρίου 2025 στις 9:11 π.μ. UTCidentified

   We are monitoring our services, and starting to see some recovery.

3. 20 Οκτωβρίου 2025 στις 10:02 π.μ. UTCmonitoring

   We are continuing to monitor as we see further recovery with all services.

4. 20 Οκτωβρίου 2025 στις 10:14 π.μ. UTCresolved

   This incident has been resolved.

5. 27 Οκτωβρίου 2025 στις 7:18 μ.μ. UTCpostmortem

    **Date**: Oct 27, 2025 **Date of Incident:** Oct 20, 2025 **Description**: RCA for Service Degradation Linked to AWS US-EAST-1 Regional Disruption ‌ **Summary:** On October 20, 2025, between approximately 07:00 UTC and 10:00 UTC, the JumpCloud platform experienced significant performance degradation. This primarily affected the responsiveness of our core APIs, administrative console access, user console access, single sign-on \(SSO\) functions for customers, failed trial creation, user updates, and the potential loss of some Directory Insights events.  Residual performance degradation, affecting services like our Privileged Access Management \(PAM\) feature and specific Support Portal APIs, persisted until approximately 17:00 UTC, at which point all services were fully restored. ‌ The degradation was not caused by a failure within the JumpCloud platform code or infrastructure configuration, but was a direct consequence of a severe, cascading failure within the Amazon Web Services \(AWS\) US-EAST-1 region. Although many of our services are deployed across multiple Availability Zones \(AZs\) within the region for resilience, the nature of the AWS issue - impacting fundamental regional services - compromised inter-AZ communication preventing some of our standard failover mechanisms from operating successfully. Services returned to full operational status after AWS reported stability with the impacted foundational services. ‌ **Root Cause:** Based on the [post-incident analysis of AWS,](https://aws.amazon.com/message/101925/) the service disruption was not a single event but a sequence of cascading failures across three fundamental AWS services. ‌ 1. DynamoDB Failure Due to Latent DNS Race Condition \(Initial Trigger\) 2. EC2 Launch Congestion and Network Propagation Delays \(Sustained Impact\) 3. Network Load Balancer \(NLB\) Health Check Instability \(Final Phase\) ‌ All affected service teams were paged by our monitoring and alerting systems, and our incident management team was engaged to coordinate efforts and assess areas where we could throttle traffic to stabilize remaining capacity. ‌ **Corrective Actions / Risk Mitigation:** 1. While our current architecture mitigates some single-Availability Zone \(AZ\) failures, our primary focus is to eliminate single-region dependency entirely. JumpCloud is actively engaged in strategic engineering initiatives to further strengthen the platform's foundation. We are conducting a thorough review of cross-region dependencies and replication strategies to enhance our service's resilience against widespread environmental disruptions, always striving to meet the highest standards of availability.