Pricing
Status historyRSS updatesStatus API

Powered by Uptimus

    Ιστορικό διακοπών του Xmatters

    54 Xmatters incidents · Νοέμβριος 2018 — official updates, affected components, duration and resolution details.

    Επιστροφή στην τρέχουσα κατάσταση

    Θέμα Ανακαλύπτεται - Διακοπή υπηρεσιών στην Ευρωπαϊκή Περιοχή - Πολλαπλές Υπηρεσίες

    Έναρξη 20 Οκτωβρίου 2026 στις 6:24 π.μ. UTC · 16h 4m

    IssuesΜικρό περιστατικό
    Επηρεαζόμενα στοιχεία
    Mobile AppAPIVoice NotificationsIntegration PlatformWeb InterfaceEmail NotificationsConferencingSMS Notifications
    1. 20 Οκτωβρίου 2026 στις 6:24 π.μ. UTCinvestigating

      τα εργαλεία παρακολούθησης xMatters έχουν εντοπίσει ένα πιθανό θέμα με το xMatters On-Demand για ορισμένους πελάτες που βρίσκονται στην περιοχή της Ευρώπης. Ερευνούμε επί του παρόντος το θέμα και θα ενημερώσουμε καθώς οι πληροφορίες θα είναι διαθέσιμες. Παρακαλούμε δείτε λεπτομέρειες περιστατικό για συγκεκριμένες υπηρεσίες που επηρεάζονται. Εάν επίσης αντιμετωπίζετε προβλήματα, ή αν δεν είστε σίγουροι αν αυτό το θέμα επηρεάζει την υπηρεσία σας, παρακαλούμε επικοινωνήστε με το xMatters Εξυπηρέτηση Πελατών στο https://support.xmatters.com/hc/en-us/quests/new - οι πράκτορες υποστήριξης μας περιμένουν να βοηθήσουν.

    2. 20 Οκτωβρίου 2026 στις 6:31 π.μ. UTCidentified

      Η ομάδα XMatters Incident Response έχει εντοπίσει την πηγή του ζητήματος και εργάζεται πάνω σε μια επιδιόρθωση. Θα ενημερώσουμε μόλις προσδιοριστεί και εφαρμοστεί μια λύση.

    3. 20 Οκτωβρίου 2026 στις 6:36 π.μ. UTCmonitoring

      Η ομάδα XMatters Incident Response έχει αναπτύξει μια λύση για το θέμα. Επί του παρόντος παρακολουθούμε την κατάσταση για να διασφαλίσουμε ότι η εφαρμογή είναι σταθερή και ότι όλες οι υπηρεσίες θα αποκατασταθούν.

    4. 20 Οκτωβρίου 2026 στις 10:27 μ.μ. UTCresolved

      Το θέμα έχει αντιμετωπιστεί, και όλες οι υπηρεσίες έχουν αποκατασταθεί. Σας ευχαριστώ για την υπομονή σας όσο ασχολούμαστε με αυτό το θέμα.

    5. 24 Οκτωβρίου 2026 στις 12:21 π.μ. UTCpostmortem

      Τι συνέβη; Στις 19 Οκτωβρίου 2026, τα εργαλεία εσωτερικής παρακολούθησης xMatters ανίχνευσαν την αποδόμηση της υπηρεσίας στις 10:03 μμ Ειρηνικού. Λίγο αργότερα, ορισμένοι πελάτες ανέφεραν ότι αντιμετωπίζουν προβλήματα πρόσβασης σε xMatters ή χρησιμοποιώντας υπηρεσίες ενσωμάτωσης και κοινοποίησης. ** Γιατί συνέβη; ** Το Google Cloud γνώρισε μια διακοπή που επηρέασε την περιοχή ΕΕ-Δύσης-2. Αυτό διέκοψε τις υπηρεσίες xMatters στην πληγείσα περιοχή. ** Πώς ανταποκριθήκαμε; ** Οι ομάδες xMatters Support, Engineering και Operations ξεκίνησαν τη διαδικασία διαχείρισης σημαντικών περιστατικών και μείωσαν τον αντίκτυπο της διακοπής από την αποτυχία των επηρεαζόμενων συστημάτων σε άλλη περιοχή. Οι υπηρεσίες έγιναν διαθέσιμες κατά τη μετάβαση των συστημάτων. Το xMatters παρέμεινε στην Παρακολούθηση και εξακριβωμένη σταθερότητα των υπηρεσιών ενώ η Google συνέχισε τις εργασίες ανάκαμψης στην πληγείσα περιοχή της. ** Τι κάνουμε για να μην ξανασυμβεί; ** Το xMatters έχει σχεδιαστεί με περιφερειακή ανθεκτικότητα για τον μετριασμό των επιπτώσεων των διακοπών του παρόχου υποδομής. Κατά τη διάρκεια αυτού του περιστατικού, τα εν λόγω μέτρα ανθεκτικότητας λειτουργούσαν όπως προβλεπόταν και επέτρεπαν τη μετάβαση των επηρεαζόμενων υπηρεσιών σε άλλη περιοχή. Η μηχανική και οι επιχειρήσεις επανεξετάζουν την απάντηση στην ανάλυση περιστατικών για τον εντοπισμό ευκαιριών για περαιτέρω εξορθολογισμό των διαδικασιών ανάκτησης και τη μείωση του χρόνου ανάκτησης κατά τη διάρκεια παρόμοιων γεγονότων.

    Αυτόματη μετάφραση από την επίσημη ενημέρωση του συμβάντος.

    Θέμα Ανακαλύψτηκε - Διακοπή υπηρεσιών στη Βόρεια Αμερική - Πολλαπλές Υπηρεσίες

    Έναρξη 1 Οκτωβρίου 2026 στις 5:22 μ.μ. UTC · 1h 33m

    OutageΣοβαρό περιστατικό
    Επηρεαζόμενα στοιχεία
    Mobile AppIntegration PlatformSMS NotificationsAPIConferencingEmail NotificationsVoice NotificationsWeb Interface
    1. 1 Οκτωβρίου 2026 στις 5:22 μ.μ. UTCinvestigating

      τα εργαλεία παρακολούθησης xMatters έχουν εντοπίσει ένα πιθανό ζήτημα με το xMatters On-Demand για ορισμένους πελάτες που βρίσκονται στην περιοχή της Βόρειας Αμερικής. Ερευνούμε επί του παρόντος το θέμα και θα ενημερώσουμε καθώς οι πληροφορίες θα είναι διαθέσιμες. Παρακαλούμε δείτε λεπτομέρειες περιστατικό για συγκεκριμένες υπηρεσίες που επηρεάζονται. Εάν επίσης αντιμετωπίζετε προβλήματα, ή αν δεν είστε σίγουροι αν αυτό το θέμα επηρεάζει την υπηρεσία σας, παρακαλούμε επικοινωνήστε με το xMatters Εξυπηρέτηση Πελατών στο https://support.xmatters.com/hc/en-us/quests/new - οι πράκτορες υποστήριξης μας περιμένουν να βοηθήσουν.

    2. 1 Οκτωβρίου 2026 στις 5:30 μ.μ. UTCinvestigating

      Συνεχίζουμε να ερευνούμε αυτό το ζήτημα.

    3. 1 Οκτωβρίου 2026 στις 5:40 μ.μ. UTCidentified

      Η ομάδα XMatters Incident Response έχει εντοπίσει την πηγή του ζητήματος και εργάζεται πάνω σε μια επιδιόρθωση. Θα ενημερώσουμε μόλις προσδιοριστεί και εφαρμοστεί μια λύση.

    4. 1 Οκτωβρίου 2026 στις 5:45 μ.μ. UTCmonitoring

      Η ομάδα XMatters Incident Response έχει αναπτύξει μια λύση για το θέμα. Επί του παρόντος παρακολουθούμε την κατάσταση για να διασφαλίσουμε ότι η εφαρμογή είναι σταθερή και ότι όλες οι υπηρεσίες θα αποκατασταθούν.

    5. 1 Οκτωβρίου 2026 στις 6:56 μ.μ. UTCresolved

      Το θέμα έχει αντιμετωπιστεί, και όλες οι υπηρεσίες έχουν αποκατασταθεί. Σας ευχαριστώ για την υπομονή σας όσο ασχολούμαστε με αυτό το θέμα.

    6. 13 Οκτωβρίου 2026 στις 8:12 μ.μ. UTCpostmortem

      Τι συνέβη; Την 1η Οκτωβρίου 2026, η παρακολούθηση xMatters εντόπισε για πρώτη φορά την υποβάθμιση της υπηρεσίας στις 9:27 π.μ. του Ειρηνικού, και η πρώτη αναφορά πελατών ελήφθη στις 10:03 π.μ. Οι πελάτες στην US-EAST1 παρουσίασαν διαλείποντα θέματα πρόσβασης στο Web Interface, χρονόμετρα API, και καθυστερήσεις ή αποτυχίες όταν οι ενσωματώσεις ξεκίνησαν ειδοποιήσεις και ειδοποιήσεις. Ορισμένοι πελάτες στην US-CENTRAL1 μπορεί επίσης να έχουν βιώσει προβλήματα φωνητικής ειδοποίησης κατά την ίδια περίοδο. ** Γιατί συνέβη; ** Κατά τη διάρκεια της αυτοματοποιημένης κλιμάκωσης των υποδομών στις ΗΠΑ-ΕΑΣΤ1, οι πολλαπλές υπηρεσίες εφαρμογών συγκεντρώθηκαν στον ίδιο πόρο υπολογισμού. Ο εν λόγω πόρος έφθασε στην ικανότητα επεξεργασίας του, μειώνοντας τη διαθέσιμη ικανότητα για εξυπηρέτηση των αιτημάτων των πελατών. Η αποικοδόμηση των υπηρεσιών συνεχίστηκε σε συνάρτηση με τις λειτουργίες xMatters ενώ προστέθηκε επιπλέον ικανότητα επεξεργασίας. ** Πώς ανταποκριθήκαμε; ** Μόλις ο πρώτος πελάτης ανέφερε ένα θέμα, η υποστήριξη πελατών xMatters ξεκίνησε την εσωτερική διαδικασία διαχείρισης σημαντικών περιστατικών και ενεπλάκη στις ομάδες Μηχανικών και Επιχειρήσεων. Η ομάδα αντιμετώπισης συμβάντων αύξησε την ικανότητα εξυπηρέτησης, παρακολούθησε τις πληγείσες υπηρεσίες, και επαλήθευσε την ανάκτηση σε όλη τη Διεπαφή Ιστού, APIs, ενσωματώσεις, και επεξεργασία ειδοποιήσεων. Η υπηρεσία άρχισε να σταθεροποιείται μέχρι τις 10:37 π.μ., και ο κύριος αντίκτυπος του πελάτη τελείωσε στις 10:46 π.μ. Η ομάδα συνέχισε την παρακολούθηση πριν επιλύσει το δημόσιο περιστατικό στις 11:56 πμ. ** Τι κάνουμε για να μην ξανασυμβεί; ** Η μηχανική επανεξετάζει την ικανότητα των υπηρεσιών και τους ελέγχους διανομής φόρτου εργασίας για να μειώσει τον κίνδυνο πολλαπλών κρίσιμων υπηρεσιών που ανταγωνίζονται για τους ίδιους πόρους υπολογισμού. Η ομάδα επανεξετάζει επίσης τους υγειονομικούς ελέγχους των υπηρεσιών και την προειδοποίηση, έτσι ώστε τα ζητήματα χωρητικότητας να προσδιορίζονται νωρίτερα για να εξασφαλιστεί η απαιτούμενη χωρητικότητα είναι πάντα διαθέσιμη.

    Αυτόματη μετάφραση από την επίσημη ενημέρωση του συμβάντος.

    Το Θέμα Ανακαλύφθηκε - Διαταραχή στην Υπηρεσία σε όλες τις περιφέρειες – Ειδοποίηση SMS στις ΗΠΑ και τον Καναδά

    Έναρξη 21 Σεπτεμβρίου 2026 στις 6:14 μ.μ. UTC · 24d 21h

    OutageΣοβαρό περιστατικό
    Επηρεαζόμενα στοιχεία
    SMS NotificationsSMS NotificationsSMS Notifications
    1. 21 Σεπτεμβρίου 2026 στις 6:14 μ.μ. UTCinvestigating

      το xMatters έχει εντοπίσει ένα θέμα με το xMatters SMS Notification για ορισμένους πελάτες σε όλες τις περιφέρειες σε αριθμούς στις Ηνωμένες Πολιτείες και τον Καναδά από τον σύντομο κωδικό μας 87961. Αποτύχαμε στον δευτερεύοντα σύντομο κωδικό μας 66936. Ερευνούμε επί του παρόντος το θέμα και θα ενημερώσουμε καθώς οι πληροφορίες θα είναι διαθέσιμες. Εάν επίσης αντιμετωπίζετε προβλήματα, ή αν δεν είστε σίγουροι αν αυτό το θέμα επηρεάζει την υπηρεσία σας, παρακαλούμε επικοινωνήστε με το xMatters Εξυπηρέτηση Πελατών στο https://support.xmatters.com/hc/en-us/quests/new - οι πράκτορες υποστήριξης μας περιμένουν να βοηθήσουν.

    2. 21 Σεπτεμβρίου 2026 στις 6:41 μ.μ. UTCmonitoring

      Όλη η κυκλοφορία SMS στις ΗΠΑ και τον Καναδά παραδίδεται τώρα μέσω του δευτερεύοντος σύντομου κωδικού μας 66936. Επί του παρόντος παρακολουθούμε την κατάσταση για να διασφαλίσουμε ότι η εφαρμογή είναι σταθερή και ότι όλες οι υπηρεσίες θα αποκατασταθούν.

    3. 16 Οκτωβρίου 2026 στις 4:03 μ.μ. UTCresolved

      Το σύντομο ζήτημα κώδικα έχει αντιμετωπιστεί, και όλες οι ειδοποιήσεις SMS προς τον Καναδά και τις Ηνωμένες Πολιτείες τώρα παραδίδονται από τον 87961 σύντομο κωδικό μας. Σας ευχαριστώ για την υπομονή σας όσο ασχολούμαστε με αυτό το θέμα.

    4. 16 Οκτωβρίου 2026 στις 4:24 μ.μ. UTCpostmortem

      Τι συνέβη; Στις 21 Σεπτεμβρίου 2026, οι πελάτες ανέφεραν ένα θέμα στο xMatters Client Assistance όπου τα μηνύματα SMS αποτυγχάνουν να στείλουν σε τελικούς χρήστες στη Βόρεια Αμερική. ** Γιατί συνέβη; ** Οι αποτυχίες παράδοσης SMS συνέβησαν επειδή ο σύντομος κωδικός που χρησιμοποιήθηκε για την αποστολή μηνυμάτων SMS δεν ήταν διαθέσιμος. Το σύστημα θα πρέπει να έχει δρομολογήσει αυτόματα αποτυχημένα μηνύματα μέσω του δευτερεύοντος παρόχου μηνυμάτων. Ωστόσο, το συγκεκριμένο σφάλμα που επέστρεψε για αυτές τις αποτυχίες δεν προκάλεσε τη ρυθμισμένη διαδικασία αποτυχίας, προκαλώντας διατάραξη της παράδοσης SMS σε τελικούς χρήστες στις ΗΠΑ και τον Καναδά. ** Πώς ανταποκριθήκαμε; ** Μόλις η Μηχανική εντόπισε το θέμα παράδοσης SMS, ξεκίνησαν τη διαδικασία Διαχείρισης Μείζονος Περιστατικού και άρχισαν να αφαιρούν τον επηρεασμένο σύντομο κώδικα από τη διαμόρφωση μηνυμάτων. Κατά τη διάρκεια αυτής της διαδικασίας, η Μηχανική αντιμετώπισε πρόσθετα ζητήματα που εμπόδιζαν το σύστημα να μεταβάλλει αυτόματα την κυκλοφορία SMS στον δευτερεύοντα πάροχο μηνυμάτων. Το μηχανοστάσιο παρενέβη χειροκίνητα για να ολοκληρώσει το διακόπτη προς τον δευτερεύοντα πάροχο και να αποκαταστήσει την παράδοση SMS. Μετά την ανάκτηση της υπηρεσίας, η Μηχανική παρακολούθησε το σύστημα για να επιβεβαιώσει ότι η αποστολή SMS λειτουργούσε κανονικά. ** Τι κάνουμε για να μην ξανασυμβεί; ** Μηχανική δρομολόγησε την κυκλοφορία SMS μέσω του δευτερεύοντος παρόχου μηνυμάτων, ενώ εργάστηκαν για την αποκατάσταση του προηγούμενου πρωτεύοντος σύντομου κώδικα. Η μηχανική αντιμετωπίζει επίσης τη συμπεριφορά αποτυχίας που εμπόδισε το σύστημα να αλλάξει αυτόματα τους παρόχους όταν συνέβη αυτό το συγκεκριμένο σφάλμα. Αυτές οι βελτιώσεις στη διαδικασία θα μειώσουν τον κίνδυνο μιας παρόμοιας έκδοσης σύντομου κώδικα που θα διαταράξει την παράδοση SMS στο μέλλον.

    Αυτόματη μετάφραση από την επίσημη ενημέρωση του συμβάντος.

    Θέμα Ανακαλύψτηκε - Διακοπή υπηρεσιών στη Βόρεια Αμερική - Πολλαπλές Υπηρεσίες

    Έναρξη 1 Ιουλίου 2026 στις 7:53 μ.μ. UTC · 28m

    IssuesΜικρό περιστατικό
    Επηρεαζόμενα στοιχεία
    Mobile AppIntegration PlatformSMS NotificationsAPIConferencingEmail NotificationsVoice NotificationsWeb Interface
    1. 1 Ιουλίου 2026 στις 7:53 μ.μ. UTCinvestigating

      τα εργαλεία παρακολούθησης xMatters έχουν εντοπίσει ένα θέμα με το xMatters On-Demand για ορισμένους πελάτες που βρίσκονται στην περιοχή της Βόρειας Αμερικής, που περιορίζεται σε μια συστάδα βάσεων δεδομένων. Ερευνούμε επί του παρόντος το θέμα και θα ενημερώσουμε καθώς οι πληροφορίες θα είναι διαθέσιμες. Εάν επίσης αντιμετωπίζετε προβλήματα, ή αν δεν είστε σίγουροι αν αυτό το θέμα επηρεάζει την υπηρεσία σας, παρακαλούμε επικοινωνήστε με το xMatters Εξυπηρέτηση Πελατών στο https://support.xmatters.com/hc/en-us/quests/new - οι πράκτορες υποστήριξης μας περιμένουν να βοηθήσουν.

    2. 1 Ιουλίου 2026 στις 8:07 μ.μ. UTCmonitoring

      Η ομάδα XMatters Incident Response έχει αναπτύξει μια λύση για το θέμα. Επί του παρόντος παρακολουθούμε την κατάσταση για να διασφαλίσουμε ότι η εφαρμογή είναι σταθερή και ότι όλες οι υπηρεσίες θα αποκατασταθούν.

    3. 1 Ιουλίου 2026 στις 8:21 μ.μ. UTCresolved

      Το θέμα έχει αντιμετωπιστεί, και όλες οι υπηρεσίες έχουν αποκατασταθεί. Σας ευχαριστώ για την υπομονή σας όσο ασχολούμαστε με αυτό το θέμα.

    4. 9 Ιουλίου 2026 στις 7:21 μ.μ. UTCpostmortem

      Τι συνέβη; Την 1η Ιουλίου 2026, ορισμένοι πελάτες ανέφεραν ένα θέμα στην υποστήριξη πελατών xMatters όπου συναντούσαν σφάλματα ταυτοποίησης κατά την πρόσβαση στη διασύνδεση χρήστη web. ** Γιατί συνέβη; ** Αυτό το θέμα συνέβη επειδή ένα σύμπλεγμα βάσεων δεδομένων έφτασε το 100% της χρήσης πόρων, προκαλώντας αιτήματα βάσης δεδομένων για χρονικό διάστημα. Ένα εξαιρετικό μοτίβο κυκλοφορίας εξέθεσε ένα σύνολο από αργοπορημένα ερωτήματα, το οποίο αύξησε σημαντικά τη λανθάνουσα λανθάνουσα ζήτησι για τη συστάδα. Καθώς η καθυστέρηση αυξανόταν, υπήρχε λιμοκτονία των πόρων και περαιτέρω υποβάθμιση της ανταπόκρισης του συμπλέγματος, με αποτέλεσμα τα χρονικά όρια. ** Πώς ανταποκριθήκαμε; ** Μόλις η Υποστήριξη Πελατών επιβεβαίωσε το ζήτημα, ενεπλάκησαν τις ομάδες Μηχανικών να ερευνήσουν. Όπως σχεδιάστηκε, το σύστημα αποκαταστάθηκε πριν η Μηχανική μπορέσει να εφαρμόσει οποιαδήποτε δράση μετριασμού ή ανάκτησης, αλλά οι ομάδες ήταν ακόμα σε θέση να εντοπίσουν και να απομονώσουν τη βασική αιτία. ** Τι κάνουμε για να μην ξανασυμβεί; ** Για να αποτραπεί η επανεμφάνιση αυτού του ζητήματος, η Ομάδα Μηχανικών σχεδίασε, ανέπτυξε και δοκίμασε μια επιδιόρθωση. Ανέπτυξαν τη δόση το πρωί της Τετάρτης 8 Ιουλίου 2026, και έχει εφαρμοστεί επιτυχώς για όλες τις περιπτώσεις.

    Αυτόματη μετάφραση από την επίσημη ενημέρωση του συμβάντος.

    Issue Discovered - Service disruption in Europe Region – Web User Interface

    Έναρξη 1 Μαΐου 2026 στις 8:45 π.μ. UTC · 34m

    OutageΣοβαρό περιστατικό
    Επηρεαζόμενα στοιχεία
    Web Interface
    1. 1 Μαΐου 2026 στις 8:45 π.μ. UTCinvestigating

      xMatters have identified a potential issue with the xMatters Web User Interface for some clients located in the Europe region. Accessing some pages including Alert details produces an error. We are currently investigating the issue and will update as information becomes available. If you are also experiencing issues, or if you're not sure whether this issue impacts your service, please contact xMatters Client Assistance at https://support.xmatters.com/hc/en-us/requests/new - our support agents are waiting to help.

    2. 1 Μαΐου 2026 στις 9:01 π.μ. UTCidentified

      The xMatters Incident Response team has identified the source of the issue and is working on a fix. We will update once a solution has been identified and implemented.

    3. 1 Μαΐου 2026 στις 9:03 π.μ. UTCmonitoring

      The xMatters Incident Response team has deployed a fix for the issue. We are currently monitoring the situation to ensure the implementation is stable and that all services are restored.

    4. 1 Μαΐου 2026 στις 9:19 π.μ. UTCresolved

      The issue has been addressed, and all services have been restored. Thank you for your patience while we addressed this matter.

    5. 12 Μαΐου 2026 στις 8:32 μ.μ. UTCpostmortem

      **What happened?**  On May 1st, 2026, some customers reported an issue to xMatters Customer Support where attempting to send a message via the web user interface or viewing an alert on the Alerts report resulted in an error being displayed. The issue only affected the Reporting functions in the EMEA region; the system continued to accept signals, generate alerts, and send notifications across all regions.  **Why did it happen?**  The issue occurred when, during routine database maintenance, the database called a mismatched version of the library, resulting in an internal database error. The version mismatch within the cluster was traced to a prior database engine upgrade where a subset of replica nodes did not restart into the upgraded version. At no point was there any risk to data integrity.  **How did we respond?**  As soon as Customer Support confirmed the issue, they engaged the Engineering teams, who were able to identify the root cause and restore version consistency across all nodes. The teams validated stability and confirmed that all services were restored.  **What are we doing to prevent it from happening again?**  The Engineering Team has added explicit post-upgrade verification checks and monitoring to ensure node alignment is confirmed and maintained. This will provide safeguards to ensure node version alignment after upgrades and prevent this issue from reoccurring.

    Issue Discovered - Service disruption in North American Region – API

    Έναρξη 1 Απριλίου 2026 στις 2:06 μ.μ. UTC · 30m

    IssuesΜικρό περιστατικό
    Επηρεαζόμενα στοιχεία
    API
    1. 1 Απριλίου 2026 στις 2:06 μ.μ. UTCinvestigating

      xMatters monitoring tools have identified a potential issue with the API for some clients located in the North America region. We are currently investigating the issue and will update as information becomes available. If you are also experiencing issues, or if you're not sure whether this issue impacts your service, please contact xMatters Client Assistance at https://support.xmatters.com/hc/en-us/requests/new - our support agents are waiting to help.

    2. 1 Απριλίου 2026 στις 2:30 μ.μ. UTCidentified

      The xMatters Incident Response team has identified an issue with the API for some clients located in the North America region which is causing some intermittent API timeouts. This is also causing issues with running reports in the system. We are currently investigating the issue and will update as information becomes available.

    3. 1 Απριλίου 2026 στις 2:33 μ.μ. UTCmonitoring

      The xMatters Incident Response team has deployed a fix for the issue. We are currently monitoring the situation to ensure the implementation is stable and that all services are restored.

    4. 1 Απριλίου 2026 στις 2:36 μ.μ. UTCresolved

      The issue has been addressed, and all services have been restored. Thank you for your patience while we addressed this matter.

    5. 6 Απριλίου 2026 στις 11:21 μ.μ. UTCpostmortem

      **What happened?**  On April 1st, 2026, some customers reported an issue to xMatters Customer Support where the Alerts or Notifications reports were timing out and failing to load.  **Why did it happen?**  This issue occurred because a backend service was experiencing significant unexpected load that caused report processing to be delayed. The ongoing resource constraint resulted in timeouts.  **How did we respond?**  As soon as customers reported an issue, Customer Support launched an investigation and escalated to the Engineering teams. The teams initiated rolling restart procedures for the applicable backend services to restore functionality. Once the rolling restart was completed, service was fully restored.  **What are we doing to prevent it from happening again?**  The Engineering teams are currently working to identify any possible bottlenecks that may have caused performance issues. In the interim, they have increased and adjusted resource allocation for several services to handle potential processing delays and prevent potential recurrences.

    Issue Discovered - Service disruption in All Regions – SSO login to Web User Interface

    Έναρξη 10 Μαρτίου 2026 στις 8:36 μ.μ. UTC · 17m

    IssuesΜικρό περιστατικό
    Επηρεαζόμενα στοιχεία
    Web InterfaceWeb InterfaceWeb Interface
    1. 10 Μαρτίου 2026 στις 8:36 μ.μ. UTCinvestigating

      We have reports that some users are not able to login to xMatters using SSO to login to the xMatters Web User Interface in All Regions. We are currently investigating the issue and will update as information becomes available. Native Login still works to bypass SSO. If you are also experiencing issues, or if you're not sure whether this issue impacts your service, please contact xMatters Client Assistance at https://support.xmatters.com/hc/en-us/requests/new - our support agents are waiting to help.

    2. 10 Μαρτίου 2026 στις 8:51 μ.μ. UTCidentified

      The xMatters Incident Response team believes they have identified the source of the issue and is working on a fix. We will update once a solution has been identified and implemented.

    3. 10 Μαρτίου 2026 στις 8:54 μ.μ. UTCresolved

      The issue has been addressed, and all services have been restored. Thank you for your patience while we addressed this matter.

    4. 20 Μαρτίου 2026 στις 6:01 μ.μ. UTCpostmortem

      **What happened?**  On March 10th, 2026, some customers reported an issue to xMatters Customer Support where they were encountering a 404 error page when attempting to log in to their instances via SSO. Some users may also have encountered a “We’ve run into a problem while retrieving your data.” error message.  **Why did it happen?**  This issue occurred during a routine maintenance update to the xMatters platform. Although several components of the platform were updated, specific configurations of the component related to SSO-based authentication conflicted with the update and resulted in 404 errors. This issue was limited to those few customers that had specific criteria set for their SSO configuration.  **How did we respond?**  As soon as customers reported the issue, Customer Support verified the issue and escalated immediately to Engineering. The team traced the issue to the maintenance deployment and initiated rollback procedures to restore functionality. Once the rollback was completed, the 404 errors ceased and service was confirmed restored.  **What are we doing to prevent it from happening again?**  The Engineering teams have implemented additional testing for any configuration criteria related to the SSO-based authentication component. In addition, they have begun working on an improved maintenance plan to prevent further issues that could occur during similar deployments.

    Issue Discovered - Service disruption in North American Region - Multiple Services

    Έναρξη 21 Νοεμβρίου 2025 στις 1:54 μ.μ. UTC · 8m

    IssuesΜικρό περιστατικό
    Επηρεαζόμενα στοιχεία
    Mobile AppIntegration PlatformSMS NotificationsAPIConferencingEmail NotificationsVoice NotificationsWeb Interface
    1. 21 Νοεμβρίου 2025 στις 1:54 μ.μ. UTCinvestigating

      xMatters monitoring tools have identified a potential issue with xMatters On-Demand for some clients located in the North America region. This is affecting users located near the Asia Pacific Region attempting to access xMatters. We are currently investigating the issue and will update as information becomes available. Please see incident details for specific services impacted. If you are also experiencing issues, or if you're not sure whether this issue impacts your service, please contact xMatters Client Assistance at https://support.xmatters.com/hc/en-us/requests/new - our support agents are waiting to help.

    2. 21 Νοεμβρίου 2025 στις 1:55 μ.μ. UTCinvestigating

      We are continuing to investigate this issue.

    3. 21 Νοεμβρίου 2025 στις 2:01 μ.μ. UTCresolved

      The issue has been addressed, and all services have been restored. Thank you for your patience while we addressed this matter.

    4. 27 Νοεμβρίου 2025 στις 6:42 μ.μ. UTCpostmortem

      **What happened?** On November 21, 2025, at 12:50 PM UTC, the xMatters internal monitoring tools detected irregular behavior in how internal traffic was being routed. Some customers in the APAC region communicating with services in North America \(specifically US-East\) may have encountered intermittent request failures or increased latency. Only traffic between these two regions was affected; all other systems and regions continued normal operations. **Why did it happen?** A temporary network disruption between Australia Southeast and US-East caused one internal routing node in Australia to lose accurate information about available backend systems in USEast. The node generated an incomplete routing configuration and temporarily stopped directing traffic to US-East. Under normal circumstances, routing updates refresh automatically when connectivity returns. In this case, the affected node did not recover cleanly and remained in a stale state until Engineering intervened. **How did we respond?** As soon as Engineering was alerted through internal monitoring, they engaged with the platform engineering team, service owners and Customer Support to launch an investigation. The teams reached out to impacted customers to validate issue symptoms and restarted routing components in both affected regions to force a configuration refresh. Once the restart completed, routing returned to normal levels while the teams continued to monitor and investigate the root cause. They were able to confirm that only one routing node and specific cross-region traffic was impacted. **What are we doing to prevent it from happening again?** While teams were mitigating this issue, they created new alerting rules to detect the routing patterns they observed during the incident and expanded internal monitoring to help identify when routing nodes fail to refresh their configuration or otherwise enter a ‘stale’ state. The teams also have planned and prepared infrastructure updates that will further reduce the risk of similar issues. These include improved configuration recovery behavior, enhanced stability for routing components, and additional logging and observability improvements for diagnosing routing anomalies. They will deploy these updates once the current code freeze window has elapsed.

    Issue Discovered - Service disruption in All Regions – Conferencing

    Έναρξη 20 Οκτωβρίου 2025 στις 10:20 π.μ. UTC · 1d 6h

    OutageΣοβαρό περιστατικό
    Επηρεαζόμενα στοιχεία
    Voice NotificationsVoice NotificationsConferencingConferencingConferencingVoice Notifications
    1. 20 Οκτωβρίου 2025 στις 10:20 π.μ. UTCidentified

      xMatters monitoring tools have identified a issue with xMatters Conferencing and Voice calls for clients in All Regions. The issue is due to downstream provider outages.

    2. 20 Οκτωβρίου 2025 στις 12:23 μ.μ. UTCmonitoring

      Our downstream providers have deployed a fix and testing shows conferencing and voice systems are now working as expected. We are currently monitoring the situation to ensure the implementation is stable and that all services are restored.

    3. 21 Οκτωβρίου 2025 στις 4:34 μ.μ. UTCresolved

      Our provider has confirmed the issue has been addressed, and all services have been restored. Thank you for your patience while we addressed this matter.

    4. 29 Οκτωβρίου 2025 στις 9:52 μ.μ. UTCpostmortem

      **What happened?**  On October 20th, 2025, at approximately 1:21 AM Pacific, customers began reporting an issue affecting live call routing, conferences, and voice notifications. During this issue, customers in all regions would have been affected. **Why did it happen?**  This issue was caused by a global AWS outage that impacted one of our downstream providers, resulting in voice notifications, live call routing, and conferencing they were handling to fail.  **How did we respond?**  As soon as the first customer reported an issue, Customer Support engaged the Engineering teams and launched an investigation. Once they identified the root cause, the teams updated the primary provider for affected regions to an alternate provider that was not affected by the AWS outage. When the new provider was assigned, customers reported that all services were operating correctly.  **What are we doing to prevent it from happening again?**  While this issue was out of our control or that of our provider, we are working to identify potential ways to improve resilience in case of external factors such as this.

    Issue Discovered - Degraded performance in North American Region – Integration Platform

    Έναρξη 10 Οκτωβρίου 2025 στις 10:58 μ.μ. UTC · 3d 17h

    IssuesΜικρό περιστατικό
    Επηρεαζόμενα στοιχεία
    Integration Platform
    1. 10 Οκτωβρίου 2025 στις 10:58 μ.μ. UTCmonitoring

      The xMatters monitoring tools have alerted Customer Support to a potential issue with the integration platform in the North America region. Our technical teams have identified intermittent request failures and occasional slowdowns, though the service appears to be responsive and recovering correctly. We are continuing to monitor the situation and mitigating where possible, though some users may encounter a 5xx response code when submitting a request. If you are also experiencing issues, or if you're not sure whether this issue impacts your service, please contact xMatters Client Assistance at https://support.xmatters.com/hc/en-us/requests/new - our support agents are waiting to help.

    2. 14 Οκτωβρίου 2025 στις 3:58 μ.μ. UTCresolved

      The issue has been addressed, and all services are running as expected. Thank you for your patience while we addressed this matter.

    3. 31 Οκτωβρίου 2025 στις 6:48 μ.μ. UTCpostmortem

      **What happened?**  On October 10th, 2025, at approximately 4:14 PM Pacific, the xMatters internal monitoring tools alerted Customer Support to service degradation related to an issue that was already being internally monitored with the Integration Platform in the North American region. While this issue was being investigated and mitigated, customers may have experienced intermittent request failures and occasional slowdowns.  **Why did it happen?**  These particular issues were caused by a security update that caused conflicts with the underlying runtime environment, specifically the memory management routine. Slow performance of the routine was triggering frequent health checks for a request processing service and causing automatic restarts.  **How did we respond?**  As soon as the internal monitoring tools alerted Engineering to a potential issue, they began performing manual rolling restarts and deployed more forgiving liveness checks to avoid increasing error rates and to minimize any potential impact to customers. They also increased resources for the impacted service to improve responsiveness of the underlying routine and deployed a configuration fix to the Http Client Cache to try and stabilize the system. While mitigating the potential impact, they also increased monitoring levels as they continued to investigate the root cause and were able to deploy an update to the service and environment configuration on October 13 that resolved the issue. They continued monitoring and confirmed that the system was stable and all services were operational.  **What are we doing to prevent it from happening again?**  Engineering has updated the backend service and deployed additional updates and monitoring to the system configuration that will improve overall stability for the environment and prevent this issue from reoccurring.

    Issue Discovered - Service disruption in All Regions – Mobile App

    Έναρξη 24 Σεπτεμβρίου 2025 στις 3:59 π.μ. UTC · 44m

    IssuesΜικρό περιστατικό
    Επηρεαζόμενα στοιχεία
    Mobile AppMobile AppMobile App
    1. 24 Σεπτεμβρίου 2025 στις 3:59 π.μ. UTCinvestigating

      xMatters identified a issue with the Apple iOS Push Notifications for clients in All Regions. We are currently investigating the issue and will update as information becomes available. If you are also experiencing issues, or if you're not sure whether this issue impacts your service, please contact xMatters Client Assistance at https://support.xmatters.com/hc/en-us/requests/new - our support agents are waiting to help.

    2. 24 Σεπτεμβρίου 2025 στις 4:19 π.μ. UTCidentified

      xMatters has identified the issue with the Apple Push notifications for clients located in the in All Regions. We are currently investigating a solution and will update as information becomes available. If you are also experiencing issues, or if you're not sure whether this issue impacts your service, please contact xMatters Client Assistance at https://support.xmatters.com/hc/en-us/requests/new - our support agents are waiting to help.

    3. 24 Σεπτεμβρίου 2025 στις 4:32 π.μ. UTCidentified

      We are continuing to work on a fix for this issue.

    4. 24 Σεπτεμβρίου 2025 στις 4:34 π.μ. UTCinvestigating

      xMatters has identified the issue with the Apple Push notifications for clients located in the in All Regions. We are currently deploying a solution and will update as information becomes available. Asia Pacific customers can now receive Apple push notifications.

    5. 24 Σεπτεμβρίου 2025 στις 4:43 π.μ. UTCresolved

      The issue has been addressed, and Apple Push notifications have been restored in all regions. Thank you for your patience while we addressed this matter.

    6. 15 Οκτωβρίου 2025 στις 6:16 π.μ. UTCpostmortem

      **What happened?**  On September 23rd, 2025, at approximately 8:31 PM Pacific, the xMatters internal monitoring tools alerted the xMatters Support Team to an issue with Apple iOS notifications. Users were not receiving notifications on Apple iOS devices; other device types and alert and response processing were not impacted and continued to operate without interruption.  **Why did it happen?**  The issue occurred because of a backend service patch applied by the Engineering teams that affected necessary dependencies for Apple Push notification delivery.  **How did we respond?**  As soon as the monitoring tools alerted the Support team, they engaged the Engineering teams to launch an investigation. The teams quickly identified the source of the issue as a recent backend service patch and initiated a rollback of the patch. Once the rollback was complete, the teams confirmed that all services had been restored.  **What are we doing to prevent it from happening again?**  The Engineering team is working to enhance testing of future patches and improving visibility into the health of services. This will help to proactively resolve issues or, when necessary, initiate rollbacks more quickly without impacting customers.

    Issue Discovered - Service disruption in North America – Multiple Services

    Έναρξη 6 Αυγούστου 2025 στις 3:27 π.μ. UTC · 1h 51m

    OutageΣοβαρό περιστατικό
    Επηρεαζόμενα στοιχεία
    SMS NotificationsEmail NotificationsVoice Notifications
    1. 6 Αυγούστου 2025 στις 3:27 π.μ. UTCinvestigating

      xMatters monitoring tools have identified a potential issue with xMatters On-Demand for clients in North America that are hosted in the on us-central1 region. We are currently investigating the issue and will update as information becomes available. Please see incident details for specific services impacted. If you are also experiencing issues, or if you're not sure whether this issue impacts your service, please contact xMatters Client Assistance at https://support.xmatters.com/hc/en-us/requests/new - our support team is waiting to help.

    2. 6 Αυγούστου 2025 στις 3:44 π.μ. UTCidentified

      The xMatters Incident Response team has identified an issue with our message routing system and is working on a fix. We will update once a solution has been identified and implemented.

    3. 6 Αυγούστου 2025 στις 3:59 π.μ. UTCmonitoring

      The xMatters Incident Response team has deployed a fix for the issue. We are currently monitoring the situation to ensure the implementation is stable and that all services are restored.

    4. 6 Αυγούστου 2025 στις 5:18 π.μ. UTCresolved

      The issue has been addressed, and all services have been restored. Thank you for your patience while we addressed this matter.

    5. 14 Αυγούστου 2025 στις 1:16 π.μ. UTCpostmortem

      ### What happened? On August 5th, 2025, at approximately 7:53 PM Pacific, the xMatters internal monitoring tools alerted the xMatters Support team to an issue in the North America region with notification delivery. Some xMatters customers may have experienced delays when receiving notifications or noticed failed alerts in the system. ### Why did it happen? The issue occurred because of a network problem that disrupted communication between parts of the queuing system. This caused some components to become temporarily out of sync, leading to timeouts, internal connectivity failures, and a small number of messages not being processed during the disruption. As the system recovered, some performance was degraded until normal operation could be restored. ### How did we respond? As soon as the xMatters monitoring tools reported an issue, the xMatters Support Team initiated the internal Major Incident Management process and engaged the Engineering and incident response teams. The teams quickly mitigated the issue and restored performance by redirecting traffic from our message queuing system in the affected region \(us-central\) to our message queuing system in another local region \(us-east\). After the issue was resolved, the teams directed traffic back to the restored region. ### What are we doing to prevent it from happening again? The Engineering team has determined that the best approach to prevent this issue from reoccurring is to replace the current message queuing system. Work on the replacement system is well underway and the teams will retire the current system as soon as they have finished the replacement.

    xMatters - Support Center - support.xmatters.com

    Έναρξη 23 Ιουνίου 2025 στις 8:14 μ.μ. UTC · 32m

    Pending
    1. 23 Ιουνίου 2025 στις 8:14 μ.μ. UTCinvestigating

      We have identified a problem affecting the xMatters Support Center. Our team is investigating and working to resolve the issue. Details: Users will be unable to access the xMatters Support Center. Tickets can be created via email or through Support line.

    2. 23 Ιουνίου 2025 στις 8:46 μ.μ. UTCresolved

      We have now resolved the issue with our support center.

    Issue Discovered - Service disruption in North American Region – Integration Platform

    Έναρξη 12 Ιουνίου 2025 στις 6:47 μ.μ. UTC · 3h 26m

    IssuesΜικρό περιστατικό
    Επηρεαζόμενα στοιχεία
    Integration Platform
    1. 12 Ιουνίου 2025 στις 6:47 μ.μ. UTCinvestigating

      xMatters monitoring tools have identified a potential issue with the xMatters Integration Platform for some clients located in the North America region. We are currently investigating the issue and will update as information becomes available. If you are also experiencing issues, or if you're not sure whether this issue impacts your service, please contact xMatters Client Assistance at https://support.xmatters.com/hc/en-us/requests/new - our support agents are waiting to help.

    2. 12 Ιουνίου 2025 στις 7:04 μ.μ. UTCidentified

      The xMatters Incident Response team has identified the source of the issue with an upstream provider and is working with them on identifying a fix. We will update once a solution has been identified and implemented.

    3. 12 Ιουνίου 2025 στις 7:34 μ.μ. UTCidentified

      We are still working with our upstream providers. Their engineers are continuing to mitigate the issue and we have confirmation that the issue is recovered in some locations. We do not have an ETA on full mitigation at this point. We continue to operate in a degraded state.

    4. 12 Ιουνίου 2025 στις 8:06 μ.μ. UTCidentified

      We are continuing to work with our upstream providers. Their engineers have confirmed that the underlying dependency is recovered in most regions and all their engineering teams are actively engaged and working on service recovery. We do not have an ETA for full service recovery.

    5. 12 Ιουνίου 2025 στις 8:53 μ.μ. UTCmonitoring

      At this time, we are seeing all services recovering. We will continue to monitor until we have confirmation from our upstream provider that they have fully resolved the issue.

    6. 12 Ιουνίου 2025 στις 10:12 μ.μ. UTCresolved

      We are currently seeing error rates down to normal levels and we are considering this issue resolved at this time

    7. 23 Ιουνίου 2025 στις 5:40 μ.μ. UTCpostmortem

      **What happened?**  On June 12th, 2025, at approximately 11:29 AM Pacific, the xMatters internal monitoring tools alerted to an issue in the North America region with the xMatters integration platform. While the issue was in progress, some xMatters customers may have experienced workflows not executing correctly or signals not being accepted as expected.  **Why did it happen?**  The issue occurred because multiple Google Cloud and Google Workspace products experienced external API issues for about three hours between the hours of 10:49 AM to 1:49 PM Pacific.  **How did we respond?**  As soon as the xMatters monitoring tools reported an issue with the xMatters integration platform, the xMatters Customer Support team initiated the internal Major Incident Management process and engaged the Engineering and incident response teams. The teams quickly determined that the issues were due to problems with an upstream provider and immediately opened a support case for them to investigate. They continued to work with the provider throughout the incident and confirmed with their engineers as they recovered the underlying dependencies.  **What are we doing to prevent it from happening again?**  The xMatters Support and Engineering teams requested an RCA and prevention methods from the upstream provider. In their response, they confirmed that they will be modularizing their Service Control architecture to prevent failures from affecting the entire service.

    Issue Discovered - Service disruption in European Region - Multiple Services

    Έναρξη 13 Μαΐου 2025 στις 5:45 μ.μ. UTC · 3h 1m

    IssuesΜικρό περιστατικό
    Επηρεαζόμενα στοιχεία
    Mobile AppAPIVoice NotificationsIntegration PlatformWeb InterfaceEmail NotificationsConferencingSMS Notifications
    1. 13 Μαΐου 2025 στις 5:45 μ.μ. UTCinvestigating

      xMatters monitoring tools have identified a potential issue with xMatters On-Demand for some clients located in the Europe region. We are currently investigating the issue and will update as information becomes available. Please see incident details for specific services impacted. If you are also experiencing issues, or if you're not sure whether this issue impacts your service, please contact xMatters Client Assistance at https://support.xmatters.com/hc/en-us/requests/new - our support agents are waiting to help.

    2. 13 Μαΐου 2025 στις 5:53 μ.μ. UTCidentified

      The xMatters Incident Response team has identified the source of the issue and is working on a fix. We will update once a solution has been identified and implemented.

    3. 13 Μαΐου 2025 στις 5:54 μ.μ. UTCidentified

      We are continuing to work on a fix for this issue.

    4. 13 Μαΐου 2025 στις 6:03 μ.μ. UTCmonitoring

      The xMatters Incident Response team has deployed a fix for the issue. We are currently monitoring the situation to ensure the implementation is stable and that all services are restored.

    5. 13 Μαΐου 2025 στις 8:46 μ.μ. UTCresolved

      The issue has been addressed, and all services have been restored. Thank you for your patience while we addressed this matter.

    6. 27 Μαΐου 2025 στις 8:50 μ.μ. UTCpostmortem

      **What happened?**  On May 13, 2025 at approximately 10:45 AM Pacific, xMatters internal monitoring tools identified an issue where customers in the EU region experienced intermittent web UI and API timeouts.  **Why did it happen?**  The issue occurred because a backend queueing service experienced network timeouts during an unpredictable rapid increase in usage. The increase in resource consumption due to the surge in network usage caused service timeouts and restarts, as well as higher latency which caused further delays in responses to backend requests.  **How did we respond?**  xMatters internal monitoring tools alerted the xMatters Incident Response Team to the issue, then the team launched the internal SEV-1 process. Due to early detection, Engineering teams were able to scale up the queueing services to prevent further service degradation and availability issues. The network timeouts were resolved after resources were scaled up to accommodate the increase in usage.  **What are we doing to prevent it from happening again?**  The Engineering teams have adjusted resources to better compensate for sudden usage increases and to prevent them from affecting backend services. The improvement in resource allocation and adaptability should prevent similar issues from occurring in the future.

    Issue Discovered - Service disruption in All Regions – Integration Platform

    Έναρξη 3 Απριλίου 2025 στις 5:15 μ.μ. UTC · 4h 30m

    IssuesΜικρό περιστατικό
    Επηρεαζόμενα στοιχεία
    Integration PlatformIntegration PlatformIntegration Platform
    1. 3 Απριλίου 2025 στις 5:15 μ.μ. UTCinvestigating

      xMatters monitoring tools have identified a potential issue with xMatters Integration Platform for some clients in All Regions. We are currently investigating the issue and will update as information becomes available. If you are also experiencing issues, or if you're not sure whether this issue impacts your service, please contact xMatters Client Assistance at https://support.xmatters.com/hc/en-us/requests/new - our support agents are waiting to help.

    2. 3 Απριλίου 2025 στις 6:28 μ.μ. UTCidentified

      The xMatters Incident Response team has identified the source of the issue and is still working on a fix. We will update once a solution has been identified and implemented.

    3. 3 Απριλίου 2025 στις 6:44 μ.μ. UTCidentified

      The xMatters Incident Response team has identified the source of the issue and is actively working on a fix, there is no current estimate on resolution time. We will update once a solution has been implemented.

    4. 3 Απριλίου 2025 στις 8:26 μ.μ. UTCidentified

      The xMatters Incident Response team has identified the source of the issue and is still actively working on a fix, there is no current estimate on resolution time. We will update once a solution has been implemented.

    5. 3 Απριλίου 2025 στις 9:00 μ.μ. UTCidentified

      The xMatters Incident Response team has identified the source of the issue and is working on a fix. We will update once a solution has been identified and implemented.

    6. 3 Απριλίου 2025 στις 9:13 μ.μ. UTCidentified

      The xMatters Incident Response team has identified the source of the issue and are currently testing a fix. We will provide another update shortly.

    7. 3 Απριλίου 2025 στις 9:18 μ.μ. UTCmonitoring

      A fix has been implemented and we are monitoring the results.

    8. 3 Απριλίου 2025 στις 9:45 μ.μ. UTCresolved

      This incident has been resolved.

    9. 7 Απριλίου 2025 στις 6:55 μ.μ. UTCpostmortem

      **What happened?** On April 3, 2025, at approximately 9:32 AM Pacific, the xMatters internal monitoring systems identified an issue where the system was not processing events initiated via Flow Designer across multiple regions. Customers may have observed the system not processing events or creating alerts while the issue was in progress. **Why did it happen?** The issue occurred when a routine update to add new permissions to the xMatters' Google Cloud Platform \(GCP\) unexpectedly removed required permissions. When the teams performed the update, which should not have had any impact to customers, the policy used in the automation script was the authoritative resource at the GCP project level rather than authoritative at the individual resource level. Although the teams tested the change before deploying it and found no changes beyond what was included in the update, when the update was deployed to production GCP removed all permissions that were not in the policy in the background. Because the policy only included the new permissions, all other permissions were removed. **How did we respond?** As soon as the xMatters monitoring tools reported an issue with the system not processing events, the incident response teams initiated the internal Major Incident Management process and engaged the Engineering and Support teams. The teams were able to quickly identify the recent update as the root cause of the issue and reverted the change to restore the permissions that were managed by the automation process. This restored access and functionality for most of the xMatters services, but restoring permissions for Flow Designer proved to be more complicated. The teams determined that missing permissions for the xMatters infrastructure were Google-generated permissions essential for specific xMatters services and engaged GCP Support to aid in the investigation. The teams generated a list of all permissions that existed prior to the update and designed a fix to re-apply them to the development environment. Once the teams had implemented the change and validated that the missing permissions and all services had been restored in the development environment, they moved to quickly apply the fix across all staging and production environments. Monitoring tools and customers confirmed that services were fully functional, and the teams continued to monitor the system as it processed all messages queued by the Flow Designer services. The system was fully restored at 2:46 PM Pacific. **What are we doing to prevent it from happening again?** The Engineering teams were able to identify all of the permissions created in the xMatters environment, including those that are created by Google, and are ensuring they are added to the management scripts. The teams are adding additional rigor to the application of these types of infrastructure changes to run idempotence tests after the changes are applied to ensure that there are no changes pending. Should a change be applied that fails this test, it will cause failures in development environments, which would catch and prevent a similar issue from occurring.

    Issue Discovered - Service disruption in North American Region - Multiple Services

    Έναρξη 5 Ιουλίου 2024 στις 4:19 μ.μ. UTC · 57m

    Pending
    Επηρεαζόμενα στοιχεία
    SMS NotificationsEmail NotificationsVoice Notifications
    1. 5 Ιουλίου 2024 στις 4:19 μ.μ. UTCinvestigating

      xMatters monitoring tools have identified a potential issue with xMatters On-Demand for some clients located in the North America region. We are currently investigating the issue and will update as information becomes available. Please see incident details for specific services impacted. If you are also experiencing issues, or if you're not sure whether this issue impacts your service, please contact xMatters Client Assistance at https://support.xmatters.com/hc/en-us/requests/new - our support agents are waiting to help.

    2. 5 Ιουλίου 2024 στις 4:36 μ.μ. UTCmonitoring

      A fix has been implemented and we are monitoring the results.

    3. 5 Ιουλίου 2024 στις 5:16 μ.μ. UTCresolved

      This incident has been resolved.

    4. 12 Ιουλίου 2024 στις 8:07 μ.μ. UTCpostmortem

      **What happened?** On July 5th, at approximately 8:35AM Pacific, the xMatters monitoring tools alerted Customer Support to an issue where alert notifications were not being sent out for some customers in the North America region. Some customers attempting to initiate alerts may have encountered long delays in processing or may have had requests time out. **Why did it happen?** This issue occurred due to a sudden spike in the number of resources required by our backend services. The resulting memory overload issue caused some request handlers to time out before they could properly process incoming alerts. **How did we respond?** As soon as the xMatters Customer Support team confirmed the issue from the monitoring tools, they initiate the internal major incident management process and engaged the xMatters Engineering teams. To immediately mitigate the issue and restore service quickly, the incident response teams performed a rolling restart for the affected services. As soon as the restart was completed, the system resumed processing alerts and all services were restored. **What are we doing to prevent it from happening again?** The Engineering teams have implemented a performance enhancement for backend service queries. In addition, the teams are evaluating and testing additional methods to help mitigate resource spikes and prevent them from impacting alert notifications in the future. Once development and testing are complete, we'll deploy these changes with our regularly scheduled maintenance. **Timeline:** July 5th, 2024 8:35AM PT - xMatters internal monitoring tools alert to potential issue. 9:22AM PT - Issue identified. 9:36AM PT - Rolling restart initiated. 10:16AM PT - Issue Resolved.

    Issue Discovered - Service disruption in North America Region – Partial Outage

    Έναρξη 1 Ιουλίου 2024 στις 6:23 π.μ. UTC · 1h 58m

    OutageΣοβαρό περιστατικό
    Επηρεαζόμενα στοιχεία
    Web Interface
    1. 1 Ιουλίου 2024 στις 6:23 π.μ. UTCinvestigating

      xMatters monitoring tools have identified a potential issue with xMatters for some clients in the North America region. We are currently investigating the issue and will update as information becomes available. If you are also experiencing issues, or if you're not sure whether this issue impacts your service, please contact xMatters Client Assistance at https://support.xmatters.com/hc/en-us/requests/new - our support agents are waiting to help.

    2. 1 Ιουλίου 2024 στις 6:39 π.μ. UTCmonitoring

      The xMatters Incident Response team has deployed a fix for the issue. We are currently monitoring the situation to ensure the implementation is stable and that all services are restored.

    3. 1 Ιουλίου 2024 στις 8:21 π.μ. UTCresolved

      The issue has been addressed, and all services have been restored. Thank you for your patience while we addressed this matter.

    Issue Discovered - Service disruption in North America Region – Partial Outage

    Έναρξη 18 Ιουνίου 2024 στις 11:16 μ.μ. UTC · 22m

    IssuesΜικρό περιστατικό
    Επηρεαζόμενα στοιχεία
    Web Interface
    1. 18 Ιουνίου 2024 στις 11:16 μ.μ. UTCinvestigating

      xMatters monitoring tools have identified a potential issue with xMatters for some clients in the North America region. We are currently investigating the issue and will update as information becomes available. If you are also experiencing issues, or if you're not sure whether this issue impacts your service, please contact xMatters Client Assistance at https://support.xmatters.com/hc/en-us/requests/new - our support agents are waiting to help.

    2. 18 Ιουνίου 2024 στις 11:26 μ.μ. UTCidentified

      The xMatters Incident Response team has identified the source of the issue and is working on a fix. We will update once a solution has been identified and implemented.

    3. 18 Ιουνίου 2024 στις 11:34 μ.μ. UTCmonitoring

      The xMatters Incident Response team has deployed a fix for the issue. We are currently monitoring the situation to ensure the implementation is stable and that all services are restored.

    4. 18 Ιουνίου 2024 στις 11:38 μ.μ. UTCresolved

      The issue has been addressed, and all services have been restored. Thank you for your patience while we addressed this matter.

    Intermittent scheduler services in APAC region

    Έναρξη 24 Μαΐου 2024 στις 1:00 π.μ. UTC · 22m

    OutageΣοβαρό περιστατικό
    Επηρεαζόμενα στοιχεία
    Web InterfaceVoice NotificationsConferencingEmail NotificationsSMS NotificationsIntegration PlatformAPIMobile App
    1. 24 Μαΐου 2024 στις 1:00 π.μ. UTCinvestigating

      We are currently investigating an issue with one of our scheduler services impacting customers in Australia.

    2. 24 Μαΐου 2024 στις 1:03 π.μ. UTCinvestigating

      We are continuing to investigate this issue.

    3. 24 Μαΐου 2024 στις 1:04 π.μ. UTCinvestigating

      xMatters monitoring tools have identified a potential issue with xMatters On-Demand for some clients located in the Asia Pacific region. We are currently investigating the issue and will update as information becomes available. Please see incident details for specific services impacted. If you are also experiencing issues, or if you're not sure whether this issue impacts your service, please contact xMatters Client Assistance at https://support.xmatters.com/hc/en-us/requests/new - our support agents are waiting to help.

    4. 24 Μαΐου 2024 στις 1:08 π.μ. UTCidentified

      The xMatters Incident Response team has identified the source of the issue and is working on a fix. We will update once a solution has been identified and implemented.

    5. 24 Μαΐου 2024 στις 1:12 π.μ. UTCmonitoring

      The xMatters Incident Response team has deployed a fix for the issue. We are currently monitoring the situation to ensure the implementation is stable and that all services are restored.

    6. 24 Μαΐου 2024 στις 1:23 π.μ. UTCresolved

      The issue has been addressed, and all services have been restored. Thank you for your patience while we addressed this matter.