Ιστορικό διακοπών του Keeper

1. 31 Ιουλίου 2026 στις 8:00 μ.μ. UTCresolved

   Οι συνδέσεις που δημιουργούνται μέσω του Φύλακα Router και του Gateway στην περιοχή της ΕΕ αποτυγχάνουν. Πρόσθετες λεπτομέρειες θα αναρτηθούν στη μεταθανάτια αναφορά.

2. 31 Ιουλίου 2026 στις 8:00 μ.μ. UTCpostmortem

   # Έκθεση μετά το συμβάν ** 31 Ιουλίου 2026 ** Περίληψη Στις 30 Ιουλίου 2026 στις 11:33 μμ CT, η υπηρεσία δρομολόγησης σύνδεσης KeeperPAM στην περιοχή της ΕΕ άρχισε να ρίχνει λάθη. Τα σφάλματα Router/Gateway διήρκεσαν περίπου 12 ώρες και 45 λεπτά, με την πλήρη υπηρεσία να αποκαθίσταται περίπου στις 12:18 PM CT στις 31 Ιουλίου. Κατά τη διάρκεια αυτού του παραθύρου, η κύρια πλατφόρμα της ΕΕ Φύλακας \ (πρόσβαση σε vault, ταυτοποίηση, και όλες οι άλλες υπηρεσίες Φύλακα\) παρέμεινε πλήρως λειτουργική σε όλη τη διάρκεια. Τι συνέβη Στις 22 Ιουλίου, μια συνηθισμένη ανάπτυξη στην υπηρεσία δρομολόγησης σύνδεσης μας \(“Keeper Router”\) περιελάμβανε μια ενημερωμένη εξάρτηση που άλλαξε τον τρόπο με τον οποίο η υπηρεσία ανακτά τις ρυθμίσεις εκκίνησης της. Μια εσφαλμένη διαμόρφωση στο τελικό σημείο οδήγησε την περιοχή της ΕΕ να αναζητήσει τελικά σημεία FIPS, τα οποία δεν είναι διαθέσιμα. Ως αποτέλεσμα, όταν τα εμπορευματοκιβώτια υπηρεσιών στην περιοχή της ΕΕ επανεκκινήθηκαν μετά την ανάπτυξη, δεν μπόρεσαν να ανακτήσουν τη διαμόρφωση εκκίνησης τους και εισήλθαν σε μια αποτυχημένη κατάσταση. Δύο παράγοντες επέτρεψαν αυτή η αποτυχία να πάει απαρατήρητη για πάνω από μια εβδομάδα. Πρώτον, τα υπάρχοντα εμπορευματοκιβώτια συνέχισαν να εξυπηρετούν την κυκλοφορία ενώ τα αντικαταστάσιμα εμπορευματοκιβώτια αθόρυβα απέτυχαν να ξεκινήσουν, οπότε δεν υπήρξε άμεση επίπτωση του πελάτη από την ανάπτυξη της 22ας Ιουλίου. Η QA πέρασε επίσης όλες τις δοκιμές επαλήθευσης της παραγωγής. Δεύτερον, η βλάβη διαμόρφωσης-φορτώματος καταχωρήθηκε σε επίπεδο INFO και όχι σε ERROR ή CRITIC, οπότε δεν δημιουργήθηκαν ειδοποιήσεις PagerDuty. Τη νύχτα της 30ης Ιουλίου, τα τελευταία υγιή δοχεία απογειώθηκαν, ο ισοσταθμιστής φορτίου της υπηρεσίας είχε μηδενικούς υγιείς στόχους, και το τελικό σημείο άρχισε να επιστρέφει HTTP 503 λάθη. Αυτοματοποιημένη παρακολούθηση ελέγχου υγείας εντόπισε την διακοπή μέσα σε δευτερόλεπτα και κάλεσε την ομάδα εφημερίας μας. Τι αλλάζουμε 1. **Ανάθεση σε αποτυχημένη rollover δοχείο. ** Αναπτύσσουμε συναγερμούς του CloudWatch σε όλες τις περιοχές και περιβάλλοντα που πυροδοτούν όταν οι εργασίες του εμπορευματοκιβωτίου αποτυγχάνουν επανειλημμένα να ξεκινήσουν, για να ανιχνεύσουν σιωπηλές αποτυχίες ανάπτυξης. 2. ** Σοβαρότητα σύνδεσης για τις αποτυχίες εκκίνησης. ** Οποιαδήποτε αστοχία φόρτωσης των απαιτούμενων ρυθμίσεων εκκίνησης θα καταγράφεται τώρα σε ΣΦΑΛΜΑ/ΚΡΙΤΙΚΗ σοβαρότητα αντί INFO που ενεργοποιούν τις κατάλληλες ενεργές ειδοποιήσεις. Ζητούμε συγγνώμη από τους πελάτες μας της ΕΕ για την αναστάτωση.

Αυτόματη μετάφραση από την επίσημη ενημέρωση του συμβάντος.

1. 8 Ιουλίου 2026 στις 6:51 μ.μ. UTCidentified

   Γνωρίζουμε για τα σφάλματα σύνδεσης Keeper Gateway και KSM λόγω "μη έγκυρης έκδοσης πελάτη" απαντήσεις. Αναγνωρίσαμε το ζήτημα και θα επιλυθεί σε λίγα λεπτά.

2. 8 Ιουλίου 2026 στις 7:14 μ.μ. UTCresolved

   Τα λάθη API έχουν επιλυθεί. Η έκδοση του Keeper Gateway και του KSM client δεν επιστρέφει πλέον ένα σφάλμα.

Αυτόματη μετάφραση από την επίσημη ενημέρωση του συμβάντος.

1. 19 Ιουνίου 2026 στις 3:05 μ.μ. UTCidentified

   We are aware of an issue with direct record sharing for enterprise users as a result of yesterday's Vault 18.3 update. We will issue a patch before 12PM PST. To work around the issue, you can click the "Can share to users outside the enterprise" option in Role enforcements, or just use the Desktop App.

2. 19 Ιουνίου 2026 στις 11:00 μ.μ. UTCresolved

   The issue was resolved at 12:30PM PST.

1. 1 Απριλίου 2026 στις 1:40 π.μ. UTCresolved

   On March 30 at 8:00 PM PST, Keeper performed a scheduled minor upgrade to our multi-region AWS RDS clusters as part of routine maintenance. The upgrade itself completed in approximately 3 minutes, followed by a standard application restart process that took about 15 minutes. While this process is typically non-disruptive, recent changes to service health checks caused unexpected customer impact during the restart window. This maintenance was not communicated in advance due to an internal miscommunication and the expectation of no impact. We apologize for the disruption and are taking steps to ensure all future maintenance—regardless of expected impact—is communicated ahead of time, as well as reviewing our health check configurations to prevent similar issues.

1. 23 Μαρτίου 2026 στις 5:07 μ.μ. UTCinvestigating

   We are investigating errors in establishing KeeperPAM connections in the US Data Center.

2. 23 Μαρτίου 2026 στις 6:47 μ.μ. UTCidentified

   We have identified the cause of the KeeperPAM connection errors which are related to a networking issue within the AWS ECS environment. We are actively troubleshooting with the AWS support team and will update the ticket as soon as there is a status update.

3. 23 Μαρτίου 2026 στις 8:03 μ.μ. UTCmonitoring

   A fix has been implemented and we are monitoring the KeeperPAM connection stability. We will update this case with additional details soon.

4. 23 Μαρτίου 2026 στις 9:45 μ.μ. UTCresolved

   The issue has been resolved. See postmortem page with additional details.

5. 23 Μαρτίου 2026 στις 11:17 μ.μ. UTCpostmortem

   At 9:05 AM PST, alerts were triggered for issues affecting KeeperPAM connections managed through Keeper’s ECS deployments in the US-EAST region. There were no recent changes to the application or environment. Investigation identified a low-level concurrency bug in the Keeper EPM service that caused request failures under high simultaneous load. These failures led to instability in the ECS services supporting KeeperPAM connections. As a temporary mitigation, we blocked the error condition, restoring KeeperPAM connectivity by approximately 1:00 PM PST. The engineering team then developed and deployed an updated Keeper Router version to address the underlying issue and prevent EPM agents from triggering server errors. The fix was fully validated by 3:00 PM PST, at which point all KeeperPAM services were stable and operating normally.

1. 16 Μαρτίου 2026 στις 6:25 μ.μ. UTCidentified

   We have identified an issue causing KeeperPAM connection errors in the US Data Center. It will be resolved shortly.

2. 16 Μαρτίου 2026 στις 6:52 μ.μ. UTCresolved

   The issue has been resolved. We will update the ticket with additional details shortly.

3. 16 Μαρτίου 2026 στις 6:52 μ.μ. UTCpostmortem

   At 11:08 AM PST, DevOps and customers reported sporadic error messages occurring after login for KeeperPAM customers with connections and tunnels enabled. While vault login itself was not affected, the errors were triggered asynchronously after login due to a communication issue with the KeeperPAM router endpoint. Further investigation determined that a "scheduler" database within the AWS RDS environment - used for certain PAM scheduling operations - was encountering an unexpected error. The operations team resolved the underlying database issue and restarted the affected services. This scheduler service was inadvertently impacting connection establishment. To prevent similar issues in the future, we will be implementing software changes to ensure that scheduling-related errors cannot affect connection functionality. Additional monitoring and alerting have also been implemented for the scheduler databases to help detect and prevent similar issues going forward. Full service for connection and tunneling capabilities was restored by 11:40 AM PST.

1. 19 Φεβρουαρίου 2026 στις 5:30 π.μ. UTCresolved

   As part of our planned GovCloud region maintenance involving database upgrades, restarts of the Redshift cluster were required. As a result, downstream services required configuration updates and all services are now restored.

2. 19 Φεβρουαρίου 2026 στις 5:30 π.μ. UTCpostmortem

   Maintenance in the GovCloud region was scheduled to begin at 8:00 PM PST with a planned duration of 30 minutes. During the maintenance window, certain services required additional reconfiguration beyond the original scope. Core services were restored within the planned 30-minute window. However, due to the extended reconfiguration activities, some users experienced API errors during login for approximately 15 minutes following service restoration.

1. 2 Δεκεμβρίου 2025 στις 10:48 μ.μ. UTCidentified

   During scheduled system updates, we have encountered unexpected NGINX 503 / 502 errors in the EU region. As of 4:50 PM CST, work is still in progress. During this time, users in the EU region will experience brief or intermittent access interruptions when using Keeper.

2. 3 Δεκεμβρίου 2025 στις 12:47 π.μ. UTCresolved

   The issue has been resolved. We will update the postmortem report shortly.

3. 4 Δεκεμβρίου 2025 στις 10:29 μ.μ. UTCpostmortem

   On December 2, our **EU region** experienced an unexpected outage during planned infrastructure updates. The issue began at **1:24 PM PT** and service was fully restored by **4:10 PM PT**. **What Happened** During an update to our server configuration, a region-specific configuration error caused the EU environment to begin throwing API errors. Although the underlying issue was identified quickly, each corrective change required a full autoscaling group rollout \(up to 30 minutes per cycle\) which extended the total recovery time. **Resolution** Our engineering team implemented the necessary fixes and restored service. We have also updated our infrastructure-as-code to ensure this issue cannot recur in future deployments. We apologize for the disruption and appreciate your patience while we resolved the incident.

1. 20 Οκτωβρίου 2025 στις 5:25 μ.μ. UTCidentified

   There are intermittent connectivity issues due to the AWS outage in US-EAST-1 affecting KeeperPAM cloud connections, scheduled rotations and discovery operations.

2. 20 Οκτωβρίου 2025 στις 8:31 μ.μ. UTCresolved

   Clearing this alert as most AWS services are restored.

1. 20 Οκτωβρίου 2025 στις 9:20 π.μ. UTCinvestigating

   AWS is reporting errors in the US-EAST region that are affecting multiple services, affecting email and SMS delivery which impacts Keeper device approvals. The AWS Service Health dashboard can be monitored here: https://health.aws.amazon.com/health/status

2. 20 Οκτωβρίου 2025 στις 10:00 π.μ. UTCidentified

   SMS delivery services have been restored, but AWS Lambda is still down which affects our email delivery of device approvals. We recommend using the "Keeper Push" or TOTP method of device approval.

3. 20 Οκτωβρίου 2025 στις 3:17 μ.μ. UTCresolved

   All Keeper email delivery services in US-EAST-1 were restored around 4AM PST.

1. 10 Σεπτεμβρίου 2025 στις 3:42 μ.μ. UTCidentified

   We are aware of Auth API errors in EU region and we have identified the isssue. The issue will be resolved shortly.

2. 10 Σεπτεμβρίου 2025 στις 3:49 μ.μ. UTCresolved

   The issue with Auth API errors in the EU region has been resolved. Additional details will be posted in the postmortem shortly.

3. 26 Σεπτεμβρίου 2025 στις 4:24 μ.μ. UTCpostmortem

   On **September 10 at 8:42 AM PDT**, our monitoring detected elevated errors with the **Auth API in the EU region**. The issue was identified promptly and fully resolved by **8:49 AM PDT**. Total impact duration was approximately **7 minutes**. ### Impact * **Region affected:** EU * **Service affected:** Authentication API * **Customer impact:** Customers in the EU experienced failed authentication attempts during the incident window. ### Timeline \(PDT\) * **08:42** – Incident detected. Elevated Auth API errors in the EU region reported. * **08:43** – Engineering and DevOps teams engaged. * **08:46** – Root cause isolated to a Terraform-related configuration change involving a VPC endpoint. * **08:49** – Configuration change rolled back; Auth API errors resolved. ### Root Cause The outage was caused by a configuration change in our production tenant during an ongoing Terraform migration. As part of the migration, a new VPC endpoint was being added to production environments. During Terraform apply, a subnet association was also being created. A network disruption occurred between the creation of the VPC endpoint and the subnet association \(behavior not seen in prior testing\) leading to temporary Auth API connectivity failures in the EU region. A new Terraform migration plan has been created that eliminates the possibility of disruption.

1. 21 Αυγούστου 2025 στις 5:36 μ.μ. UTCresolved

   Resolved - At 9:49 AM PST, we experienced login API request failures due to errors on an RDS database in the US region. The database failover completed successfully, and all systems were fully operational by 10:00 AM PST.

1. 8 Ιουλίου 2025 στις 3:18 π.μ. UTCresolved

   Several customers have been experiencing SSO login errors this morning. The team determined the cause of the issue which is detailed in the postmortem report.

2. 8 Ιουλίου 2025 στις 3:18 π.μ. UTCpostmortem

   On June 30, 2025, Keeper published an updated SAML Service Provider \(SP\) certificate as part of our standard annual process. This certificate is used for signing SP requests and does not impact SSL termination. During the update, the certificate subject name was unintentionally changed from sso.keepersecurity.com to sso.staging.keepersecurity.com. While most identity providers do not validate the certificate subject name, certain providers—such as JumpCloud and Shibboleth—may enforce strict matching and reject SAML requests when the subject name does not align. On Sunday July 6, 2025, routine backend infrastructure changes caused the cert to be propagated across production systems throughout the morning hours. The issue with the subject name was identified by our team after several troubleshooting calls with customers. The DevOps team then corrected the issue by publishing the certificate with the proper subject name around 12pm PST. After reviewing the case history, this affected a small number of customers. Next year, when the SP cert change is approaching, we will notify all customers with the exact date and time when the cert will be updated, so those customers using strict SP cert matching can be prepared. We apologize for the error and we have taken steps to ensure this does not occur again.

1. 24 Ιουνίου 2025 στις 3:27 μ.μ. UTCinvestigating

   We are investigating vault login errors

2. 24 Ιουνίου 2025 στις 3:28 μ.μ. UTCinvestigating

   We are continuing to investigate this issue.

3. 24 Ιουνίου 2025 στις 4:01 μ.μ. UTCresolved

   The issue has been resolved. The outage was caused by a high volume of database locks related to a large BreachWatch data update. The engineering team is tracking down the root cause and will be addressing a preventative solution today.

4. 24 Ιουνίου 2025 στις 8:42 μ.μ. UTCpostmortem

   At 8:15 AM PST, our DevOps team was alerted to elevated API error rates in production. Upon investigation, we identified the following: * The production database was experiencing high table lock contention. * This was triggered by several large-scale BreachWatch data updates. * A recent backend migration had unintentionally doubled the number of parallel update operations, which exacerbated the issue. * A database failover was performed to restore service stability. The issue was fully resolved by 8:40 AM PST. The engineering team has addressed the underlying problem in the update processes and is implementing safeguards to prevent recurrence, including query optimization and improved job orchestration. We apologize for the disruption.

1. 11 Ιουνίου 2025 στις 12:04 μ.μ. UTCinvestigating

   We are investigating login errors in the EU data center.

2. 11 Ιουνίου 2025 στις 12:16 μ.μ. UTCidentified

   We have identified the issue causing login errors in the EU data center. Our team is working on a resolution.

3. 11 Ιουνίου 2025 στις 12:46 μ.μ. UTCresolved

   Login errors in the EU region have been fully resolved. We will update this incident postmortem with additional information shortly.

4. 12 Ιουνίου 2025 στις 5:09 π.μ. UTCpostmortem

   At 4:37AM PST, Keeper experienced API errors in the EU region. Service was restored by 5:35AM PST following a database failover. The issue was caused by a bug in the new Device Management API, released the day before. During device registration, the system attempted to delete old devices, which created high database load for accounts with large device counts. The bug was identified, fixed, and a patch was deployed. We apologize for the disruption. To ensure uninterrupted access, we recommend enabling “Work Offline” mode for your account.

1. 13 Μαρτίου 2025 στις 7:17 μ.μ. UTCidentified

   Some users are receiving errors when downloading file attachments. We are working on resolution.

2. 14 Μαρτίου 2025 στις 1:15 π.μ. UTCresolved

   The issue has been resolved. Additional information is in the postmortem report.

3. 16 Μαρτίου 2025 στις 11:32 μ.μ. UTCpostmortem

   ### **Incident Summary** In the US, EU, and AU regions, users experienced vault errors when attempting to download a file attachment immediately after adding it to a record. This issue affected only newly uploaded attachments in these three data center regions. The Gov, JP, and CA regions were not impacted. ### **Root Cause** During our investigation, we identified a discrepancy in how file attachment triggers were implemented across different regions. A recent Terraform change inadvertently removed the triggers responsible for moving file attachments in the US, EU, and AU regions, preventing newly uploaded files from being immediately available for download. ### **Resolution** We corrected the issue by restoring the missing triggers and applying the necessary changes. Additionally, any affected files were successfully moved to their intended locations. ### **Impact & Mitigation** * No data loss occurred. * All impacted files were successfully resolved by our corrections. * We corrected the issue to prevent similar discrepancies in future infrastructure changes. We apologize for any inconvenience and appreciate your patience while we resolved this issue.

1. 11 Μαρτίου 2025 στις 4:40 π.μ. UTCresolved

   Between 5:15PM PST and 5:30PM PST, there were a high number of Vault Login errors.

2. 11 Μαρτίου 2025 στις 4:40 π.μ. UTCpostmortem

   ### **Incident Summary** The Keeper DevOps team was alerted to API errors in the US region. Upon investigation, we identified a high number of database deadlocks affecting performance. To mitigate the immediate impact, we restarted the affected writer instance. ### **Root Cause** Historically, similar issues were linked to specific queries impacting MySQL 8.x updates on the RDS Aurora platform. In this case, we discovered that a utility designed to monitor deadlocks and capture diagnostic data was inadvertently contributing to the problem. A known bug in the MySQL minor version used in production caused our data-gathering query—intended to analyze deadlocks—to trigger additional locking, exacerbating the issue. ### **Resolution & Mitigation** * We promptly disabled the problematic monitoring utility and applied database parameter optimizations in collaboration with the AWS team to enhance performance under peak load. * These changes were fully implemented within 24 hours, and no further excessive row locks have been observed. * Our DBA team is actively working with backend engineers to further optimize write operations for improved performance. * A backend code update with additional optimizations is scheduled for release the week of March 17. We appreciate your patience and will continue to monitor and enhance system performance to prevent similar issues in the future.

1. 6 Μαρτίου 2025 στις 6:23 μ.μ. UTCinvestigating

   We are investigating login errors and our team is currently working on the issue.

2. 6 Μαρτίου 2025 στις 6:24 μ.μ. UTCinvestigating

   We are continuing to investigate this issue.

3. 6 Μαρτίου 2025 στις 6:25 μ.μ. UTCresolved

   The issue has been resolved

1. 24 Φεβρουαρίου 2025 στις 4:48 π.μ. UTCidentified

   We are still working on the issue which is causing login errors in the US region. Please use Keeper's "work offline" while we are working to resolve the issue.

2. 24 Φεβρουαρίου 2025 στις 5:05 π.μ. UTCresolved

   The issue has been resolved and root cause determined. We will update the postmortem on this issue shortly.

3. 24 Φεβρουαρίου 2025 στις 7:07 π.μ. UTCpostmortem

   We identified a backend API and resulting database query which triggered a high database load and affected production services. We are publishing a permanent fix on Monday, Feb 24.

1. 24 Φεβρουαρίου 2025 στις 3:54 π.μ. UTCinvestigating

   We are currently investigating login errors in the US region.

2. 24 Φεβρουαρίου 2025 στις 4:28 π.μ. UTCresolved

   The issue has been resolved.