Kisi 장애 기록

Mindbody API doesn't return any client information which subsequently led to revocation of access rights during the daily sync.

We have contacted the Mindbody support team and waiting for updates from their side. We are manually restoring access to as many as users as we can.

We are continuing to work on a fix for this issue.

We have restored access to all users who lost their access during the daily sync. Webhooks will continue to work as expected. We will pause daily syncs on Mindbody integration until the issue is resolved.

Mindbody team has confirmed that they have fixed the issue. Daily sync should proceed as usual. If you notice any issues, please reach out to Kisi Support team.

Between 1:15 AM and 6:15 AM UTC our API experienced a resource overload incident affecting a small subset of resources that require access to our caching backend. A fix has been deployed and we are actively monitoring the situation. We are working on restoring metadata. The services that were affected: - Event webhook delivery - Intercom calls - Incident alerts - Camera snapshots

This incident has been resolved.

We are currently experiencing issues in our events and incidents services due to an issue in our cloud services provider (GCP). We are actively working to mitigate the issue.

Our events and incidents services have been partially restored. We are working to restore missing events. Incidents dispatch may also be delayed.

This incident has been resolved.

We are currently investigating the issue. This affects all marketplace applications.

We are continuing to investigate this issue.

We have deployed a new version that fixes the issue. We are monitoring the recovery of the service.

Our systems have fully recovered.

We are currently investigating the issue. It appears to be related to the VORTEX API. We have already reached out to the VORTEX team to address this matter.

VORTEX team is working on a fix for the issue.

This incident has been resolved.

We are currently investigating this issue. We are coordinating with VORTEX team to resolve this issue as soon as possible.

The video exports from VORTEX cameras are working as expected now.

This incident has been resolved.

We are currently investigating the issue. It appears to be related to the VORTEX API. We have already reached out to the VORTEX team to address this matter.

The video exports are functioning as intended. We will continue to monitor the situation closely.

This incident has been resolved.

Between 13:10 and 16:50 UTC, customers may have experienced a delay in synchronization of reader, controller and door configurations from the cloud to the devices. The issue has since been resolved. All device configurations are now up-to-date.

On June 16, 2025, the Kisi production API experienced a temporary degradation in availability. Investigation revealed that the root cause was an unexpected behavior in Google Cloud Platform’s (GCP) Cloud Run during an internal stack migration and service redistribution process managed by GCP. According to GCP’s engineering response, a background migration and redistribution of workloads triggered a one-time instance restart in our Cloud Run service. This resulted in the application falling below its configured minimum instances temporarily, impacting our ability to serve requests reliably. While the redistribution process is part of GCP’s infrastructure optimization efforts, the side effect of under-scaling was not communicated in advance and not within our control. GCP has confirmed that a fix has been rolled out to address this issue and prevent recurrence in similar scenarios.

We’re currently investigating an increase in 5xx and 4xx errors affecting our public API. This may impact door unlocks, as well as access to the web and mobile dashboards. We’ll share more updates as we identify the root cause.

We are continuing to investigate this issue.

A fix has been implemented and we are monitoring the results.

The incident has been resolved. It lasted for approximately 30 minutes, from 17:15 to 17:45 UTC.

Google Cloud Platform is experiencing an outage. Unlocks are not affected, but there may be delays for event related processing (webhooks, event history, ...).

Google Cloud Platform has restored service, and all of our affected systems are operating normally again.

The Event Service—Kisi’s core system for capturing and forwarding events—experienced a degradation after a recent feature deployment. An unclosed-connection bug gradually exhausted CPU and memory resources, throttling database writes. Our monitoring detected elevated error rates. A redeployment addressed the bug, but due to a large backlog some delayed event webhooks were unfortunately dropped. However, no events were lost. Service Impact: - Event visibility in the Kisi Admin Dashboard and via API was degraded during the degradation window (new events were not yet persisted). - Event Webhook deliveries and report generation were paused until full recovery. Next Steps: - We will adapt our system to process all event webhooks and event related jobs regardless of backlog size - We will isolate our event pipeline to avoid spillover effects from new functionality

The most recent Kisi API release caused elevated error rates. The release has been rolled back and API error rates are back to normal. Unlocks were affected during this time.

An issue is ongoing causing email delays for all customers. We have identified our email delivery service as the cause of this incident. See https://status.mailgun.com/incidents/30w5l1zqcw5r Currently, a majority of our outgoing emails are affected.

A fix has been implemented by our email delivery service provider and we are monitoring results. As of 08:30 UTC a majority of emails are being delivered.

Fix implemented by our email service provider seems to have fixed the issue. All our outgoing emails have been delivered since 08:30 UTC.

We are investigating an issue causing email delays for all customers. The incident started 13:10 UTC, and is ongoing.

We have identified our email delivery service as the cause of this incident. See https://status.mailgun.com/incidents/8bn2px3nyfy4 Currently, a majority of our outgoing emails are affected.

A majority of emails are being delivered as of 13:52 UTC. We will continue to monitor the issue.

Our email delivery provider has implemented a fix and is monitoring the results. Emails are being delivered successfully since 13:52 UTC.

The most recent Kisi API release caused elevated error rates. We are rolling back and expect the issue to be resolved in a minute or two. Unlocks are affected.

The release has been rolled back. Unlock failure rates and API error rates are back to normal.

The rate of failed Cisco Meraki camera snapshots is higher than normal, we are investigating the issue. We will provide an update by Monday, 2023-04-25 09:00 UTC with current details.

Cisco Meraki camera snapshots are working normally. The elevated error rates could be attributed to a single Kisi organization.

We experienced a short (less than 10 minutes) API outage. As a result, unlocks via cloud were failing and web dashboard was unavailable. Offline cache unlocks (https://docs.kisi.io/concepts/offline_support) were working as usual. The issue has now been resolved and we are doing everything to make sure it will not happen again.

We discovered an issue that caused in-app unlocks to fail for a small group of users that belong to groups with restrictions between 2024-01-24 11:12 UTC and 2024-01-25 14:19 UTC. Although we have resolved the underlying problem causing unlock failures, there was a misreporting of these failures as successes in the unlock events. We are actively addressing and correcting the affected events.

We are currently correcting the affected unlock events.

All incorrect unlock events have been fixed.

We have identified an issue with background jobs leading to severely degraded throughput. Outgoing emails, alerts, and webhooks are among the impacted features. The root cause seems to be an outage at one of our service providers (https://status.elastic.co/incidents/07bw653d2677). Our team is working on mitigating the issue.

We are continuing to work on a fix for this issue.

We are working on a fix to mitigate the impact of the service outage. The fix will be deployed shortly.

Our systems are processing jobs at their normal rate again.