Database Failures
- investigating
We are again investigating an elevated number of database failures.
- investigating
We are continuing to investigate the issue.
- monitoring
We have identified the problem, adjusted our scaling strategy, and are seeing healthy recovery of all systems.
- resolved
This incident has been resolved.
- postmortem
## Summary & Impact Close sincerely apologizes for the interruption of our service. We take the stability of our platform very seriously, and this incident is not acceptable to us. Below is an explanation of what happened and how we will prevent another such interruption from occurring. On August 11th, 2026, Close was intermittently unavailable for a total of about 2.5 hours between 08:00 UTC and 17:03 UTC. During those periods, the Close web app, mobile apps, and API returned errors or failed to load, affecting customers in all regions. Because the disruption was intermittent rather than continuous, many customers likely experienced it as a longer period of unreliability across the working day rather than as a handful of discrete outages. There was also added confusion because we marked our status page "resolved" twice during the day, at 09:26 UTC and again at 15:15 UTC, before the underlying cause had been fixed. We are improving our process so that we do not mark an incident as resolved until we have a good understanding of the situation and applied appropriate mitigations, not just because the symptoms have subsided. ## What went wrong? A routine code change to our email and calendar syncing service altered the way it looks up account information in our database. The new version looked equivalent to the code it replaced, but it caused our database to perform dramatically more internal work for each lookup, enough to exhaust the memory on our primary database server and repeatedly crash it. Because each crash was followed by an automatic recovery, the pattern looked like general instability rather than the symptom of a specific change, which is why it took us longer than we would have wished to trace it back and revert it. ## How we’ll fix this We have already reverted the problematic change and made configuration adjustments to our database hosts to reduce the severity of this class of failure. Close is currently stable. Going forward, we will improve the design of our Email & Calendar Syncing service and similar services which touch Organizations, Users, and Memberships. We will also improve our query performance visibility and alerting, so that slow or inefficient queries and new, unexpected query patterns are identified sooner than they were here. ## Technical Details For those interested in the specifics, here is the full sequence of events and the underlying database behavior. ### Timeline \(all times UTC\) * **August 10, 20:34** — The change finishes deploying to production. Our primary database shard briefly becomes unstable and fails over to a redundant node, producing a few minutes of errors. Because the change deployed after peak usage and deployment hours, traffic subsided shortly afterward and the database remained stable overnight. * **August 11, 08:00** — As application usage and engineering deployments begin increasing, error rates on our primary database shard start to climb. * **08:17** — Our monitoring alerts fire. * **08:26** — Close Engineering is investigating and has posted a status page incident. * **08:46** — We restart the affected database processes, allowing the system to recover. * **09:26** — All Close systems have returned to what looked like normal operation, and we mark the status page resolved. * **12:11 – 17:03** — The same loop of elevated errors, instability in the primary shard, failover to a redundant node, and temporary recovery repeats itself several times. * **14:45** — We find that the database is exhausting its allowed number of open files and raise that limit. This improves stability but does not address the underlying cause. * **15:15** — Systems appear to have recovered again, and we mark the incident resolved a second time. * **15:50** — We identify the previous day's deployment as the likely culprit. * **16:51 – 17:18** — We begin reverting the change when the database becomes unstable again. Customer-facing errors subside at 17:03, and the rollback completes at 17:18. ### The underlying database behavior Our Email & Calendar syncing service regularly queries a database for an up-to-date state of Organizations, Users, and their active Memberships. We store this information in our MongoDB cluster, on its primary shard. We were refactoring some of this code as part of normal tech debt work to bring older code in line with our current patterns. Previously, this query used a `$in: […long list of active organizations' IDs…]` clause as the filter for this query. This wasn’t necessarily very efficient, but it was acceptable because the query ran at a relatively low rate \(only happened every 50 seconds for one syncing process, of which we have 120-180 running depending on the demand\) and runs asynchronously. The refactor changed that query to find applicable Memberships via a `$or: [{organization: someOrgId, user: someUserId}, …long list of such conditions for all users with email/calendar accounts…]` clause. This at first looked like it should have performance roughly equal to the previous query. However, we have since discovered that, while our MongoDB cluster handled large `$in` queries decently well, a large set of compound queries all put into a root-level `$or` clause had disastrous performance characteristics. Even ~3-4 such queries per second, all running roughly around the same time alongside other normal requests, could overwhelm even our heavily scaled database node. Our current understanding, based on our review of the MongoDB source code and ad hoc testing, is that the issue relates to how MongoDB plans large root-level `$or` queries. MongoDB may evaluate the clauses of this query shape as separate sub-plans rather than as a single index scan. In our case, a query could contain approximately 50,000 clauses. At three such queries per second, the database was being presented with roughly 150,000 `$or` clauses per second. We believe this generated tens of thousands of index scans and WiredTiger cursors, churned the query-plan cache, and placed extreme pressure on memory and other database resources. This behavior is consistent with the out-of-memory terminations, file-handle exhaustion, and other low-level failures we observed. Reverting the new query shape eliminated the resource spikes and restored database stability. Pinpointing the query responsible for the database load took longer than anticipated because the seemingly small query rate, a flood of other slow queries, and an immediate database recovery obscured the underlying cause.