Circleapp 장애 기록

We are currently investigating this issue.

The issue has been identified and a fix is being implemented.

A fix has been implemented and we are monitoring the results.

The issue has been fully resolved and all systems are operating normally. A detailed post-mortem report will be published here shortly. Thank you for your patience and cooperation.

## Delays in Asynchronous Actions and Workflows ‌ **Issue Summary** On May 28, 2026, we experienced an incident that caused delays in asynchronous actions and automated workflows. **Timeline \(UTC\)** * **11:29** – The issue began, causing delays in newly scheduled automated workflows. * **15:51** – Our engineering team identified the root cause of the scheduling interruption. * **16:42** – A fix was deployed, restoring normal operations to the background processing framework. * **18:45** – The resulting backlog of delayed tasks was completely cleared and normal service was confirmed. **Root Cause** A routine system update included a software dependency upgrade that unexpectedly conflicted with our background processing framework. This conflict prevented scheduled background jobs from moving into the active processing queue. As a result, automated workflows and other asynchronous actions were delayed, though no underlying data was lost or corrupted during this time. **Resolution** Upon investigating the anomaly in the scheduling queue, our engineering team determined that the recent dependency upgrade was responsible. We immediately reverted the update, which restored the flow of scheduled tasks. To resolve the issue as swiftly as possible, we temporarily scaled up our infrastructure's processing capacity to rapidly clear the backlog of delayed actions. **Preventative Measures** To prevent this type of issue from occurring in the future, we are planning on implementing the following improvements: * **Enhanced Alerting:** We are deploying strict, automated alerts specifically designed to detect anomalies in scheduled task queues, drastically reducing our time to detection. * **Upgraded Telemetry:** We are adding granular, time-series visualizations to our monitoring systems to better track the exact states of background jobs \(scheduled, enqueued, and processed\) in real time. We sincerely apologize for any disruption this may have caused to your operations. The issue was fully mitigated, and all backlogged tasks have been successfully processed.

We are currently investigating this issue.

This incident has been resolved.

We are currently investigating this issue.

This incident has been resolved.

We are currently investigating this issue.

This incident has been resolved.

We are currently investigating this issue.

This incident has been resolved.

Some users experienced an issue where app.circle.so was unavailable. This was caused by the Redis cluster reaching 100% CPU due to a large volume of workflow jobs being re-enqueued. We identified the root cause, scaled down the background deployment pods to relieve the load, and restored service within approximately 3 minutes. We are continuing to investigate and address the underlying issue.

We are currently investigating this issue.

This incident has been resolved.

We are currently investigating this issue.

This incident has been resolved.

We are currently investigating this issue.

This incident has been resolved.

We are currently investigating this issue.

This incident has been resolved.

We are currently investigating this issue.

This incident has been resolved.

We are currently investigating this issue.

This incident has been resolved.

Search functionality was degraded starting at approximately 6:52 PM EDT on April 14. The issue was caused by Elasticsearch cluster degradation. An incident report was filed internally. This issue should have had a status page entry since customers reported broken search. Customers checking [status.circle.so](http://status.circle.so) saw 'All Systems Operational' during this period.

Search functionality was degraded starting at approximately 6:52 PM EDT on April 14. The issue was caused by Elasticsearch cluster degradation. An incident report was filed internally. This issue should have had a status page entry since customers reported broken search. Customers checking [status.circle.so](http://status.circle.so/) saw 'All Systems Operational' during this period.

We are currently investigating this issue.

This incident has been resolved.

Communities experienced approximately 19 minutes of downtime starting at 12:17 AM EDT. The database crashed due to an out-of-memory error at the infrastructure level, a known AWS RDS bug. Our team raised a second escalation to AWS \(case 177616832700336\), and both cases have been escalated to AWS's highest priority tier. We are actively working with AWS to resolve the underlying infrastructure issue and have additional safeguards in place.

We are currently investigating this issue.

This incident has been resolved.

Communities experienced approximately 27 minutes of downtime starting at 9:50 PM EDT. The primary database crashed and went fully offline. Our infrastructure team performed a manual failover to a standby database in a different availability zone to restore service. The crash was caused by an AWS RDS bug that we had already escalated to AWS support. We are working with AWS to prevent recurrence and have additional monitoring in place to detect database instability earlier.

We are currently investigating this issue.

This incident has been resolved.

Communities experienced approximately 6 minutes of degraded availability at 5:14 PM EDT. This was a continuation of database load from the previous incident caused by an unoptimized query. The fix was deployed shortly after and resolved both issues.

We are currently investigating this issue.

This incident has been resolved.

Communities experienced approximately 3 minutes of degraded availability at 4:21 PM EDT. The issue was caused by an unoptimized database query in the live streams feature making 71 calls per second to the primary database. Our team identified the specific query and deployed a fix the same day to reduce query frequency.

We are currently investigating this issue.

This incident has been resolved.

Brief disruption \(~2 min\) at 6:40 AM EDT caused by a memory leak in web application pods. Pods exceeded their memory limits and were automatically restarted by the infrastructure. Our team increased the minimum pod count and enabled memory-based autoscaling at 70% threshold to prevent recurrence. Investigation into the root cause of the memory leak is ongoing.

Mobile API endpoints experienced intermittent 500 errors starting at 6:57 AM EDT on April 9. This was caused by a Rails 8.1 regression where a change in how ActiveSupport::CurrentAttributes managed state between requests caused crashes in OpenTelemetry instrumentation. Affected workers returned errors on every request, including health checks, causing pods to restart repeatedly. The root cause was identified at 7:45 AM EDT, and a fix was deployed. This issue should have had a status page entry since it impacted customers using the mobile app.

Mobile API endpoints experienced intermittent 500 errors starting at 6:57 AM EDT on April 9. This was caused by a Rails 8.1 regression where a change in how ActiveSupport::CurrentAttributes managed state between requests caused crashes in OpenTelemetry instrumentation. Affected workers returned errors on every request, including health checks, causing pods to restart repeatedly. The root cause was identified at 7:45 AM EDT, and a fix was deployed. This issue should have had a status page entry since it impacted customers using the mobile app.

We are currently investigating this issue.

This incident has been resolved.

Brief disruption \(~2 min\) at 7:21 AM EDT. This was the final occurrence of an intermittent issue caused by a Rails 8.1 framework regression. The root cause was identified 24 minutes later and a permanent fix was deployed.

We are currently investigating this issue.

This incident has been resolved.

Brief disruption \(~4 min\) caused by an intermittent application error related to a framework upgrade. The root cause was identified later that morning and a fix was deployed.