Justcall 장애 기록

1. 2026년 9월 1일 PM 5:44 UTCidentified

   우리는 현재 GCP us-central1의 지속적인 인프라 문제로 인해 Calling, Sales Dialer 및 Inbound Call Services에 영향을 미치는 부분적인 서비스 분해를 경험하고 있으며, 음성 공급자에서 API 고장을 높였습니다. 몇몇 고객은 간헐적인 외침 실패 또는 연결 문제점을 경험할지도 모릅니다. 우리의 팀은 적극적으로 사건을 모니터링하고 가능한 한 빨리 전체 서비스 가용성을 복원 작업. GCP 상태: https://status.cloud.google.com/incidents/J5ia5t9p3g9Q5Wi7r8Ev 음성 공급자 상태: https://status.twilio.com/incidents/3txtvjthljkt

2. 2026년 9월 1일 PM 7:51 UTCresolved

   이 사건은 해결되었습니다.

공식 인시던트 업데이트를 자동 번역했습니다.

1. 2026년 8월 18일 PM 3:27 UTCidentified

   문제는 식별되고 수정이 구현됩니다.

2. 2026년 8월 18일 PM 5:37 UTCmonitoring

   수정이 구현되어 결과를 모니터링하고 있습니다.

3. 2026년 8월 18일 PM 6:50 UTCresolved

   이 사건은 해결되었습니다.

공식 인시던트 업데이트를 자동 번역했습니다.

1. 2026년 7월 10일 PM 3:23 UTCinvestigating

   우리는 현재이 문제를 조사하고 있습니다.

2. 2026년 7월 10일 PM 3:25 UTCinvestigating

   이 문제를 조사하고 있습니다.

3. 2026년 7월 10일 PM 4:05 UTCmonitoring

   수정이 구현되어 결과를 모니터링하고 있습니다.

4. 2026년 7월 10일 PM 4:08 UTCresolved

   이 사건은 해결되었습니다.

공식 인시던트 업데이트를 자동 번역했습니다.

1. 2026년 6월 11일 PM 3:50 UTCinvestigating

   We’re aware that some customers are experiencing slow load times in the JustCall. Our team is actively investigating the cause.

2. 2026년 6월 11일 PM 4:05 UTCmonitoring

   We’ve identified the issue and applied a fix. Latencies are back to normal levels, and we’re monitoring closely to confirm full recovery.

3. 2026년 6월 11일 PM 4:06 UTCresolved

   This incident is now resolved. We apologize for the disruption — if you’re still seeing slowness, please reach out to our support team.

1. 2026년 3월 9일 PM 12:22 UTCinvestigating

   We are currently investigating this issue.

2. 2026년 3월 9일 PM 12:24 UTCidentified

   The issue has been identified and a fix is being implemented.

3. 2026년 3월 9일 PM 12:25 UTCresolved

   This incident has been resolved. This seems to have been caused by an underlying GCP issue with GKE in the region where our system is hosted.

4. 2026년 3월 9일 PM 12:35 UTCpostmortem

   We experienced a temporary service disruption that caused a subset of Voice requests to experience elevated response times and intermittent errors. This occurred due to a brief internal connectivity issue within our infrastructure, which prevented a subset of requests in that zone from reaching healthy service instances. This seems to have been caused by an underlying GCP issue with GKE in the region where our system is hosted. Connectivity was restored shortly thereafter, and the system automatically recovered as traffic resumed normal routing.

1. 2025년 12월 1일 PM 6:21 UTCinvestigating

   We're currently investigating the slowness in the JustCall webapp for some of our customers.

2. 2025년 12월 1일 PM 6:23 UTCidentified

   The issue has been identified and a fix is being implemented.

3. 2025년 12월 1일 PM 6:23 UTCresolved

   This incident has been resolved.

1. 2025년 11월 3일 AM 11:03 UTCinvestigating

   We are currently investigating this issue.

2. 2025년 11월 3일 AM 11:04 UTCidentified

   The issue has been identified as an invalid intermediate certificate configuration affecting a subset of edge nodes. A fix is being deployed.

3. 2025년 11월 3일 AM 11:05 UTCmonitoring

   The fix has been deployed, and SSL validation errors should now be resolved. We are monitoring system stability before marking this incident as resolved.

4. 2025년 11월 3일 AM 11:06 UTCresolved

   This incident has been resolved.

1. 2025년 10월 20일 PM 6:26 UTCmonitoring

   We are currently experiencing service disruptions caused by an ongoing AWS outage affecting our upstream provider. Our engineering teams are actively working with the provider to restore full functionality as soon as possible. Impact: Some customers may experience intermittent failures when making or receiving calls or SMS messages. Workaround: If you encounter a failure, please retry the action as a temporary workaround. We’ll continue to monitor the situation closely and will share updates as more information becomes available.

2. 2025년 10월 20일 PM 6:47 UTCmonitoring

   We are continuing to monitor for any further issues.

3. 2025년 10월 20일 PM 7:15 UTCresolved

   This incident has been resolved.

1. 2025년 10월 20일 AM 9:47 UTCinvestigating

   We are currently investigating this issue.

2. 2025년 10월 20일 AM 9:49 UTCidentified

   We have identified that the issue originates from our upstream providers, who are currently impacted by a widespread AWS outage in the US-East-1 (N. Virginia) region. You can view the latest status update from AWS here - https://health.aws.amazon.com/health/status?eventID=arn:aws:health:us-east-1::event/MULTIPLE_SERVICES/AWS_MULTIPLE_SERVICES_OPERATIONAL_ISSUE/AWS_MULTIPLE_SERVICES_OPERATIONAL_ISSUE_BA540_514A652BE1A

3. 2025년 10월 20일 AM 9:51 UTCmonitoring

   SMS services have been successfully restored. Our team continues to monitor their performance and is actively collaborating with our upstream providers to expedite the restoration of calling services.

4. 2025년 10월 20일 AM 10:40 UTCmonitoring

   Calls can now be made and received through the Mobile App. Our team is actively working to restore outbound calling functionality on the Web Dialer. In the meantime, we recommend using the Mobile App to place outbound calls as a temporary workaround.

5. 2025년 10월 20일 AM 10:41 UTCmonitoring

   We are continuing to monitor for any further issues.

6. 2025년 10월 20일 AM 11:32 UTCresolved

   This incident has been resolved.

1. 2025년 9월 12일 PM 12:30 UTCinvestigating

   We are currently investigating this issue.

2. 2025년 9월 12일 PM 12:31 UTCidentified

   The issue has been identified and a fix is being implemented.

3. 2025년 9월 12일 PM 12:31 UTCresolved

   This incident has been resolved.

4. 2025년 9월 12일 PM 12:36 UTCpostmortem

   **Date:** 12th September 2025 **Duration:** 02:45 UTC - 02:53 UTC **Affected Services** * Sales Dialer **Impact** A subset of customers experienced degraded performance or complete unavailability of the Sales Dialer app. Some operations might also be delayed. **Root Cause** * Outdated cached offset limits caused inefficient query execution * High memory and CPU consumption on the database cluster * Resource exhaustion leading to temporary unavailability * Overall reduction in system responsiveness until intervention **Resolution** * Engineers identified and corrected the outdated caching behavior in query handling * Optimized query execution to prevent excessive resource usage * Applied additional safeguards and monitoring to mitigate similar issues in the future

1. 2025년 7월 16일 AM 7:58 UTCidentified

   We have identified an issue affecting a subset of customers, where credit deductions are occurring at a higher rate than intended. Our team is actively investigating and working to resolve the problem.

2. 2025년 7월 16일 AM 7:59 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

3. 2025년 7월 16일 AM 7:59 UTCresolved

   This incident has been resolved.

4. 2025년 7월 16일 AM 8:23 UTCpostmortem

   **Date:** 2025-07-11 **Duration:** 12:00 - 12:30 UTC ### **Affected Services** * Outbound/Inbound Calling * SMS Delivery * IQ Transcriptions * Other usage-based services relying on balance deduction logic ### **Impact** A subset of customers experienced **excessive and inaccurate balance deductions** across multiple services including: * **Outbound calls** * **SMS messaging** * **IQ and transcription usage** This incorrect deduction affected customer billing visibility and usage quotas, resulting in: * Unexpected balance exhaustion * Interrupted service usage due to perceived insufficient balance ### **Root Cause** The incident was caused by **flawed cache key reuse** in the currency conversion logic used during balance deduction operations: * Cache entries related to currency conversion rates \(e.g., USD to other currencies\) were **improperly reused across services**. * As a result, **USD-denominated prices were incorrectly multiplied by cached exchange rates** meant for other contexts or currencies. * This led to **multiplied deductions**, far beyond actual usage cost. ### **Resolution** * Engineers immediately turned on the circuit breaker to ensure customers are able to use Calling, SMS and other services. * Engineers identified the faulty cache key reuse in the balance deduction service. * Flushed the cache to restore cache integrity * Affected customer balances were audited and **refunds/credits processed** where applicable.

1. 2025년 7월 9일 PM 2:42 UTCinvestigating

   Some customers are experiencing issues with loading the JustCall dashboard, and failures in sending SMS, MMS, and making or receiving calls via the web interface. Investigation is ongoing.

2. 2025년 7월 9일 PM 2:44 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

3. 2025년 7월 9일 PM 2:45 UTCresolved

   This incident has been resolved.

4. 2025년 7월 10일 PM 3:50 UTCpostmortem

   **Date:** 9th July 2025 **Duration:** 14:51 UTC - 15:25 UTC **Affected Services:** * Customer-facing Dashboard * SMS Delivery * MMS Delivery * Inbound Calling * Outbound Calling * Associated internal services **Impact** A subset of customers experienced degraded performance or complete unavailability in: * Accessing the dashboard * Users experienced delays in sending and receiving SMS and MMS messages. * Intermittent degradation in both inbound and outbound call performance. * Delayed or failed service actions tied to within a designated cloud service region. **Root Cause** A malfunction in one of our cache systems triggered a reboot, leading to the invalidation of all cache entries in that system: * Massive cache miss rate * Significant surge in requests directly hitting primary databases and downstream services * Snowball/piggyback effect where database latency grew, increasing queue sizes and causing cascading slowdowns across services * Overwhelmed compute resources and degraded service health in that region. **Resolution** * Engineers manually restored cache to a known good state from cache backups. * Temporarily scaled up database resources and async queues.

1. 2025년 4월 7일 PM 12:01 UTCinvestigating

   We've observed an increased rate of errors where users are experiencing difficulty in making & receiving calls.

2. 2025년 4월 7일 PM 12:04 UTCidentified

   The issue has been identified and a fix is being implemented.

3. 2025년 4월 7일 PM 12:05 UTCresolved

   This incident has been resolved.

1. 2024년 12월 10일 PM 5:45 UTCinvestigating

   We are currently investigating this issue.

2. 2024년 12월 10일 PM 5:53 UTCresolved

   This incident has been resolved.

1. 2024년 1월 31일 PM 4:45 UTCinvestigating

   We are currently investigating this issue.

2. 2024년 1월 31일 PM 5:27 UTCresolved

   This incident has been resolved.

3. 2024년 1월 31일 PM 5:58 UTCpostmortem

   We experienced an unexpected increase in latency in one of our API services. This issue resulted in degraded performance for some of our users, causing inconvenience and disruptions. Users were temporarily re-routed from https://app.justcall.io to https://justcall.io After a thorough investigation, the root cause of the API latency was identified as increased database response time. The API latency issue has been successfully resolved, and normal operations have been restored. We sincerely apologize for any inconvenience this may have caused and appreciate your patience and understanding.

1. 2023년 9월 1일 PM 12:15 UTCinvestigating

   The web application is loading slow for some of the users. Our team is investigating it.

2. 2023년 9월 1일 PM 12:18 UTCidentified

   The issue has been identified and a fix is being implemented.

3. 2023년 9월 1일 PM 12:19 UTCresolved

   This incident has been resolved.

1. 2023년 6월 14일 PM 2:18 UTCinvestigating

   We've observed an increased rate of errors where users are trying to open the JustCall Dialer.

2. 2023년 6월 14일 PM 2:19 UTCidentified

   The issue has been identified and a fix is being implemented.

3. 2023년 6월 14일 PM 2:20 UTCresolved

   This incident has been resolved.

1. 2023년 6월 1일 PM 3:50 UTCinvestigating

   We've observed an increased rate of errors where users are experiencing difficulty in receiving incoming calls.

2. 2023년 6월 1일 PM 3:55 UTCidentified

   The issue has been identified and a fix is being implemented.

3. 2023년 6월 1일 PM 3:55 UTCresolved

   This incident has been resolved.

1. 2023년 5월 23일 PM 4:05 UTCinvestigating

   High latency in loading the JustCall app was reported by our monitoring system.

2. 2023년 5월 23일 PM 4:06 UTCinvestigating

   We are continuing to investigate this issue.

3. 2023년 5월 23일 PM 4:08 UTCidentified

   The issues have been identified and the team is actively working to make the system stable and implement a fix.

4. 2023년 5월 23일 PM 4:15 UTCmonitoring

   The team was able to address the issue. We will continue monitoring the service to ensure a full recovery.

5. 2023년 5월 23일 PM 4:19 UTCmonitoring

   We are continuing to monitor to ensure a full recovery.

6. 2023년 5월 23일 PM 4:20 UTCresolved

   This incident has been resolved.

1. 2023년 4월 28일 PM 12:22 UTCresolved

   On April 26th at 7:00 PM (GMT), we discovered issues with contact syncing and logging of calls and SMS in Keap (formerly Infusionsoft) CRM. We resolved these problems by April 27th at 0:00 AM (GMT) and are currently working on syncing the affected calls and SMS. We anticipate that the syncing process will be finished by April 30th at 0:00 AM (GMT).