Cycode outage history

GitHub component: Pull Requests Original GitHub incident: https://stspg.io/ssd9z8l2g46v

GitHub component: Pull Requests Original GitHub incident: https://stspg.io/ssd9z8l2g46v

GitHub component: API Requests Original GitHub incident: https://stspg.io/3xn46bst0bjh

GitHub component: API Requests Original GitHub incident: https://stspg.io/3xn46bst0bjh

GitHub component: API Requests Original GitHub incident: https://stspg.io/3xn46bst0bjh

Customers may experience degraded performance in scans. Pull request and CLI scans may be affected.

The team has identified the root cause of the issue and is working on the solution.

The root cause has been resolved. The system began to stabilize itself, and all the scans are starting to get processed with regular performance.

All scan types except for SAST are fully operational. SAST continues to stabilize and will soon be fully stable.

System should be back to being fully operational.

**Root Cause** The incident was caused by a deployment of one service that ran a database index creation. The team has identified an issue with the way we perform index creations as database migrations. During a deployment the pods with newest image of the service attempted to create an index on a big table. The team has identified that the index creation took 7 minutes. However, during index creation pods were not responsive, and as a result, Kubernetes deemed them as unhealthy pods and attempted to retry those pods after 5 minutes. As a result, because the pod got killed before the index creation was fully completed, the database transaction was rolled back. Then, subsequent pods attempted to create the index again, dying after 5 minutes. This lead to the database being in unhealthy state, and the service was down. The team has rolled back the deployment, and killed all replicas that attempted to create the index. Thanks to that, the service and the database was in healthy state again. **Why safety measures did not help** Cycode provides a safety mechanism that unblocks all Pull Request scans after a specific period of time, giving each scan a maximum duration before the Pull Request is unblocked. However, because the service that is responsible for triggering and completing scans, as well as this safety net, was down, the process couldn't behave as expected. We acknowledge this gap and are working on strengthening this area of our system. **Action items** • The team is actively investigating enhancements and new safety protocols that can be put in place in order to have another safety net preventing Pull Request scans being stuck in case of any incident. • The team is investigating changes to the index creation process.

We’re investigating an issue causing older events to be reprocessed. **Impact:** Some workflows may run again, which could result in duplicate alerts. PR scans may also be delayed.

We have resolved the processing backlog caused by an issue during a Kafka migration. The issue resulted in older events being processed alongside new events, which caused delays and may have updated some violations with an outdated status. Normal processing has been restored. However, customers may still see some violations with an outdated status while we identify and correct the affected records. PR scans were not delayed or reprocessed, and workflows were not affected. We are continuing remediation and closely monitoring the system.

Normal processing has been restored, and the incident is now in monitoring. A small number of customers may still see a limited number of violations with an outdated status as a result of the incident. We have identified the potentially affected environments and are working to correct the impacted records.

The system is fully operational. A small number of customers may still see a limited number of violations with an outdated status as a result of the incident. We have identified the potentially affected environments and are working to correct the impacted records.

The platform is now fully operational and processing normally

The team identified an issue with degraded performance with scanning. There can be delays in starting, running, and completing scans. All scan types may be affected (Pull request and CLI scans as well). The team has identified an issue with a malfunction in deployment, and the issue should be resolved shortly.

The team has identified the root cause and resolved it. We are seeing the system getting back to stability.

The system is now back to being fully operational.

**Root Cause** The incident was caused by a deployment of one service that ran a database migration. The team has identified that this migration contained faulty code and as a result lead to database overload when attempting to deploy the service. As a result, the service was partially down until the deployment was reverted. During this time, all scans were processed with lower than expected performance.

GitHub component: API Requests Original GitHub incident: https://stspg.io/vr201n49yl53

GitHub component: API Requests Original GitHub incident: https://stspg.io/vr201n49yl53

GitHub component: Pull Requests Original GitHub incident: https://stspg.io/sm1tp7kfm4vj

GitHub component: Pull Requests Original GitHub incident: https://stspg.io/sm1tp7kfm4vj

GitHub component: Pull Requests Original GitHub incident: https://stspg.io/sm1tp7kfm4vj

We have noticed degraded performance in IaC Pull Request scans. We are troubleshooting the issue.

We have identified the root cause of the slownesses and are putting countermeasures in place.

The lag that led to slownesses is almost over. We are monitoring the situation.

The issue has been resolved, and we are continuing to monitor the situation.

GitHub component: API Requests, Pull Requests Original GitHub incident: https://stspg.io/j5c80shxqm53

GitHub component: API Requests, Pull Requests Original GitHub incident: https://stspg.io/j5c80shxqm53

GitHub component: Webhooks Original GitHub incident: https://stspg.io/syhr80rth84z

GitHub component: Webhooks, Pull Requests Original GitHub incident: https://stspg.io/syhr80rth84z

GitHub component: Webhooks, Pull Requests Original GitHub incident: https://stspg.io/syhr80rth84z

At **1:41** PM EDT, we identified an issue causing elevated error rates during request processing in the Platform UI. The issue was mitigated promptly, and the platform is currently operating normally. Our team continues to actively monitor the platform to ensure service stability.

The incident has been resolved, and the platform is operating normally.

We have noticed a portion of CLI Secrets scans failing. We are actively triaging the problem.

We have identified that CLI version 3.17.1 introduced the faulty behavior. Degrading the CLI version to 3.17.0 should temporarily resolve the issue while we continue to understand and resolve the root cause.

A fix has been applied and functionality is fully restored; we are continuing to monitor to ensure everything remains stable.

Due to increased scan loads, we observed delays in violation status updates which includes auto-resolution of violations. The source of the sudden increase has been mitigated and the delay is already decreasing. We will keep monitoring the situation until we're back to normal

We’re continuing to monitor detection processing and the associated delays in violation status updates (including auto-resolution)

We have noticed degraded performance in multiple system components. We are identifying all affected components and identifying the root cause.

We have noticed degraded performance in multiple components of the application UI. Scans, as well as Pull Request and CLI scans may also be affected.

We are observing elevated Redis timeout rates. We are scaling Redis cluster capacity to mitigate the impact and restore stable service performance

The region has recovered and is operating normally. We are closely monitoring system performance to ensure continued stability

The system is now fully operational. There should be no more degraded performance. **Summary** We observed a period of slowness and intermittent timeouts affecting various system functions in the EU region, including the application interface and pull request (PR) scans. The issue was primarily caused by a processing system reaching its network and memory capacity limits, exacerbated by a high volume of automated activity from a single source. We have since upgraded the underlying infrastructure and implemented safeguards to prevent similar high-volume activity from impacting the system. The issue is now fully resolved, and all services have returned to expected performance levels. **Key Timeline (IDT)** • **July 13, 2026, 11:44 IDT**: Incident detected following reports of UI slowness and PR scan delays. • **July 13, 2026, 12:19 IDT**: Infrastructure bottleneck identified; decision made to upgrade the processing cluster. • **July 13, 2026, 12:26 IDT**: A high-volume automated process was identified and disabled to reduce immediate load. • **July 13, 2026, 13:09 IDT**: Infrastructure upgrade completed; network throughput returned to normal levels. • **July 13, 2026, 15:35 IDT**: All backlogs cleared, and the incident was officially resolved. **Root Cause** The incident was triggered by a combination of factors: a processing cluster reached its maximum network bandwidth and memory capacity due to an undersized configuration for the current workload. This was further strained by a specific automated workflow that generated an unusually high volume of update requests. Additionally, a configuration difference in the message processing pipeline in the EU region prevented the system from effectively handling the resulting backlog. **Actions Taken** • **Upgraded Infrastructure**: The processing cluster was upgraded to a higher-capacity instance type to provide more network bandwidth and memory. • **Disabled High-Volume Source**: A specific client identifier responsible for excessive traffic was temporarily disabled to restore system stability. • **Restored Connectivity**: Affected service components were restarted to ensure they re-established clean connections to the upgraded infrastructure. • **Increased Processing Parallelism**: The number of partitions in the affected message queue was increased to allow the system to process the backlog more quickly. **Action Items** • **Enhance Monitoring**: Implement new alerts for network and memory utilization to detect capacity issues before they impact customers. • **Optimize Update Workflow**: Refactor the status update process to batch requests, significantly reducing the load on the processing system. • **Implement Rate Limiting**: Introduce safeguards to prevent any single source from consuming disproportionate system resources. • **Standardize Regional Configurations**: Conduct an audit to ensure infrastructure and message queue settings are consistent across all regions.

We’ve identified an issue that may cause **inaccurate violation counts** in **some** product dashboards and custom dashboard panels that rely on violation data (not all dashboards are affected). We’ve already begun corrective work, but it will take time to fully complete, and you may see counts change as data is corrected. We’ll share another update once the fix has finished running and data accuracy is fully restored.

We have made significant progress in correcting the inaccurate violation counts affecting some product and custom dashboards. • **Current Status:** The fix has successfully completed for the vast majority of accounts, and full data accuracy has been restored. • **Next Steps:** We are actively resolving the issue for the small number of remaining affected accounts.

Functionality is fully restored; we are continuing to monitor to ensure everything remains stable.

We are currently investigating an issue affecting Maestro Risk Explorability, Risk AI Remediation, Maestro Remediation, and Graph AI services.

We identified a firewall configuration issue that was impacting Maestro AI services. The configuration has been updated, and the affected services have recovered. We are continuing to monitor the situation and are working on further stabilization. Some degraded performance may still be observed while we complete additional improvements.

The issue affecting Maestro AI services has been resolved. Maestro Risk Explorability, Risk AI Remediation, Maestro Remediation, and Graph AI are now available and operating normally. **Summary** On July 9, 2026, customers using the Maestro service in the European production environment experienced a period of service unavailability. The issue began following a configuration update that inadvertently changed the service's regional routing. This caused the system to attempt connections through a network path that lacked the necessary permissions and to a region where specific processing models were unavailable. The issue has been fully resolved, and service has been restored to all affected customers. **Key Timeline (IDT)** • **July 9, 2026, 12:02 IDT:** The incident was identified and an investigation was initiated. • **July 9, 2026, 12:07 IDT:** Public notification was issued regarding the service interruption. • **July 9, 2026, 13:00 IDT:** A network configuration fix was applied, restoring primary connectivity. • **July 9, 2026, 13:39 IDT:** Service was fully restored after implementing model fallbacks, and the incident was marked as resolved. **Root Cause** The service interruption was triggered by a recent update to the authentication and configuration process. This update introduced a conflict in how the system identified its operating region. Specifically, an automated update process overrode manual settings, routing traffic to a different regional endpoint. This new path was blocked by a missing network security rule and attempted to use a processing model that was not supported in that specific region, leading to service failure. **Actions Taken** • **Restored Network Connectivity:** Manually updated network security rules to allow secure traffic through the new regional endpoint. • **Implemented Model Fallbacks:** Configured the system to use alternative processing models to ensure immediate service availability while long-term regional configurations were adjusted. • **Updated Status Communications:** Maintained real-time updates for stakeholders and customers throughout the recovery process. **Action Items** • **Standardize Configuration Precedence:** Update the deployment workflow to prevent automated processes from silently overriding critical environment settings. • **Infrastructure Audit:** Conduct a comprehensive review of network security rules across all regions to ensure consistency and prevent similar connectivity gaps. • **Enhance Automated Monitoring:** Implement end-to-end health checks and synthetic probes to detect regional connectivity issues automatically before they impact users. • **Improve Deployment Policies:** Establish new guidelines to ensure that configuration changes are deployed and validated in production-like environments more frequently to reduce the risk of "stale" updates.

We are experiencing delays in infrastructure provisioning caused by cloud provider API rate limiting. We are actively investigating the issue with our cloud provider.

Please refer to the AWS Health Status page for details on the related incident: [https://health.aws.amazon.com/health/status](https://health.aws.amazon.com/health/status "https://health.aws.amazon.com/health/status")

Mitigation: We temporarily scaled up the managed node group to get pods scheduled while we wait for AWS to fully resolve the underlying issue.

We are starting to see stabilization and a reduction in API errors. However, we continue to closely monitor the situation.

AWS has confirmed that the issue has been fully mitigated and we are currently not observing any related issues.

**Investigating - Issues with Violations and Custom Dashboards (Prod-US)** We are currently investigating an issue in our **Prod-US** environment where violations are failing to load. As a result, custom dashboard panels that rely on violation data may also fail to render or display errors. Our engineering team is actively looking into the root cause, and we will provide updates here as we learn more. We apologize for the inconvenience.

A fix has been deployed for the issues affecting violations and custom dashboards in Prod-US. We are actively monitoring the environment to ensure services are fully restored.

The core issue has been resolved, and violations and custom dashboards should now be working as normal. Our team is actively monitoring the data sync to resolve any remaining discrepancies with newer violations. We will provide a final update once the sync is complete.

We are continuing to monitor the data synchronization process for new violations in the UI. While functionality has been restored, it may take up to **6 hours** for all recent data to fully catch up and reflect accurately. We will provide a final update once the synchronization is complete.

The data synchronization is complete, and all recent violations have successfully populated in the UI. Violations and custom dashboards are functioning normally, and the incident is fully resolved. We appreciate your patience as we worked to restore full service.

Functionality is fully restored; we are continuing to monitor to ensure everything remains stable.

We have identified the source of an issue and currently deploying the fix. At the same time we scaled our scanning platform up to accelerate scanning

The fix was deployed. The queue is decreasing and we're monitoring it

The system has processed all jobs with higher priorities. There is a queue of lower priority jobs that should not impact overall Cycode scanning performance

**Summary** During the incident, customers experienced significant delays and temporary disruptions across SAST, SCA, CCA, and Secret repository scans and push events. The issue was caused by a surge in reachability scanner jobs that overwhelmed the processing queue, compounded by scanner pods requesting excessive CPU and memory, infrastructure resource limits being reached, and inefficiencies in job prioritization and retry logic. As a result, processing capacity was improperly consumed and a large job backlog accumulated. A series of corrective updates were deployed to stabilize the environment, and the processing environment has since returned to expected performance levels. **Impact** Customers experienced delays for SAST, SCA, CCA, and Secret repository scans and push events, with some requests delayed by several hours and a peak queue size of over 64,000 jobs. Lower priority scans such as Trivy, Syft, and CCA were most affected, though high-priority jobs were eventually processed without further delay. **Key Timeline (IDT)** • **21.06.2026, 17:16 IDT**: A surge in reachability scanner jobs caused the CycodeX queue to grow rapidly. • **22.06.2026, 10:07 IDT**: The issue was identified by an on-call engineer. • **22.06.2026, 12:55 IDT**: We increased the scanning platform resources to process more jobs. • **22.06.2026, 14:10 IDT**: A fix that lowered new reachability scanners was deployed to production. • **22.06.2026, 18:43 IDT**: Existing reachability scanners' priority was lowered. • **23.06.2026, 09:12 IDT**: Scans with higher priority were processed. Only lower priority scans remained, including CCA. • **23.06.2026, 13:51 IDT**: A fix that reduced communication overload to Kubernetes was deployed. The scanning platform started processing scan jobs much faster. • **23.06.2026, 17:34 IDT**: The queue was fully processed. **Root Cause** The issue was triggered by a combination of factors: 1. **Reachability scanner job surge** -- A surge in reachability scanner jobs caused the CycodeX queue to grow rapidly, which led to resource bottlenecks in the cluster and a peak queue size of over 64,000 jobs. 2. **Excessive pod resource requests** -- Due to configuration bugs, scanner pods requested excessive CPU and memory, which prevented efficient scheduling and amplified the resource bottlenecks in the cluster. 3. **Infrastructure resource limits** -- AWS VPC subnet IP and EKS API limits were reached, restricting the cluster's ability to scale and schedule new work. 4. **Job prioritization and retry inefficiencies** -- Inefficiencies in job prioritization and retry logic meant lower priority scans (Trivy, Syft, CCA) competed for capacity and were most affected, while the backlog continued to grow. **Actions Taken** • Increased cluster and node pool capacity. • Fixed job prioritization to deprioritize reachability scans. • Capped resource requests for scanner pods to enable efficient scheduling. • Deployed additional fixes to the scanning platform. • Opened AWS support tickets to address resource limits. • Restored monitoring and logging. • Cleared the job backlog; the queue now processes new jobs as they arrive. **Action Items** • Improve monitoring to better understand the behavior of the processing environment. • Improve scanning optimization and prioritization for all scan types.

**Problem**: SAST (Static Application Security Testing) scans for pull requests were running slowly **Impact**: Some users experienced slow pull request scans potentially delaying code reviews and deployments.

The issue was resolved. The system is fully stable now