Historia awarii Cloudgov

From roughly 9:20 to 9:55 AM on Wednesday, August 5 2026, the SES service broker experienced a partial outage. During this time, the SES service broker was unavailable, meaning that customers could not create new services, update existing services, or delete existing services. However, existing services created by the broker continued to function as normal. As of 9:55 AM, the outage was resolved and the SES service broker was fully operational. To identify the causes of this incident and to prevent it from recurring, the Cloud.gov team will conduct an incident post-mortem and share our findings in the coming days. Thanks for being a Cloud.gov customer.

# SES service broker outage on August 5, 2026 ## Summary From approximately 9:13 AM ET to 9:45 AM ET on Wednesday August 5, 2026,[ ](http://cloud.gov)the [SES service broker](https://docs.cloud.gov/platform/services/aws-ses/) was unavailable. During this time, customers could not create, update, or delete brokered SES services. The outage affected only the management of brokered SES services. Existing service instances and email delivery were not interrupted. ## Impact During the incident, customers were unable to: * Create new brokered SES services * Update existing brokered SES services * Delete existing brokered SES services No action was required from customers after service was restored. ## Cause An automated cleanup process deleted a container image that was still being used by the deployed SES service broker. When the broker restarted during a production deployment, it could not retrieve the image and failed to start. ## Timeline All times are Eastern Time. * August 4, 4:24 AM - An automated pipeline created a new container image for the SES service broker * August 4, 3:03 PM - An automated process deleted the container image used by the deployed broker * August 5, 7:17 AM - A [Cloud.gov](http://cloud.gov) production deployment began. * August 5, approximately 9:13 AM - The broker attempted to restart but could not retrieve its container image. * August 5, approximately 9:28 AM - The [Cloud.gov](http://cloud.gov) team received an alert and began investigating. * August 5, approximately 9:45 AM - Engineers manually redeployed the broker with an available container image and restored service. ‌ **Resolution** [Cloud.gov](http://cloud.gov) manually redeployed the SES service broker using an available container image. The team then reran the automated deployment pipeline and confirmed that the deployment and acceptance tests completed successfully. **Follow-up Actions** [Cloud.gov](http://cloud.gov) is taking the following actions to reduce the risk of a similar incident and improve recovery: * Improve monitoring and alerting for SES service broker outages. * Update the container image cleanup process so it does not delete images that are still in use. * Update the deployment process so the SES service broker references the intended current container image. Thank you for your patience while we resolved this issue. Questions may be sent to us at [[email protected]](mailto:[email protected]).

Service brokers that Cloud.gov customers use to manage AWS resources (e.g. RDS, S3, etc) are currently experiencing an outage. The Cloud.gov team is investigating and will provide further updates as we know more.

The Cloud.gov team has identified the cause of the outage as inadvertent access key rotation for Cloud.gov applications that manage AWS resources, such as service brokers for RDS databases. The AWS access keys themselves that customers use to access their resources (e.g. RDS databases, S3 buckets) **are unaffected and are still working**. The Cloud.gov team is actively working to restore functionality for services that manage AWS resources.

All services on Cloud.gov that manage AWS resources should now be operational. If you are still experiencing issues, please contact us at [email protected].

Since we deployed our fixes at approximately 5:27 PM ET, the Cloud.gov team has observed no further indications of failures from the services that manage AWS resources. All customer-facing services and service brokers should be fully operational. If you are still experiencing issues, please contact us at [email protected]. As always, the Cloud.gov team will be conducting a post-mortem analysis of this incident in the coming days. We will share our findings and next steps once our analysis is complete.

**Summary** From approximately 12:22 PM to 7:19 PM ET on Thursday, June 11, 2026, [Cloud.gov](http://cloud.gov) service broker management actions were unavailable. During this window, customers could not create, update, or delete brokered services. Existing brokered services, including S3 buckets, RDS databases, and other already provisioned services, remained available. Customer applications continued to run, and we did not observe application downtime from this incident. **Impact** During the incident, customers were unable to: * Provision new brokered services * Update existing brokered services * Delete existing brokered services Existing service instances and customer applications were not interrupted. **Timeline** All times are Eastern Time. * 12:22 PM - An automated credential rotation process changed IAM credentials used by [Cloud.gov](http://cloud.gov) service brokers. * 12:57 PM - A [Cloud.gov](http://cloud.gov) engineer identified that some service broker IAM credentials were no longer working as expected. * 1:35 PM - The [Cloud.gov](http://cloud.gov) team began coordinated incident response. * 2:34 PM - The [Cloud.gov](http://cloud.gov) team began restoring IAM credentials and redeploying affected service brokers. * 7:19 PM - The final service broker was redeployed. All [Cloud.gov](http://cloud.gov) service brokers were confirmed operational. **Resolution** [Cloud.gov](http://cloud.gov) created replacement IAM credentials, redeployed the affected service brokers, and tested broker operations. After validation, the team confirmed that customers could again create, update, and delete brokered services. **Follow-up Actions** [Cloud.gov](http://cloud.gov) will take the following actions to reduce the change of a similar incident and improve recovery: * Add alerting to detect service broker IAM credential failures sooner. * Review IAM credential rotation procedures for service brokers. * Add stronger safeguards for automated credential rotation. * Improve automation for redeploying service brokers after credential changes. * Review incident response runbooks for service broker credential failures. Thank you for your patience while we resolved this issue. If you have any questions, please contact us at [[email protected]](mailto:[email protected]).

Some customers trying to access logs.fr.cloud.gov might get HTTP 404 errors. Cloud.gov support staff have identified the issue and are working on deploying a resolution. Customers are asked to keep re-trying the website to gain access while the fix is being implemented. PLEASE NOTE - this issue only involves customer access to the logs front end. Logs and metrics are still flowing from the platform and apps into the logs system and are not affected. Log alerting is also not affected at this time. Thank You

The fix has been implemented and Cloud.gov support is monitoring the system at this time. Customers should be able to access the front end of logs.fr.cloud.gov again.

This incident has been resolved.

Zendesk, the support ticket system used by Cloud.gov, is currently experiencing email relay issues and not generating tickets when emails are sent to [email protected]. The Cloud.gov support staff are receiving the emails and can respond directly to customers while Zendesk fixes their relay issues. Zendesk status: https://status.zendesk.com/

This incident has been resolved.

Summary From March 17, 2026 around 9:30 AM ET to March 19, 2026 around 3:10 PM ET, some application JSON logs were not ingested successfully to Cloud.gov Logs. Timeline - March 6, 2026, 10:25 AM ET - Changes were merged to the logging system configuration to make some subfields of application JSON logs searchable and aggregatable - March 17, 2026, 9:30 AM ET - Changes to JSON log field parsing begin deploying to Cloud.gov Logs system. Some JSON application logs began to fail ingestion at this point - March 19, 2026, 2:39 PM ET - Automated testing alerts Cloud.gov engineers to ingestion failures in Cloud.gov Logs. Engineers begin to investigate. - March 19, 2026, 2:59 PM ET - A Cloud.gov engineer determines that updated JSON application log parsing of timestamp fields is causing some logs to fail ingestion and slowing the overall log ingestion rate - March 19, 2026, 3:06 PM ET - A fix is deployed to change the field type to “string” for JSON log timestamp fields March 19, 2026, 3:10 PM ET - JSON log ingestion errors are resolved and logs are ingesting successfully Impact Only JSON application logs that included a “ts” field or a “timestamp” field which could not be properly parsed as a date (e.g. “1.8543923523”) failed to ingest to Cloud.gov Logs during the incident. While these logs were not ingested to Cloud.gov Logs successfully, they were ingested to offline storage that is not accessible to customers, but can be accessed by the Cloud.gov engineers if necessary. Resolution The type of the timestamp field for JSON application logs was changed to “string”, which allows even values that aren’t valid timestamps or dates to be ingested. Next Steps The Cloud.gov team will hold a retrospective to further analyze the causes of this incident and how to improve our operations. We will post our findings as a post-mortem to this incident. Thank you for your patience. If you have any questions, please contact us at [email protected].

Cloud.gov support is looking into an issue when inviting new users to the platform using the Cloud.gov IDP provider. PLEASE NOTE - this only effects new Cloud.gov IDP users to the platform and not preventing EXISTING users from signing in. New users on existing IDP providers (census.gov, DOJ.gov, dol.gov, EPA.gov, FDIC.gov, fec.gov, frtib.gov, GSA.gov, mcc.gov, nih.gov, OMB.gov, onrr.gov, usaid.gov, and SSA.gov) are NOT affected.

Cloud.gov support is continuing to investigate the root cause of inviting new users. If users are being blocked please reach out to Cloud.gov support at [email protected] for a manual workaround.

Cloud.gov support has identified the issue with new user signups and is currently implementing a fix thorough lower environments first before production. We plan to update production tomorrow Wednesday 12/31/25. If users are being blocked please reach out to Cloud.gov support at [email protected] for a manual workaround.

Cloud.gov has applied the new user signup fix into production and the process is now working as expected. If you encounter any issues signing up for Cloud.gov, please reach out to Cloud.gov support at [email protected].

From approximately December 11, 2025 7:00 AM ET to December 31, 2025 7:30 AM ET, new [Cloud.gov](http://Cloud.gov) users attempting to complete new user signup not using their own agency IdP could not have completed the process. ‌ New users on existing IDP providers \([census.gov](http://census.gov), [DOJ.gov](http://DOJ.gov), [dol.gov](http://dol.gov), [EPA.gov](http://EPA.gov), [FDIC.gov](http://FDIC.gov), [fec.gov](http://fec.gov), [frtib.gov](http://frtib.gov), [GSA.gov](http://GSA.gov), [mcc.gov](http://mcc.gov), [nih.gov](http://nih.gov), [OMB.gov](http://OMB.gov), [onrr.gov](http://onrr.gov), [usaid.gov](http://usaid.gov), and [SSA.gov](http://SSA.gov)\) were NOT affected. ‌ The issue was related to a newer version of UAA for cross-site request forgery \(CSRF\) token enforcement of authenticated users from unknowingly executing unwanted actions with their current session.  An updated version of the activation form was rolled out to production the morning of December 31st. ‌ There is no customer required action.  If customers experience problems signing up at [Cloud.gov](http://cloud.gov), please reach out to [Cloud.gov](http://cloud.gov) support at [[email protected]](mailto:[email protected]).  ‌ Next update: No further updates are planned.

Update, December 19, 2025 - The initial announcement described this incident as having only lasted for a few hours on December 15, 2025, but further analysis revealed that the incident really began on November 19, 2025 and continued until December 15, 2025. See the post-mortem analysis for fuller details. Around 3:30 PM ET Monday, we detected that log ingestion into the Cloud.gov logging service was delayed by up to an hour, impacting the availability of recent logs. Upon investigation, the team quickly identified that newly deployed log parsing code was contributing to the slowdown. By 4:30 PM ET Monday, we began a targeted rollback of that code, and ingestion throughput began trending back to normal. Over the following hours, ingestion returned to near real-time performance and full service availability. During the event, we also observed an elevated rate of ingestion failures affecting only a subset of log streams, meaning some customers may have experienced partial or no log ingestion loss. We continue reviewing those failure patterns to ensure no residual impact remains. If you have any questions or need assistance, please contact [email protected].

From November 19, 2025, 11:33 AM ET to December 15, 20925, 6 PM ET, [Cloud.gov](http://cloud.gov) Logs experienced elevated ingestion failures for a small subset of application logs. Based on ingestion metrics collected during the incident window, approximately 0.0042% of total log events failed ingestion. The issue was caused by a change to JSON log parsing combined with gaps in monitoring and test coverage. The parsing change was rolled back on December 15, and ingestion rates returned to normal. Log ingestion failures stopped by 6 PM ET. There is no required customer action. We are adding monitoring and alerts for ingestion failures and improving test coverage for JSON log parsing scenarios. Next update: No further updates are planned.

Resolved: Cloudwatch logs (for example, RDS logs) did not ingest into Cloud.gov Logs between December 4, 2025 at 11:14 AM ET and December 17, 1:45 PM ET. The issue was caused by a production change that required AES-256 encryption on the S3 buckets used for log ingestion. This change prevented a Lambda in the ingestion pipeline from successfully writing logs to S3. The Lambda did not surface errors, which delayed detection. At 1:45 PM ET on December 17, the bucket policy was corrected and log ingestion was resumed. No further customer action is required. Impact: Customers may see gaps in Cloudwatch log data for the affected time window. Prevention: We are improving ingestion monitoring, expanding end-to-end smoke tests, and updating our error handling so failures surface immediately. Next update: No further updates are planned.

On November 13, 2025 around 1:40 PM ET, a change was deployed to the log ingestion filters which caused brokered service logs and metrics to be parsed incorrectly. As a result of this failed log parsing, brokered database logs were not visible to customers. On November 19, 2025 around 7:30 PM ET, the Cloud.gov team fixed the bug in the log ingestion filter code, so brokered service logs and metrics are now ingesting correctly and should be visible to customers. To restore the missing logs, the Cloud.gov team plans to re-ingest the brokered service logs and metrics that were parsed incorrectly from November 13 to November 19. To prevent a similar issue from occurring in the future, the Cloud.gov team plans to improve our automated tests to validate that all log types are ingested and parsed successfully after deploying changes.

## Summary From November 13, 2025 around 1:40 PM ET to November 19, 2025 around 7:30 PM ET, brokered service logs and metrics were ingesting, but not visible to customers. ## Timeline * November 5 - A change to the parsing logic for brokered service logs and metrics is made * November 13, ~1:40 PM ET - Change to parsing logic is deployed into production, causing the logs to become invisible to customers * November 19, ~7:30 PM ET - A [Cloud.gov](http://Cloud.gov) engineer notices the ingestion failures and makes a temporary fix for the bug, making brokered service logs and metrics visible to customers again * November 20 - A permanent fix for the bug is prepared and merged ## Impact * Brokered service logs \(currently only available for RDS databases\) and brokered service metrics \(available for RDS, S3, and Elasticsearch/OpenSearch\) were not visible to customers during the incident ## Resolution The bug in the parsing logic was fixed, allowing brokered service logs and metrics to be ingested and restoring visibility to customers. ## Next Steps * Re-ingest the brokered service logs and metrics affected by the incident so they are visible to customers * Improve automated testing for the logs system to ensure that all log types ingest successfully and and are searchable, which will ensure that there is no regression in log visibility from future changes Thank you for your patience. If you have any questions, please contact us at [[email protected]](mailto:[email protected]).

All services are operational, but please expect slower support response times. The Cloud​.gov team will continue to operate all services during the lapse in FY26 appropriations, however all custom work, e.g. for Pages Studio, is on pause. For questions or help with your account or services, reach us at [email protected].

This incident has been resolved.

From approximately 11:22 AM ET to 11:45 AM ET, some customers may have experienced unexpected 403 responses for requests to anything hosted on the cloud.gov platform. The effect was not widespread, as only a small amount of all traffic to cloud.gov overall received 403 responses. The temporary 403 response rate spike was the result of a change to AWS WAF rules which was intended to block malicious traffic, but also had the side effect of blocking some legitimate traffic. As a result, we have disabled the WAF rule which caused the 403 errors and we will not re-enable it. Thanks for your patience. if you have any questions, please contact us at [email protected].

## Summary On August 21, 2025 from 11:22 AM ET to 11:45 AM ET, some [Cloud.gov](http://cloud.gov) customers experienced elevated 403 response rates when accessing hosted services. The issue was quickly identified and resolved, and service has remained stable since. ## Timeline * 11:22 AM - a configuration change was applied to traffic filtering rules. * 11:42 AM - Some customers report elevated 403 response rates * 11:44 AM - The configuration update was rolled back and error rates returned to normal.  * 11:45 AM - Service confirmed stable following log analysis. ## Impact * Some customers experienced elevated 403 errors between 11:22 AM and 11:45 AM. * Not all customers were affected due to the nature of the filtering rules. * For many, the impact was limited or not observed at all. ## Resolution The traffic filtering rule updates were reverted, restoring normal service. Monitoring confirmed that the elevated error rates stopped after the rollback. ## Next Steps * Establish a more formal review process for changes to traffic filtering. * Define when and how customers should be notified of planned changes. * Evaluate technical approaches to reduce the potential impact of similar updates. Thank you for your patience. If you have any questions, please contact us at [[email protected]](mailto:[email protected]).

We are again having issues with our log ingestion rate for https://logs.fr.cloud.gov. Consequently, there may be a delay before customer application logs appear on the system. We are actively investigating the cause of the slow log ingestion and working towards a solution.

After adjusting some data node configuration, log ingestion has recovered and has stayed up to date. As always, the Cloud.gov team takes these incidents very seriously. We will conduct a post-mortem analysis of this incident in the coming days and publish our findings.

**Summary** On August 14, 2025, we had intermittent issues with delayed log ingestion. During these periods of delayed log ingestion, customers may not have seen their logs from the past 1 to 2 hours on [logs.fr.cloud.gov](http://logs.fr.cloud.gov). We believe we have now identified and fixed the cause of delayed log ingestion, so we do not expect the problem to recur. **Timeline** * August 14, 9:13 AM - A smoke test fails for the logs system, [logs.fr.cloud.gov](http://logs.fr.cloud.gov). A [Cloud.gov](http://Cloud.gov) engineer begins investigating the test failure and notices that log ingestion is delayed * 10:30 AM - An engineer scales up the log ingestion infrastructure * 1:52 PM - Log ingestion rates returned to normal and the system was ingesting near real-time logs again * 3:20 PM - A [Cloud.gov](http://Cloud.gov) engineer notices that log ingestion is delayed again * 4:12 PM - An engineer makes an OpenSearch configuration change to control how data ingestion is distributed across nodes * 5:47 PM - Log ingestion rates returned to normal and the system was ingesting near real-time logs again **Impact** Intermittently, customers may not have seen their logs appearing on [logs.fr.cloud.gov](http://logs.fr.cloud.gov) in real time. Logs may have been delayed from appearing for 1 to 2 hours. While there were delays in logs appearing on [logs.fr.cloud.gov](http://logs.fr.cloud.gov), **there was no loss of logs**. **Root Cause** While investigating the second case of delayed log ingestion, we discovered that the log ingestion was being over-allocated to a specific data node, which overwhelmed that node’s CPU resources and caused the delays in log ingestion. To fix this issue, we updated a setting in OpenSearch that controls how many shards can be allocated to a single node for each index, which allowed data ingestion to be distributed more evenly across all of the data nodes. Since making the change to shard allocation, log ingestion rates have remained stable and logs have been ingested in near real-time. **Next Steps** * We are going to improve our internal documentation of how to diagnose and how to troubleshoot delayed log ingestion. * We are going to leave the log ingestion infrastructure at the scaled up levels. Thank you for your patience. If you have any questions, please contact us at [[email protected]](mailto:[email protected]).

We are having issues with our log ingestion rate for https://logs.fr.cloud.gov. Consequently, there may be a delay before customer application logs appear on the system. We are are investigating the cause of the slow log ingestion and working on a solution. In the short term, we are scaling up our log ingestion infrastructure to increase our log ingestion rate.

After scaling up our logging infrastructure, we have seen log ingestion rates return to normal. Going forward, we are going to keep the infrastructure scaling at the new levels. If you are still experiencing issues with your logs not appearing in https://logs.fr.cloud.gov, please contact us at [email protected].

Update, August 27, 2025 - Upon further investigation, the Cloud.gov team found that **there was no gap in database log ingestion**. While doing further querying to assess the impact of the apparent failed database log ingestion, a Cloud.gov engineer discovered that the query that was used to search for database logs was incorrect, which made it appear that database logs were missing and thus had failed to ingest. After fixing the query that is used to search for database logs, the engineer found that the database logs were ingested successfully with **no gap from July 22 to August 26**. Database logs are continuing to ingest successfully now. We apologize for any confusion. From roughly July 22, 2025 1:30 PM ET to August 26, 2025 1 PM ET , the ingestion of brokered database logs to https://logs.fr.cloud.gov was failing. While the database log ingestion seems to have recovered and be working as of August 26, 2025, the Cloud.gov team needs to investigate what caused this incident and how to prevent it from recurring. We will conduct a post-mortem analysis of this incident in the coming days and share our findings.

Upon further investigation, the [Cloud.gov](http://Cloud.gov) team found that **there was no gap in database log ingestion,** and thus no incident. While doing further querying to assess the impact of the apparent failed database log ingestion, a [Cloud.gov](http://Cloud.gov) engineer discovered that the query that was used to search for database logs was incorrect, which made it appear that database logs were missing and thus had failed to ingest. After fixing the query that is used to search for database logs, the engineer found that the database logs were ingested successfully with **no gap from July 22 to August 26**. Database logs are continuing to ingest successfully now. We apologize for any confusion. Please reach out to [[email protected]](mailto:[email protected]) if you have any questions.

Status: Resolved Time (ET): July 22, 2025 - 10:20 AM - 10:46 AM Impact: Some customers routing API traffic through Cloud.gov may have seen intermittent 403 (Forbidden) errors. Root cause: A recent update to our traffic-filtering rules was inadvertently too restrictive, blocking valid API calls. Resolution: At 10:46 AM ET we refined our traffic filtering rules to resolve the issue. All API services have returned to normal operation. Next steps: Post-mortem: We will be conducting a post-mortem of this incident and sharing our findings in the coming days.

**Summary**  A change to our traffic filter rules caused some APIs to return 403 errors. We fixed the rules and restored full service in 26 minutes. **Timeline \(Eastern Time\)** * 10:20 AM: An engineer applies new traffic filter rules. * Change took effect immediately. * Shortly after, customers saw 403 errors on some APIs. * The engineer identified the filter rules as the cause. * The engineer updated the rules to restore access. * 10:46 AM: All API traffic returned to normal. **Impact** * Some API calls received 403 Forbidden errors for 26 minutes. * We have no evidence of data loss or unauthorized access. **Root Cause** 1. We were unable to fully test the filter change before moving it into production. 2. We did not gather enough peer review before release. **Next Steps** * Hold a team review for any filter-rule changes. * Improve and publish clear documentation on traffic filters. * Improve real-time monitoring of 4xx error rates on our load balancers. * Require a final sign-off step before publishing filter rules. Thank you for your patience. If you have any questions, please contact us at [[email protected]](mailto:[email protected]).

While Cloud.gov is currently not experiencing an outage for the platform nor is any of our underlying services partners, some customers might be experiencing issues reaching Cloud.gov. If you are experiencing any issues in accessing, deploying, or using Cloud.gov please contact us via our normal support channel at [email protected].

The impacted internet service and platform providers are now reporting their issues are resolved, so we are closing this issue.

In the process of upgrading our Cloud.gov logging system from version to 2.19 to 3.0, the deployment experienced issues resulting in downtime for https://logs.fr.cloud.gov. We are working to fix the problem and will post an update as soon as possible.

The logging system, https://logs.fr.cloud.gov/, is now up and appears to be responding normally to requests. We will continue to monitor the system closely for any issues.

The Cloud.gov logging system is fully stable and healthy. As with all incidents, we will be conducting a post-mortem of this outage in the coming days. Once our analysis is complete, we will share our findings and our plans to prevent a future recurrence of similar outages. Thank you for your patience and for being a Cloud.gov customer.

The [Cloud.gov](http://Cloud.gov) team has conducted a post-mortem analysis of this incident. A timeline of the incident, our findings of what caused it, and our actions taken to prevent it from recurring are summarized below. **Timeline** * On Monday, May 12, at 8:15 AM, a deployment of our production customer logging system was initiated to upgrade the system from OpenSearch version 2.19 to version 3.0. * At 9:12 AM, [logs.fr.cloud.gov](http://logs.fr.cloud.gov) became inoperable due to a failure in the deployment of the nodes for OpenSearch Dashboards, which provide the visual user interface. * Once the outage was detected, the [Cloud.gov](http://Cloud.gov) team began investigating. We quickly discovered the issue and implemented a temporary fix. * At 9:38 AM, a new deployment was started in production to bring the OpenSearch Dashboards nodes back online. * By 10:10 AM, the new deployment had finished and the OpenSearch Dashboards were restored to fully health and running on version 3.0. At this time, the outage was resolved. **Findings** * The deployment plan for the OpenSearch system was configured to update the nodes for different components \(data nodes, Dashboards\) in serial, one at a time. * In the deployment plan, the data nodes were upgraded before the Dashboards nodes. * In the initial deployment where OpenSearch Dashboard nodes failed to upgrade, the upgrade of the data nodes completed successfully without any issues. * When the Dashboards nodes attempted to upgrade individually, they recognized that the data nodes were on version 3.0, but that at least one of the Dashboards nodes was still on version 2.19, which prevented the deployment from succeeding. This error was observed in the logs for OpenSearch Dashboards: ``` This version of OpenSearch Dashboards (v2.19.0) is incompatible with the following OpenSearch nodes in your cluster: v3.0.0 ``` ‌ * To fix the deployment issues, the team updated the deployment plan to upgrade all OpenSearch Dashboards nodes **at the same time** rather than **serially**. As a result, all Dashboards nodes moved to version 3.0 at the same time, so no error because of incompatibility with the data nodes occurred. **Actions taken** To prevent this incident from recurring, we have taken the following actions: * [Updated the deployment plan for the logging system so that Dashboards nodes are upgraded at the same time, not in serial](https://github.com/cloud-gov/deploy-logs-opensearch/pull/174) * [Documented the necessary order for nodes in the deployment configuration for the logging system](https://github.com/cloud-gov/deploy-logs-opensearch/pull/175) As always, we appreciate your patience as a customer of [Cloud.gov](http://Cloud.gov). If you have any questions about this incident, don’t hesitate to contact us at s[[email protected].](mailto:[email protected])

We have noticed that no logs are appearing in OpenSearch for customer logs (https://logs.fr.cloud.gov) after approximately 10:11 AM ET. We are investigating and will provide an update as soon as we know more.

Update – 12:56 PM ET Status: - Logs are flowing into the OpenSearch cluster again, but indices are still catching up to real time. - Full real-time ingestion is expected to resume within the next few hours. - In the meantime, stream live application logs with: cf logs APP_NAME ---- Technical Details Durable storage & caching: Application logs are first written to S3 for durability, then passed through a cache before landing in OpenSearch. This two‑step process ensures no data loss even if the cluster becomes temporarily unavailable. Root cause: Several OpenSearch data nodes exceeded their disk‑usage high watermark. When this threshold is crossed, OpenSearch marks the affected indices as read‑only and rejects new writes. Mitigation: We increased storage capacity on the affected nodes and rebalanced shard allocation across the cluster. The cluster is now healthy and processing the backlog of cached logs. ---- Next Update: We will continue to monitor cluster health and ingestion rates. Our next status update will be posted by 3:30 PM ET, or sooner if anything changes.

Status: • The OpenSearch cluster has processed the entire backlog and is now ingesting logs in real time without delay. • All indices are writable and healthy, and write throughput remains stable. Resolution Details: • We increased disk capacity on the affected data nodes and rebalanced shard allocation to clear the high‐watermark condition. • The cluster’s health is green and all new log events are successfully indexed. • Live log streaming via cf logs APP_NAME continues to work as expected. Next Steps: • We will keep a heightened watch on disk usage and shard distribution over the next 24 hours to ensure sustained health. • If you notice any further issues with log visibility or performance, please open a support ticket. Thank you for your patience and apologies for any inconvenience.

The Cloud.gov customer security and operations team discovered today that malicious actors are leveraging the tj-actions/changed-files supply chain attack against Cloud.gov customers. See CVE-2025-30066: https://www.cve.org/CVERecord?id=CVE-2025-30066 What you need to do: If you use GitHub Actions for CI/CD and use the action "tj-actions/changed-files", you should consider your service potentially compromised. Search your GitHub account for occurrences of the compromised action with a search like this (substitute your organization name for {MY_GITHUB_ORG}): https://github.com/search?q=org%3A{MY_GITHUB_ORG}+uses%3A+tj-actions%2Fchanged-files%40v+language%3AYAML+path%3A.github%2F&type=code If you are using this action, and have had any deployments between March 14 and March 15, 2025, consider your service compromised. We recommend these initial response steps: - Declare an incident using your internal incident response process - Freeze your GitHub Actions pipelines - Rotate service-account credentials (Cloud.gov will be providing more guidance later today) - Check for any other potential malicious code additions/deployments that may have been added with accounts that had potentially leaked credentials - Notify the Cloud.gov incident team by emailing us your findings to [email protected] We will be releasing updates with indicators of compromise (IOC) and further remediation steps. Notes: - You would be vulnerable even if your GitHub organization or code repository is private - The Cloud.gov platform itself is not impacted - This only applies to Cloud.gov customers using GitHub Actions AND the impacted action. We are conducting an audit of potential attacks, but Cloud.gov customer development teams should conduct their own code audit. If you require assistance on your investigation, please also email [email protected] and request help.

We've updated the original notification with the relevant CVE: https://www.cve.org/CVERecord?id=CVE-2025-30066 Since Cloud.gov has strong tenant isolation, the confirmed compromised customer did NOT impact any other Cloud.gov customers. We are providing this notification and updates in the interests of our customers. INDICATORS OF COMPROMISE At this time it appears the attacker used the compromised GitHub Action to obtain service account credentials (https://cloud.gov/docs/services/cloud-gov-service-account/), used those credentials to login to Cloud.gov, then used the `cf env` command to view sensitive application variables to enable further movement. If you used the compromised Github Action, assess all your applications with the command cf events APP_NAME If you see entries like the following with the actor [email protected], and the event is not part of a deploy, then it's likely a malicious actor has obtained your application's credentials: SAMPLE EVENT time event actor description 2025-03-17T09:08:49.00-0400 audit.app.environment_variables.show [email protected] Regardless of whether there are indicators of compromise or not, if you used the compromise Github Action, you should rotate service account credentials per our documentation: https://cloud.gov/docs/services/cloud-gov-service-account/#rotating-credentials You should also review GitHub Actions logs for the recent executions of the Action and see if it has leaked secrets. See "Review GitHub Actions Workflow Run Logs" at https://www.stepsecurity.io/blog/harden-runner-detection-tj-actions-changed-files-action-is-compromised

We are closing this incident since we have no further updates. It is a customer responsibility to determine if they were impacted by this vulnerability, but please reach out to Cloud.gov support if you need any assistance in doing so.

At 2:13 PM ET, the broker which allows cloud.gov customers to create and to update AWS managed services (RDS, Elasticache/Redis, Elasticsearch/OpenSearch) became unreachable. During this time, requests to create or to update any managed services may have failed with 404 errors. At 2:39 PM ET, the broker was successfully restored and became reachable again. The cause of the outage was misconfiguration in the deployment pipeline for the broker which caused the broker application to be deployed without a public URL. Without a public URL, the broker itself is not reachable from the internet and thus customer requests to it will fail with 404 errors. The misconfiguration had been present in the code since December, but the broker had not been redeployed in production since before then, so the problem only became apparent when attempting to redeploy the broker in production today. The deployment misconfiguration was fixed with this PR: https://github.com/cloud-gov/aws-broker/pull/400. As always, cloud.gov takes any outages seriously and we will work to avoid a recurrence of this incident in the future.