50 Apigee incidents · Ebrel 2021 — official updates, affected components, duration and resolution details.
Some of Apigee Edge Public Cloud customers in europe-west2 experiencing elevated number of HTTP 500 responses and increased latencies due to infrastructure issues in Apigee Datastore cluster.
Issue has been mitigated
This incident has been resolved.
We are currently investigating issues affecting Apigee Edge customers in eu-west-1. Some customers may observe elevated number of HTTP 500 responses due to issues with Apigee Datastore in the region.
Issue has been mitigated, normal traffic flow resumed. Team suspects partial network outage in single region, we continue investigating.
This incident has been resolved.
A remote code execution vulnerability, CVE-2024-6387, was recently discovered in OpenSSH. Please refer to Google Cloud Security bulletin published: GCP-2024-040 : https://cloud.google.com/apigee/docs/security-bulletins/security-bulletins#gcp-2024-040
This incident has been resolved.
Team suspects incorrect configuration leading to some percentage of customer traffic affected. We are actively investigating this issue.
Team has narrowed down potential cause and performing a mitigation attempt
Mitigation was successful, we're now monitoring the incident and confirming impact.
This incident has been resolved.
Some customers hosted on GCP may experience latency and/or 5XX errors in their runtimes due to a Networking Issue on GCP. Some Edge Cloud orgs may experience errors during API proxy deployment as well. The impact to most organizations were resolved automatically and the Apigee team is investigating the issue further and taking appropriate steps to mitigate it further. Apigee service will recover as soon as this Networking issue is resolved. The underlying issue status can be tracked at https://status.cloud.google.com/incidents/xVSEV3kVaJBmS7SZbnre#2c2sBHWU84yPDJ8y1ar4
Most regions' network functionality has recovered. We are still actively monitoring for any remaining issues.
This issue has been resolved for all affected projects as of Thursday, 2024-05-16 18:10 US/Pacific. We thank you for your patience while we worked on resolving the issue.
We are currently investigation an issue causing Developer Portals to not load. We'll provide updates as they become available. Thank you for your patience!
The issue was identified and mitigating actions taken to restore services as of ~1:15pm PT. Our technical teams are continuing to monitor the service to ensure no further service disruptions while working to understand the root cause.
The incident was restored at 1:18 PM PDT and services continue to be stable. We apologize for any inconvenience this caused you. A full RCA is in flight to identify and track actions to prevent a future occurrence.
Some of the customers whose Edge Cloud organizations in eu-west-1 region might be experiencing API traffic failures due to unknown issue with router nodes. The incident is under investigation now.
Unhealthy router nodes has been replaced, we continue to investigate this incident.
Customer traffic is no longer impacted. We continue to monitor the issue.
This incident has been resolved.
Some of the customers whose Edge Cloud organizations in europe-west1 region might be experiencing API traffic failures due to Datastore issues (Cassandra). The issue is under investigation now.
We are continuing to investigate this issue.
The incident was mitigated several hours after this initial posting but left in monitoring status for an extended time period given similar issues in this region. No further outages were recorded post mitigation. A full RCA is in flight to track actions to prevent a recurrence of this issue. We apologize for the inconvenience this caused.
Some of the customers whose Edge Cloud organizations in europe-west1 region might be experiencing API traffic failures due to Datastore issues (Cassandra). The issue is under investigation now.
We are continuing to investigate this issue.
The incident has been mitigated, and has been stable for the last 6 hours. While we investigate the root cause, we'll continue to monitor this issue closely to ensure service stability. Thank you for your patience and understanding.
The emergency maintenance to perform infrastructure maintenance was completed successfully. The technical teams will continue to monitor this cluster closely to ensure the service continues to remain stable including investigating the root cause further for opportunities to prevent recurrence. We apologize for any inconvenience this has caused for your business, but we appreciate your partnership and patience through this.
Some of the customers whose Edge Cloud organizations in europe-west1 region might be experiencing API traffic failures due to Datastore issues (Cassandra). The issue is under investigation now.
We are continuing to investigate this issue.
The issue is mitigated and it is being monitored.
Edge UI is experiencing intermittent issues for users accessing it from Europe. Proxy develop tab is not loading for some users. We are actively investigating this issue.
The issue has been identified and a mitigation is being implemented. Next update will be provided post that implementation.
This issue has been fully mitigated and continues to remain stable. We will continue to monitor this issue but do not expect any further service impact. Thanks you for your patience and partnership through this.
We are currently investigating issues accessing the Apigee Integrated Developer Portal from Apigee Edge and Apigee X. Users may experience errors like "Failed to load users", "Failed to load zone" or "Service Unavailable when interacting with the developer portal. We will post updates are they become available.
The issue with Apigee Integrated Developer Portal has been resolved for all affected users as of Monday, 2023-09-11 12:25 US/Pacific. We thank you for your patience while we worked on resolving the issue.
We're currently investigating an issue for some customers encountering intermittent 5xx errors for some of their API runtime calls in the US-West region. All of the right resources are engaged in this investigation. No other services are impacted by this issue. We'll provide an update shortly. Thank you for your patience!
The issue was identified to a single node issue in one of our Cassandra pods in the US-West region which was mitigated. We're not seeing any further errors post mitigation but will continue to monitor. We'll do a review of this incident and take the necessary actions to improve our systems to prevent a recurrence. Thanks again for your patience and partnership!
We are currently investigating this issue.
We are experiencing an issue with Apigee beginning on Thursday, 2023-06-22 03:18 US/Pacific. Our engineering team continues to investigate the issue. We apologize to all who are affected by the disruption. Customers are not able to perform any deployment activities while using Apigee Edge Cloud
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
Starting June 14th 10 AM PDT, the Apigee customers using Azure endpoints in their Apigee Hosted Targets may be experiencing timeouts. The issue is currently investigated by GCP teams. We will update as we progress.
The issue has been mitigated as of June 16th 9:30 AM PT
This incident has been resolved.
We have identified an issue with Cassandra for a subset of organizations in eu-west-1 causing datastore errors, and may impact runtime traffic.
Apigee SRE team has mitigated the cassandra datastore errors and affected customers should be seeing their API runtime traffic error rates return to normal
No further issues encountered since the last status page update. Marking this incident as resolved. Post incident analysis is in flight to determine preventative actions and other improvement opportunities to reduce the risk of recurrence. Thank you for your patience and partnership through this.
We are experiencing issues with fetching data for API monitoring in Europe region. The users of Edge Cloud will experience intermittent errors when loading api monitoring via the UI or APIs. There is no runtime traffic impact.
A fix has been implemented. We are monitoring the results now.
This incident has been resolved.
We have identified an issue with DNS failures on Apigee Edge in US-West that may result in failed runtime calls. Teams are actively working on remediation.
We are continuing our work to identify the root cause of the issue implement a mitigation. In the meantime, we have failed over customer who have multi-regional setups that we have identified as impacted to their alternate regions.
The team has rolled out a mitigation, and we are monitoring recovery in us-west. Once we have confidence the region is stable and the mitigation is completed (expected within the next 2 hours), we will begin re-enabling us-west for customers who have been failed out of that region.
This issue has been resolved following the mitigation rollout. All DNS resolutions are stable in the us-west region, and runtime issues are confirmed to be resolved.
Some EdgeCloud customers would have encountered "datastore" related error responses yesterday for API runtime related requests in the US-Central1 region. All other products and services were unaffected. We were able to identify and mitigate the impact, and have monitored service stability since the incident. If you are experiencing continued impact related to this issue, please file a case with Apigee Support for further investigation. We apologize for the impact this may have had on your business, and will take the necessary actions to prevent similar issues in the future. Impact start time: 2023-04-12 17:03 PDT Impact end time: 2023-04-12 17:48 PDT
We have an upcoming release window scheduled for Mon March 13, 2023. This release is not expected to cause any interruption to Apigee Edge UI. Once the release is complete, release notes will be published at: http://apigee.com/docs/release-notes/content/apigee-edge-release-notes
No interruptions due to this issue. Release task is complete and the release notes can be viewed at http://apigee.com/docs/release-notes/content/apigee-edge-release-notes
We are continuing to investigate this issue & will post updates here.
The incident has been resolved and the release notes can be viewed at http://apigee.com/docs/release-notes/content/apigee-edge-release-notes
