面对管道的问题
Outage重大事件
- investigating
我们看到一些问题 部署新的图件 建造。 我们正在调查根源
- monitoring
我们发现了一个问题 与我们的云配置。 我们纠正了,现在又恢复了
- resolved
与图文文物管线的问题已经解决. 在这次事件中,没有丢失任何数据,只是在我们解决问题时拖延。 我们为这次破坏道歉
自动翻译自官方事件更新。
33 Apollo incidents · 2025年6月 — official updates, affected components, duration and resolution details.
我们看到一些问题 部署新的图件 建造。 我们正在调查根源
我们发现了一个问题 与我们的云配置。 我们纠正了,现在又恢复了
与图文文物管线的问题已经解决. 在这次事件中,没有丢失任何数据,只是在我们解决问题时拖延。 我们为这次破坏道歉
自动翻译自官方事件更新。
由于Google Cloud平台(GCP)持续停用,目前我们正经历多个阿波罗服务中断的服务. 我们正在积极调查,并将在获得更多信息后分享最新情况.
AWS Uplink 端点仍然可用. 使用默认多云配置的 GraphOS 路由器和网关可以在此断时继续启动并获取其配置,但无法发布新的策略.
由于Google Cloud平台(GCP)持续停用,目前我们正经历多个阿波罗服务中断的服务. 我们正在积极调查,并将在获得更多信息后分享最新情况.
我们的第三方供应商已经证实,这一问题被隔离到我们中部1-b区的一部分;该地区剩余的地区没有受到影响。 大部分阿波罗产品都恢复了 我们的工程组正在验证剩下的受影响服务 我们将继续密切监视,然后才能解决这件事.
我们的第三方供应商已经确认这个问题现在已经解决了,所有阿波罗产品都已经恢复.
自动翻译自官方事件更新。
一些组织和用户修改(用户邀请,用户权限更改,OAuth更改)间歇性失败. 我们正在努力解决问题.
我们不再看到任何请求失败,但我们将继续进行监测,直到找到根源.
修补已出. 这些失败的操作本会作为费率限制错误出现,但费率限制被错误地应用. 我们已作出修改,以确保费率限制工程如期进行.
自动翻译自官方事件更新。
Registry service workflow latency has spiked, affecting mulitple internal services serving long-running workflows.
The issue has been resolved and we're continuing to monitor.
The issue has been resolved.
Embedded Explorer and Embedded Sandbox are temporarily down. The issue has been identified and a fix is being deployed.
The issue has been fixed.
We’re currently investigating a rise in error rates for our Proposals service.
We are still investigating elevated error rates for Proposals. We appreciate your patience.
We’ve identified the issue and are working on resolution.
We have applied a fix and have begun to see recovery in our metrics. We will continue to monitor and update the status page once it’s resolved.
The error rate for Proposals has returned to normal, no further impact expected.
We are investigating suspected data loss for field and coordinate usage metrics for all customers.
Field and coordinate usage metric ingestion is functional and lag is trending downwards. We estimate 8-10 hours for full recovery.
The backlog of object field, input object field, and enum value metrics are now being processed but we expect the backlog to take 8-10 hours to be fully processed. For a period of time between 4 am and 9pm on June 27 UTC we expect there to be data loss for input object field and enum value metrics, as well as the client and operation breakdown of object field metrics. No other metrics (operation, trace, etc.) are impacted.
The backlock of schema coordinate (object field, input object field, and enum value) metrics has now cleared and there is no more metric ingestion delay. The metrics for input object fields, enum values, and the operation and client breakdown of object fields is inaccurate for June 27 4-5am and 8-9pm, and missing between 5am to 8pm (all times in UTC). All other metrics were not impacted.
We are investigating a case where metric ingestion is delayed for some customers. There is no loss of data.
We have identified the cause though customers continue to experience metric ingestion delays while we identify the appropriate mitigation. Data backfill after remediation is still expected after mitigation is applied. We appreciate your patience.
We are incrementally applying mitigations which are showing positive results. We feel we are confidently iterating toward recovery. At this time, no data is believed to be lost and metric backfill will continue with the oldest metrics being populated first, though we currently have a substantial backlog which we will scale to process quickly. We will update this status page as things continue and we appreciate your patience.
We are confidently seeing recovery. Some metric delay will continue for the next couple hours, though many metric types have already recovered. We will keep this status page open in Monitoring state and update it when we have eliminated our metric lag. Thank you again for your patience.
The metric ingestion backlog has been fully processed and all metrics have resumed normal ingestion. We do not believe any data was lost as a result of this incident. All metric types should now be operating as expected. Thank you for your patience throughout this incident.
Some Router deployments are experiencing issues coming online due to a message around licensing. We are investigating.
We are continuing to investigate.
We have seen recovery for about 50 minutes now and we are continuing to monitor the recovery. If you find that you are impacted by a licensing message on Router startup, please open a ticket with our support team.
After extended monitoring, we have confirmed that the issue is resolved. Post-incident action items will continue after the end of the incident. As with all incidents, we will be conducting an internal post-mortem. We appreciate your patience.
We’re currently investigating an issue where the “Submit a Request” and “My Requests” buttons are not appearing properly in the Apollo Support Portal. As a temporary workaround, tickets can still be created and accessed through the “Submit a Request” button available in the header of the portal.
We’re continuing to investigate the issue affecting the visibility of the “Submit a Request” and “My Requests” buttons in the Apollo Support Portal. At this time, we are working with our external provider to identify the root cause and restore normal functionality.
We’ve implemented changes to address the issue affecting the visibility of the “Submit a Request” and “My Requests” buttons in the Apollo Support Portal. At this time, the portal appears to be functioning normally, and we are actively monitoring the situation to ensure stability. If you continue experiencing issues, please try refreshing the page or signing out and back into the portal. We’ll continue to monitor closely and provide additional updates if needed.
The issue affecting the visibility of the “Submit a Request” and “My Requests” buttons in the Apollo Support Portal has been resolved. Portal functionality has been restored, and users should now be able to access these options normally. Thank you for your patience while we worked to resolve this issue.
Opening a ticket using the Contact Support button on the [studio.apollographql.com](http://studio.apollographql.com "studio.apollographql.com") site itself is currently failing. Support tickets can still be created through our [support.apollographql.com](http://support.apollographql.com "support.apollographql.com") portal, or via our `support@` email address on the `apollographql.com` domain.
Support ticket opening functionality has been restored to Studio’s “Contact Support” functionality in the UI. We have mitigated the failures and will monitor before marking as resolved.
We have observed sustained restored functionality in our Studio-initiated support ticket form and are marking this as fully resolved.
We're seeing issues with StudioUI and are investigating to find the root cause
We're continuing to investigate to find the root cause and resolve the issue
We have identified a possible root cause, and we are testing changes to confirm
We confirmed with internal testing that our fix is working as expected, and customers should be seeing recovery. We're continuing to monitor to confirm.
We have resolved the issue. Customers should see full functionality.
We’re investigating the availability of our platform after a recent deploy.
We have restored functionality to impacted users SSO access and are monitoring.
SSO login is now fixed.
We are aware of an issue affecting GraphOS Studio where users may encounter errors when attempting to view the Changes or Schema Reference tabs within graph proposal. We are investigating and will give an update soon.
We have identified the root cause of the failures affecting the Changes and References tabs in Schema Proposals. A fix has been deployed and we continue to monitor and investigate.
Proposals errors have been clear for ~2 hours since the rollback reached prod. Marking resolved.
We’re monitoring an interruption in the transmission of transactional emails (e.g., email verification, welcome emails, proposals approvals). We are investigating.
We’ve identified the root cause of the delays, applied mitigations and recent delivery attempts are succeeding. Some previously attempted deliveries may take time to be retried.
Retries of delayed transaction emails have been successful. If you have not received an email you were expecting for account reasons, we suggest retrying the workflow which triggered the email.
We've identified an issue where our metrics ingestion pipeline was falling behind for several hours. Systems have now fully recovered and no data was lost, however some customers may have experienced delays viewing insights data in Studio and / or slower page load times during this event. Apologies for the interruption.
Platform API is also seeing degraded performance, which we are investigating
We're seeing increases in latency across a few different Apollo services. You may be seeing latency issues with proposals, checks, and studio.
We're still investigating the root cause. We're seeing some recovery, but not to full recovery
We're continuing to investigate the root cause of the issue. Some customers are reporting significant improvements in performance.
We've implemented a fix and are monitoring the system.
The issue has been resolved.
There was a period of degraded performance on schema publishing. The root cause has been identified and service restored.
We're experiencing elevates error rates due to an upstream DNS provider major incident.
We're continuing the monitor the affected services while the upstream provider remediates the issue.
Our DNS provider has provided an update: "The issue has been identified and a fix is being implemented." We will be monitoring for recovery of our systems.
At this time, our monitors indicate that only Cloud Routers hosted on Apollo infrastructure are impacted as a result of an upstream service provider incident. Studio UI has already recovered. Beyond these two impacted services, there has been no impact to APIs (including Uplink).
The upstream provider has indicated recovery on their status page (https://www.cloudflarestatus.com/incidents/8gmgl950y3h7). At the same time, we have seen our own monitors return to non-triggered states. We will continue to monitor for the next 15 minutes and then declare this Resolved. While Cloud Routers were impacted throughout the incident, the Studio UI impact was during two distinct periods: Nov 18 11:52 am – Nov 18, 12:10 pm UTC (18 minutes) and again from Nov 18, 12:51 pm – Nov 18, 1:09 pm UTC (another 18 minutes). Studio UI was otherwise available in times outside of that range.
We are marking this incident resolved on the Apollo side. For any continuing matters, please consult our support portal: https://support.apollographql.com/
Cloud Routers are experiencing elevated errors due to an upstream external provider incident.