Imgix 中断历史

1. 2026年9月1日 UTC 16:36identified

   我们发现与Google Cloud事件有关, 随着Google Cloud共享更多信息,我们将继续监测出错率并提供进一步更新.

2. 2026年9月1日 UTC 17:05identified

   我们正在继续监测出错率,同时调查这一事件可能发生的故障.

3. 2026年9月1日 UTC 17:15identified

   我们正在推动改变交通方向,并看到整个服务的误差率有所提高。 我们将继续提供最新情况并监测局势.

4. 2026年9月1日 UTC 18:06identified

   我们正在观察整个服务单位的误差率提高的情况。 我们将继续监测局势.

5. 2026年9月1日 UTC 18:31monitoring

   出错率恢复正常 。 我们将继续监测这项服务.

6. 2026年9月1日 UTC 18:48resolved

   这一事件已经得到解决.

7. 2026年9月11日 UTC 19:25postmortem

   □ 总结 在2026年9月1日大约7:49 AM到10:57 AM PDT之间,一个Google Cloud平台区域的网络断层中断了与基础设施的连接,使得Imgix的部分图像和视频渲染管道被授权. 这导致了平均误差率~该窗口所服务请求总数的2%. 在整个事件中,继续正常地提供先前的缓存图像和录像。 ∮什么是错的∮ Google Cloud平台经历了一个网络层面的断层,影响其一个美国地区的计算基础设施。 故障干扰了构成我们提供基础设施的GCP服务之间的连通性,阻碍了该地区部分提供请求的完成。 Google已证实中断是由他们网络维护期间发生的一起事件所造成,并正在继续审查这一事件. 一旦我们收到更多信息,我们将更新本报告。 尽管我们的服务具有多区结构,但整个区域的交通受到影响,导致我们经历的退化。 在整个事件中,误差率在波浪中上升和下降,我们分阶段切断了交通,以降低影响,一旦问题得到解决,交通便恢复正常。 □ 我们将如何防止未来发生 我们正在完善跨区域交通转移的标准和程序,以更好地应对今后发生的事件.

自动翻译自官方事件更新。

1. 2026年9月1日 UTC 15:30identified

   我们发现新的渲染请求 错误增加 因为与上游供应商的持续事件.

2. 2026年9月1日 UTC 15:35identified

   我们发现与Google Cloud事件有关, 随着Google Cloud共享更多信息,我们将继续监测出错率并提供进一步更新.

3. 2026年9月1日 UTC 16:07monitoring

   出错率已恢复正常水平。 我们继续监视局势,同时等待Google Cloud提供最新情况.

4. 2026年9月1日 UTC 16:22resolved

   This incident has been resolved.

自动翻译自官方事件更新。

1. 2026年8月14日 UTC 04:53identified

   We are observing an increase in 5xx errors for new render requests, correlating with an ongoing Google Cloud incident. We will continue to monitor error rates and provide further updates as Google Cloud shares more information.

2. 2026年8月14日 UTC 05:08monitoring

   Error rates have returned to normal levels. We are continuing to monitor the situation while awaiting an update from Google Cloud.

3. 2026年8月14日 UTC 05:28resolved

   While Google Cloud has not yet provided an update, error rates have remained at normal levels. We therefore consider this incident resolved.

1. 2026年6月25日 UTC 10:28monitoring

   我们发现在完成清理请求方面的延误 这一问题正在得到积极监测并正在得到解决。 以前安排的工作将予处理。 图像制作不受影响.

2. 2026年6月25日 UTC 11:27resolved

   这一事件已经得到解决。 以前提交的所有清理请求都已处理完毕.

自动翻译自官方事件更新。

1. 2026年6月4日 UTC 14:44investigating

   We are currently investigating increased 5xx errors for new image and legacy video render requests.

2. 2026年6月4日 UTC 14:52identified

   We have identified a correlation with an ongoing Google Cloud incident, which reports degradation for VM services in multiple locations. We continue monitoring errors rates and updates from Google Cloud.

3. 2026年6月4日 UTC 15:45identified

   We are seeing a large reduction in error rates. We will continue to monitor for updates.

4. 2026年6月4日 UTC 16:00monitoring

   Error rates have subsided to normal levels. We will continue to monitor for updates.

5. 2026年6月4日 UTC 16:54resolved

   This incident has been resolved.

1. 2026年4月16日 UTC 17:14investigating

   We are investigating issues with Imgix administration tools. There are reports of issues with logging into Dashboard and calling Imgix APIs. The rendering service is not affected.

2. 2026年4月16日 UTC 17:27monitoring

   A fix has been implemented and we are monitoring the admin service. The rendering service is completely operational.

3. 2026年4月16日 UTC 17:43resolved

   This incident has been resolved.

1. 2026年2月3日 UTC 13:49identified

   We are investigating uploading errors in the Asset Manager UI for Azure Sources.

2. 2026年2月3日 UTC 16:00monitoring

   A fix has been implemented and we are monitoring the results.

3. 2026年2月3日 UTC 16:21resolved

   This incident has been resolved.

1. 2026年1月26日 UTC 16:15identified

   We are investigating a billing issue with upgrading plan types. If you need to upgrade your plan, please reach out to support as a workaround.

2. 2026年1月27日 UTC 00:58resolved

   This incident has been resolved.

1. 2026年1月12日 UTC 13:53investigating

   We are investigating an issue where Asset Manager returns errors for uploaded assets. Affected assets can still be served through the Rendering API but will not be visible in Asset Manager. Customers using the legacy Long Form Video feature may also encounter video delivery errors. We are investigating this issue.

2. 2026年1月12日 UTC 14:37monitoring

   We have identified the issue and deployed a fix that applies to newly uploaded assets. We are continuing to investigate a solution to restore visibility for assets uploaded prior to the fix in Asset Manager.

3. 2026年1月12日 UTC 14:47resolved

   The uploading incident is resolved. We will follow up with affected customers to restore visibility for previously uploaded assets.

4. 2026年1月28日 UTC 00:27postmortem

   # Summary Between 2026-01-09 21:26 UTC and 2026-01-12 14:18 UTC, some asset uploads returned an upload failure error in Asset Manager despite the uploads completing successfully. Additionally, legacy video customers experienced transcoding delays as a result of the incident. # What Went Wrong A production misconfiguration caused a validation step in the Asset Manager pipeline to fail. As a result: * The legacy video encoding process did not initiate for uploaded videos * Assets did not appear in Asset Manager because they failed validation While affected assets were still available at the origin and accessible via the Rendering and Video APIs, they did not appear in the Asset Manager UI and were not available through the legacy Long-Form Video API. # What we will do to prevent this in the future We have done the following things: 1. Added more alerting and patched logging gaps to catch the issue sooner 2. Fix a flaw in our build system that allowed a two-step deploy to get out of sync

1. 2025年11月20日 UTC 02:04monitoring

   We’re currently experiencing delays in generating CDN and Error Log reports for 2025-11-20. Our team is working on backfilling the data, and we expect the reports to be available by 2025-11-21 (UTC).

2. 2025年11月20日 UTC 04:14monitoring

   The backfill for the CDN and Error Log reports is currently underway and is expected to complete within the next hour.

3. 2025年11月20日 UTC 04:36resolved

   The backfill for the CDN and Error Log reports has been completed. You should now be able to download the reports from the Dashboard or via the API as usual.

1. 2025年11月13日 UTC 05:02resolved

   Purge requests were not executed between 2025-11-12 23:00 UTC and 2025-11-13 05:00 UTC. The issue has been mitigated, and purge functionality has been fully restored. All purge requests that failed during the impact window are being automatically retried.

1. 2025年10月29日 UTC 11:34investigating

   We are currently investigating this issue.

2. 2025年10月29日 UTC 11:56identified

   The issue has been identified and a fix is being implemented.

3. 2025年10月29日 UTC 12:25monitoring

   A fix has been implemented and we are monitoring the results.

4. 2025年10月29日 UTC 12:46resolved

   This incident has been resolved.

5. 2025年11月11日 UTC 00:18postmortem

   # **Summary** Between October 21 and October 29, customers in Europe experienced 3 separate periods of increased latency for rendering requests. In a small number of cases, requests temporarily failed with “429 – concurrency limit reached” responses. # **What Went Wrong** The incident was traced to a GPU scaling issue from one of our upstream infrastructure providers. This led to temporary slowdowns and under higher-than-usual load. ### **Timeline** * **October 21:** Increased rendering latency in EU region, self-resolved. Investigation traced issue to GPU scaling in upstream infrastructure. Mitigation prepared. * **October 27:** Issue recurred. Manual mitigation deployed to stabilize rendering and automate future handling. * **October 29:** Latency alert triggered again. Previous fix mitigates impact, but latency becomes intermittent; additional configuration changes implemented to fully restore service and prevent recurrence. # What we will do to prevent this in the future While the new configurations will prevents recurring incidents, we are making further improvements to rendering resiliency and recovery speed: * Added more GPU hardware types to reduce the risk of scaling delays during peak demand. * Testing and evaluating additional hardware configurations to improve resiliency. * Finalizing fine-tuning of current configurations and exploring cross-regional load-balancing capabilities to further strengthen reliability. * Adjusted alerting thresholds to provide earlier notification of emerging issues.

1. 2025年10月28日 UTC 17:48investigating

   We are investigating issues with upgrading/downgrading account plans through the dashboard. In the meantime, please reach out to our support team if you need to change your plan.

2. 2025年10月28日 UTC 17:58resolved

   This incident has been resolved.

1. 2025年10月27日 UTC 14:07investigating

   We are investigating increased latency for cache misses in the EU region.

2. 2025年10月27日 UTC 14:23identified

   The issue has been identified and a fix is being implemented.

3. 2025年10月27日 UTC 15:25monitoring

   Latency has returned to normal for the affected EU region. We will continue to monitor the situation.

4. 2025年10月27日 UTC 15:56resolved

   This incident has been resolved.

1. 2025年10月21日 UTC 09:26investigating

   We are investigating reports of increased latency for new renders.

2. 2025年10月21日 UTC 09:50monitoring

   Rendering latency has been restored. We are monitoring the service.

3. 2025年10月21日 UTC 10:04resolved

   This incident has been resolved.

1. 2025年10月9日 UTC 01:20resolved

   Between 2025-10-08 23:50 UTC and 2025-10-09 01:50 UTC, a subset of origin fetch requests failed and returned errors. The issue has been identified and resolved.

1. 2025年9月17日 UTC 20:53investigating

   We are currently investigating an issue with the Sources and Analytics tabs in the Dashboard. The rendering service is not impacted.

2. 2025年9月17日 UTC 21:07resolved

   This incident has been resolved.

1. 2025年6月12日 UTC 18:09investigating

   We are investigating elevated render error rates for the service. Previously cached derivatives are not impacted.

2. 2025年6月12日 UTC 18:15identified

   The issue has been identified and we are investigating a solution.

3. 2025年6月12日 UTC 18:53identified

   The service is experiencing elevated error rates due to a major Google Cloud outage affecting services downstream. Previously cached derivatives are not impacted. We are investigating ways to mitigate this issue.

4. 2025年6月12日 UTC 20:26monitoring

   The service is restored. We are monitoring the situation.

5. 2025年6月12日 UTC 20:38identified

   The Rendering API has fully recovered. We are continuing to investigate Web Administration related issues (login and Management API) related to the incident.

6. 2025年6月12日 UTC 20:47monitoring

   The Rendering API is fully recovered. Web Administration tooling (logins and the Management API) are recovering. We are monitoring the results.

7. 2025年6月12日 UTC 20:52resolved

   The service is completely restored.

8. 2025年6月20日 UTC 19:03postmortem

   # Incident Summary Between **17:55 and 20:22 UTC** on **June 12, 2025**, Imgix services experienced major service disruptions across several key interfaces: * **Dashboard and Asset Manager**: These interfaces were inaccessible, preventing users from managing their assets or viewing account information. * **Management API**: Requests to the Management API consistently returned errors, affecting workflows reliant on programmatic updates or asset administration. * **Rendering API:** Approximately 8% of all Imgix service requests failed due to a high error rate \(~80%\) for **uncached** assets via the Rendering API. Requests in the EU saw a lower failure rate \(~50%\) and a faster recovery time \(45 minutes\) for these uncached requests. # What caused it The incident was triggered by a global outage within Google Cloud, which serves as a core infrastructure provider for Imgix. The outage affected most services in all regions simultaneously. You can read more about the [Google Cloud outage here](https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1SsW#RP1d9aZLNFZEJmTBk8e1). # What happened * **17:55 UTC:** Internal alerts triggered due to a spike in rendering errors and service timeouts. * **18:01 UTC:** A short investigation uncovers several timeouts and increased error rates from Google Cloud. * **18:09 UTC:** Our status page is updated. * **18:53 UTC:** A major Google Cloud outage is confirmed, after which we update our status page. * **18:00–20:16 UTC:** Mitigation efforts were hampered by the far-reaching effects of the outage, preventing us from redirecting traffic or applying configuration changes. * **Throughout:** We confirm that cached images were not affected, though the downtime of several data sources prevents evaluating the full scope and effect of the outage. * **20:16 UTC:** Google reported recovery in all regions except `us-central1`. This allowed us to verify **significantly lower error rates for EU traffic** * **20:47 UTC:** Imgix systems achieved full recovery. * **20:52 UTC:** The incident was officially resolved on our status page. * **21:23 UTC:** Google confirms a full-service recovery at 21:23 UTC. # What went wrong * Google Cloud experienced an outage that simultaneously affected nearly every service in every region worldwide, negating our multi-region redundancy for the image rendering service. * 3rd party services \(such as our CDN\) were also affected by the outage, which removed some of our options for redirecting traffic across regions based on performance. * The outage included the control panes that Google provides to its customers, which removed additional options for redirecting traffic and implementing mitigations. # What we will do to prevent this in the future * Continuing our ongoing internal discussions and evaluations of a multi-cloud render stack to enable failover in the event of a provider-wide outages. * Continue evaluating and improving tools to automatically and manually shift traffic as necessary at each layer of the stack. * Review and enhance incident communication protocols, focusing on faster root cause disclosure and update frequency.

1. 2025年5月20日 UTC 15:40investigating

   We are currently investigating elevated render error rates for uncached derivative images and the Management API in the NA region. We will update once when we obtain more information. Previously cached derivatives are not impacted.

2. 2025年5月20日 UTC 15:57identified

   The issue has been identified and a fix is being implemented.

3. 2025年5月20日 UTC 16:01monitoring

   A fix has been implemented and we are monitoring the results.

4. 2025年5月20日 UTC 16:10resolved

   This incident has been resolved.

1. 2025年4月24日 UTC 12:29identified

   Some customers are experiencing issues with logging into the dashboard. We have identified the issue and are working on a fix.

2. 2025年4月24日 UTC 14:10monitoring

   A fix has been implemented and we are monitoring the results.

3. 2025年4月24日 UTC 15:31resolved

   This incident has been resolved.