Imgix 장애 기록

1. 2026년 9월 1일 오후 4:36 UTCidentified

   우리는 지속적인 Google Cloud 사건과 상관 관계를 확인했습니다. 이는 패킷 손실과 여러 서비스에 대한 오류를보고했습니다. 우리는 오류율을 계속 모니터링하고 Google Cloud로 더 많은 정보를 제공합니다.

2. 2026년 9월 1일 오후 5:05 UTCidentified

   우리는이 사건에 대한 잠재적 인 실패 옵션에 투자하는 동안 오류율을 모니터링하고 있습니다.

3. 2026년 9월 1일 오후 5:15 UTCidentified

   우리는 트래픽을 리디렉션하고 서비스 전반에 걸쳐 개선 된 오류율을보고 변경합니다. 우리는 업데이트를 계속하고 상황을 모니터링 할 것입니다.

4. 2026년 9월 1일 오후 6:06 UTCidentified

   서비스 전반에 걸쳐 향상된 오류율을 관찰하고 있습니다. 우리는 계속 상황을 모니터링 할 것입니다.

5. 2026년 9월 1일 오후 6:31 UTCmonitoring

   오류율은 정상으로 돌아갑니다. 우리는 계속 서비스를 감시할 것입니다.

6. 2026년 9월 1일 오후 6:48 UTCresolved

   이 사건은 해결되었습니다.

7. 2026년 9월 11일 오후 7:25 UTCpostmortem

   ## 요약 약 7:49 AM 및 10:57 AM PDT 9 월 1, 2026, Google Cloud Platform 지역의 네트워크 오류는 Imgix의 이미지 및 비디오 렌더링 파이프의 일부를 전원 인프라에 연결되었습니다. 이 결과 평균 오류율 ~2% 총 요청은 해당 창에서 제공. 너무 이른 이미지와 동영상은 일반적으로 사건에 걸쳐 제공 계속. ## 무엇이 잘못된 Google Cloud Platform은 미국 지역의 인프라에 영향을 미치는 네트워크 수준의 결함을 경험했습니다. 렌더링 인프라를 구성하는 GCP 서비스 간의 잘못된 연결은 완료에서 해당 지역의 렌더링 요청의 공유를 방지합니다. Google은 네트워크 유지 보수 기간 동안 발생 한 사건에 의해 발생했으며 사건을 검토하기 위해 계속되었습니다. 우리는 우리가 더 많은 정보를 받을 한 번이 보고서를 업데이트 할 수 있습니다. 우리의 서비스를 위한 다zonal 윤곽이 있기 때문에, 교통은 우리가 경험한 degradation에서 유래하는 전체적인 지역에 충격을 받았습니다. 오류율이 상승하고 사건을 통해 파도에 떨어졌다, 우리는 영향을 줄이기 위해 건강 지역에 트래픽의 위상을 수행하고, 고정 된 트래픽을 정상에 반환 한 번 문제. ## 우리는 미래에 이것을 방지하기 위해 무엇을 할 것인가 우리는 미래 사건에 대한 우리의 응답을 개선하기 위해 교차 방향 교통 교대에 대한 우리의 기준과 절차를 정제하고 있습니다.

공식 인시던트 업데이트를 자동 번역했습니다.

1. 2026년 9월 1일 오후 3:30 UTCidentified

   우리는 업스트림 공급자와 지속적인 사고로 인해 새로운 렌더링 요청에 대한 오류의 증가를 확인했습니다.

2. 2026년 9월 1일 오후 3:35 UTCidentified

   우리는 지속적인 Google Cloud 사건과 상관 관계를 확인했습니다. 이는 패킷 손실과 여러 서비스에 대한 오류를보고했습니다. 우리는 오류율을 계속 모니터링하고 Google Cloud로 더 많은 정보를 제공합니다.

3. 2026년 9월 1일 오후 4:07 UTCmonitoring

   오류율은 정상 수준으로 돌아 왔습니다. Google Cloud에서 업데이트를 기다리고있는 동안 상황을 모니터링하고 있습니다.

4. 2026년 9월 1일 오후 4:22 UTCresolved

   This incident has been resolved.

공식 인시던트 업데이트를 자동 번역했습니다.

1. 2026년 8월 14일 오전 4:53 UTCidentified

   We are observing an increase in 5xx errors for new render requests, correlating with an ongoing Google Cloud incident. We will continue to monitor error rates and provide further updates as Google Cloud shares more information.

2. 2026년 8월 14일 오전 5:08 UTCmonitoring

   Error rates have returned to normal levels. We are continuing to monitor the situation while awaiting an update from Google Cloud.

3. 2026년 8월 14일 오전 5:28 UTCresolved

   While Google Cloud has not yet provided an update, error rates have remained at normal levels. We therefore consider this incident resolved.

1. 2026년 6월 25일 오전 10:28 UTCmonitoring

   우리는 Purge 요청을 완료하는 지연을 감지했습니다. 문제는 적극적으로 모니터링되고 현재 해결됩니다. 이전 예약 작업이 처리됩니다. 이미지 렌더링은 영향을받지 않습니다.

2. 2026년 6월 25일 오전 11:27 UTCresolved

   이 사건은 해결되었습니다. 이전에 제출 한 모든 서류는 처리되었습니다.

공식 인시던트 업데이트를 자동 번역했습니다.

1. 2026년 6월 4일 오후 2:44 UTCinvestigating

   We are currently investigating increased 5xx errors for new image and legacy video render requests.

2. 2026년 6월 4일 오후 2:52 UTCidentified

   We have identified a correlation with an ongoing Google Cloud incident, which reports degradation for VM services in multiple locations. We continue monitoring errors rates and updates from Google Cloud.

3. 2026년 6월 4일 오후 3:45 UTCidentified

   We are seeing a large reduction in error rates. We will continue to monitor for updates.

4. 2026년 6월 4일 오후 4:00 UTCmonitoring

   Error rates have subsided to normal levels. We will continue to monitor for updates.

5. 2026년 6월 4일 오후 4:54 UTCresolved

   This incident has been resolved.

1. 2026년 4월 16일 오후 5:14 UTCinvestigating

   We are investigating issues with Imgix administration tools. There are reports of issues with logging into Dashboard and calling Imgix APIs. The rendering service is not affected.

2. 2026년 4월 16일 오후 5:27 UTCmonitoring

   A fix has been implemented and we are monitoring the admin service. The rendering service is completely operational.

3. 2026년 4월 16일 오후 5:43 UTCresolved

   This incident has been resolved.

1. 2026년 2월 3일 오후 1:49 UTCidentified

   We are investigating uploading errors in the Asset Manager UI for Azure Sources.

2. 2026년 2월 3일 오후 4:00 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

3. 2026년 2월 3일 오후 4:21 UTCresolved

   This incident has been resolved.

1. 2026년 1월 26일 오후 4:15 UTCidentified

   We are investigating a billing issue with upgrading plan types. If you need to upgrade your plan, please reach out to support as a workaround.

2. 2026년 1월 27일 오전 12:58 UTCresolved

   This incident has been resolved.

1. 2026년 1월 12일 오후 1:53 UTCinvestigating

   We are investigating an issue where Asset Manager returns errors for uploaded assets. Affected assets can still be served through the Rendering API but will not be visible in Asset Manager. Customers using the legacy Long Form Video feature may also encounter video delivery errors. We are investigating this issue.

2. 2026년 1월 12일 오후 2:37 UTCmonitoring

   We have identified the issue and deployed a fix that applies to newly uploaded assets. We are continuing to investigate a solution to restore visibility for assets uploaded prior to the fix in Asset Manager.

3. 2026년 1월 12일 오후 2:47 UTCresolved

   The uploading incident is resolved. We will follow up with affected customers to restore visibility for previously uploaded assets.

4. 2026년 1월 28일 오전 12:27 UTCpostmortem

   # Summary Between 2026-01-09 21:26 UTC and 2026-01-12 14:18 UTC, some asset uploads returned an upload failure error in Asset Manager despite the uploads completing successfully. Additionally, legacy video customers experienced transcoding delays as a result of the incident. # What Went Wrong A production misconfiguration caused a validation step in the Asset Manager pipeline to fail. As a result: * The legacy video encoding process did not initiate for uploaded videos * Assets did not appear in Asset Manager because they failed validation While affected assets were still available at the origin and accessible via the Rendering and Video APIs, they did not appear in the Asset Manager UI and were not available through the legacy Long-Form Video API. # What we will do to prevent this in the future We have done the following things: 1. Added more alerting and patched logging gaps to catch the issue sooner 2. Fix a flaw in our build system that allowed a two-step deploy to get out of sync

1. 2025년 11월 20일 오전 2:04 UTCmonitoring

   We’re currently experiencing delays in generating CDN and Error Log reports for 2025-11-20. Our team is working on backfilling the data, and we expect the reports to be available by 2025-11-21 (UTC).

2. 2025년 11월 20일 오전 4:14 UTCmonitoring

   The backfill for the CDN and Error Log reports is currently underway and is expected to complete within the next hour.

3. 2025년 11월 20일 오전 4:36 UTCresolved

   The backfill for the CDN and Error Log reports has been completed. You should now be able to download the reports from the Dashboard or via the API as usual.

1. 2025년 11월 13일 오전 5:02 UTCresolved

   Purge requests were not executed between 2025-11-12 23:00 UTC and 2025-11-13 05:00 UTC. The issue has been mitigated, and purge functionality has been fully restored. All purge requests that failed during the impact window are being automatically retried.

1. 2025년 10월 29일 오전 11:34 UTCinvestigating

   We are currently investigating this issue.

2. 2025년 10월 29일 오전 11:56 UTCidentified

   The issue has been identified and a fix is being implemented.

3. 2025년 10월 29일 오후 12:25 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

4. 2025년 10월 29일 오후 12:46 UTCresolved

   This incident has been resolved.

5. 2025년 11월 11일 오전 12:18 UTCpostmortem

   # **Summary** Between October 21 and October 29, customers in Europe experienced 3 separate periods of increased latency for rendering requests. In a small number of cases, requests temporarily failed with “429 – concurrency limit reached” responses. # **What Went Wrong** The incident was traced to a GPU scaling issue from one of our upstream infrastructure providers. This led to temporary slowdowns and under higher-than-usual load. ### **Timeline** * **October 21:** Increased rendering latency in EU region, self-resolved. Investigation traced issue to GPU scaling in upstream infrastructure. Mitigation prepared. * **October 27:** Issue recurred. Manual mitigation deployed to stabilize rendering and automate future handling. * **October 29:** Latency alert triggered again. Previous fix mitigates impact, but latency becomes intermittent; additional configuration changes implemented to fully restore service and prevent recurrence. # What we will do to prevent this in the future While the new configurations will prevents recurring incidents, we are making further improvements to rendering resiliency and recovery speed: * Added more GPU hardware types to reduce the risk of scaling delays during peak demand. * Testing and evaluating additional hardware configurations to improve resiliency. * Finalizing fine-tuning of current configurations and exploring cross-regional load-balancing capabilities to further strengthen reliability. * Adjusted alerting thresholds to provide earlier notification of emerging issues.

1. 2025년 10월 28일 오후 5:48 UTCinvestigating

   We are investigating issues with upgrading/downgrading account plans through the dashboard. In the meantime, please reach out to our support team if you need to change your plan.

2. 2025년 10월 28일 오후 5:58 UTCresolved

   This incident has been resolved.

1. 2025년 10월 27일 오후 2:07 UTCinvestigating

   We are investigating increased latency for cache misses in the EU region.

2. 2025년 10월 27일 오후 2:23 UTCidentified

   The issue has been identified and a fix is being implemented.

3. 2025년 10월 27일 오후 3:25 UTCmonitoring

   Latency has returned to normal for the affected EU region. We will continue to monitor the situation.

4. 2025년 10월 27일 오후 3:56 UTCresolved

   This incident has been resolved.

1. 2025년 10월 21일 오전 9:26 UTCinvestigating

   We are investigating reports of increased latency for new renders.

2. 2025년 10월 21일 오전 9:50 UTCmonitoring

   Rendering latency has been restored. We are monitoring the service.

3. 2025년 10월 21일 오전 10:04 UTCresolved

   This incident has been resolved.

1. 2025년 10월 9일 오전 1:20 UTCresolved

   Between 2025-10-08 23:50 UTC and 2025-10-09 01:50 UTC, a subset of origin fetch requests failed and returned errors. The issue has been identified and resolved.

1. 2025년 9월 17일 오후 8:53 UTCinvestigating

   We are currently investigating an issue with the Sources and Analytics tabs in the Dashboard. The rendering service is not impacted.

2. 2025년 9월 17일 오후 9:07 UTCresolved

   This incident has been resolved.

1. 2025년 6월 12일 오후 6:09 UTCinvestigating

   We are investigating elevated render error rates for the service. Previously cached derivatives are not impacted.

2. 2025년 6월 12일 오후 6:15 UTCidentified

   The issue has been identified and we are investigating a solution.

3. 2025년 6월 12일 오후 6:53 UTCidentified

   The service is experiencing elevated error rates due to a major Google Cloud outage affecting services downstream. Previously cached derivatives are not impacted. We are investigating ways to mitigate this issue.

4. 2025년 6월 12일 오후 8:26 UTCmonitoring

   The service is restored. We are monitoring the situation.

5. 2025년 6월 12일 오후 8:38 UTCidentified

   The Rendering API has fully recovered. We are continuing to investigate Web Administration related issues (login and Management API) related to the incident.

6. 2025년 6월 12일 오후 8:47 UTCmonitoring

   The Rendering API is fully recovered. Web Administration tooling (logins and the Management API) are recovering. We are monitoring the results.

7. 2025년 6월 12일 오후 8:52 UTCresolved

   The service is completely restored.

8. 2025년 6월 20일 오후 7:03 UTCpostmortem

   # Incident Summary Between **17:55 and 20:22 UTC** on **June 12, 2025**, Imgix services experienced major service disruptions across several key interfaces: * **Dashboard and Asset Manager**: These interfaces were inaccessible, preventing users from managing their assets or viewing account information. * **Management API**: Requests to the Management API consistently returned errors, affecting workflows reliant on programmatic updates or asset administration. * **Rendering API:** Approximately 8% of all Imgix service requests failed due to a high error rate \(~80%\) for **uncached** assets via the Rendering API. Requests in the EU saw a lower failure rate \(~50%\) and a faster recovery time \(45 minutes\) for these uncached requests. # What caused it The incident was triggered by a global outage within Google Cloud, which serves as a core infrastructure provider for Imgix. The outage affected most services in all regions simultaneously. You can read more about the [Google Cloud outage here](https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1SsW#RP1d9aZLNFZEJmTBk8e1). # What happened * **17:55 UTC:** Internal alerts triggered due to a spike in rendering errors and service timeouts. * **18:01 UTC:** A short investigation uncovers several timeouts and increased error rates from Google Cloud. * **18:09 UTC:** Our status page is updated. * **18:53 UTC:** A major Google Cloud outage is confirmed, after which we update our status page. * **18:00–20:16 UTC:** Mitigation efforts were hampered by the far-reaching effects of the outage, preventing us from redirecting traffic or applying configuration changes. * **Throughout:** We confirm that cached images were not affected, though the downtime of several data sources prevents evaluating the full scope and effect of the outage. * **20:16 UTC:** Google reported recovery in all regions except `us-central1`. This allowed us to verify **significantly lower error rates for EU traffic** * **20:47 UTC:** Imgix systems achieved full recovery. * **20:52 UTC:** The incident was officially resolved on our status page. * **21:23 UTC:** Google confirms a full-service recovery at 21:23 UTC. # What went wrong * Google Cloud experienced an outage that simultaneously affected nearly every service in every region worldwide, negating our multi-region redundancy for the image rendering service. * 3rd party services \(such as our CDN\) were also affected by the outage, which removed some of our options for redirecting traffic across regions based on performance. * The outage included the control panes that Google provides to its customers, which removed additional options for redirecting traffic and implementing mitigations. # What we will do to prevent this in the future * Continuing our ongoing internal discussions and evaluations of a multi-cloud render stack to enable failover in the event of a provider-wide outages. * Continue evaluating and improving tools to automatically and manually shift traffic as necessary at each layer of the stack. * Review and enhance incident communication protocols, focusing on faster root cause disclosure and update frequency.

1. 2025년 5월 20일 오후 3:40 UTCinvestigating

   We are currently investigating elevated render error rates for uncached derivative images and the Management API in the NA region. We will update once when we obtain more information. Previously cached derivatives are not impacted.

2. 2025년 5월 20일 오후 3:57 UTCidentified

   The issue has been identified and a fix is being implemented.

3. 2025년 5월 20일 오후 4:01 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

4. 2025년 5월 20일 오후 4:10 UTCresolved

   This incident has been resolved.

1. 2025년 4월 24일 오후 12:29 UTCidentified

   Some customers are experiencing issues with logging into the dashboard. We have identified the issue and are working on a fix.

2. 2025년 4월 24일 오후 2:10 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

3. 2025년 4월 24일 오후 3:31 UTCresolved

   This incident has been resolved.