Aptible 장애 기록

1. 2026년 7월 27일 AM 10:31 UTCinvestigating

   AWS는 us-east-1에서 API를 가진 문제점을 경험하고, 많은 서비스는 이 시간에 충격을 받습니다. 이것은 실패한 가동에 지도할지도 모릅니다. 더 많은 문제를 피하기 위해, 우리는이 지역의 장애인 운영을 가지고 있으며, 우리는 전체적인 영향을 이해합니다.

2. 2026년 7월 27일 PM 12:55 UTCinvestigating

   가동은 us-east-1에 있는 무능하 남아 있습니다, 우리는 지구에 있는 AWS 네트워킹 문제점을 조사하는 것을 계속합니다. 우리는 소송을 테스트하고 작업을 다시 복원 할 것입니다. 우리는 다른 업데이트를 제공 할 것입니다 이상 10 : 00 AM 동부 시간.

3. 2026년 7월 27일 PM 1:43 UTCmonitoring

   우리는 us-east-1에 있는 mitigation와 re-enabled 가동을 적용했습니다. 우리는 안정성을 위해 모니터링하고 우리가 조건이 안정적으로 유지되도록 한 번이 사건을 해결합니다.

4. 2026년 7월 27일 PM 3:12 UTCresolved

   이 사건은 해결되었습니다.

공식 인시던트 업데이트를 자동 번역했습니다.

1. 2026년 7월 16일 PM 10:31 UTCidentified

   LLM Gateway에 지연 및 타임 아웃 요청을 인식하고 있습니다. 우리는 엔지니어가 뿌리 원인을 해결하기 위해 계속되는 동안 영향을 최소화하는 데 도움이되는 몇 가지 변화를 만들었습니다.

2. 2026년 7월 17일 AM 12:02 UTCmonitoring

   우리는 자원 용량을 늘리고 결과를 모니터링하고 있습니다.

3. 2026년 7월 17일 PM 12:14 UTCmonitoring

   9:57 UTC (5:57 EDT)에서 시작하여 11:12 UTC (7:12 EDT)까지 지속되는 성능의 회귀 후, 우리는 잠재적 인 수정을 배포하고 우리의 서비스 사기를 업데이트했습니다. 우리는이 사건을 폐쇄하기 전에이 문제를 완전히 해결하는지 여부를 모니터링하고 있습니다.

4. 2026년 7월 17일 PM 5:04 UTCresolved

   이전 업데이트 당 배포 된 수정 뒤에, 우리는 7:12 EDT 이후 LLM 게이트웨이에 안정적인 성능을 보았고 문제가 해결됩니다. 이 문제점의 뿌리 원인은 3개의 특정한 점에 있는 가혹한 성과 degradation이었습니다: • `GET /모델` • `GET /v1/models` • `GET /metadata/모델` ... 모든 가능한 모델 목록을 반환합니다. 이 엔드포인트의 성능 문제는 다른 비공식적인 엔드포인트에 대한 중요한 요청 쿼리 및 등급의 성능을 유발하기에 충분했습니다. 수정 우리는 빠른 응답을 보장하기 위해이 엔드 포인트에 추가 캐싱을 배치. 이러한 수정이 배포 된 이후, 우리는 더 이상 다른 LLM Gateway API 엔드포인트에 대한 요청 쿼리 또는 영향을 볼 수 없습니다.

공식 인시던트 업데이트를 자동 번역했습니다.

1. 2026년 6월 8일 PM 6:28 UTCinvestigating

   As part of our security policy and ongoing commitment to protecting customer workloads, we are applying patches for a newly disclosed Systemd vulnerability (CVE-2026-40226) across our platform. This patching will be rolled out automatically over the next several days. During this process, you may experience a momentary network disruption to running applications and databases. Customers configured for high-availability with horizontally-scaled services should experience little to no disruption from this update. No action is required on your part. If you have any questions or concerns, please reach out to our Support team at https://contact.aptible.com/.

2. 2026년 6월 23일 PM 9:48 UTCresolved

   This incident has been resolved.

1. 2026년 6월 3일 PM 4:02 UTCinvestigating

   There is some partial service degradation with log drains configured to send to Mezmo where not all log messages are being sent. Mezmo is rejecting logs over a certain size. We are actively investigating this issue with the upstream provider.

2. 2026년 6월 3일 PM 7:16 UTCresolved

   As of 18:42 UTC the upstream provider began to accept log messages again consistently and this issue is now remediated.

1. 2026년 5월 8일 AM 5:02 UTCmonitoring

   Aptible is aware of “Dirty Frag,” a recently disclosed Linux kernel vulnerability that may allow local privilege escalation under certain conditions [0]. We have taken action across the Aptible platform to protect all customer workloads against this vulnerability. No customer action is required at this time. We will continue to monitor upstream kernel, distribution, and security vendor guidance related to this vulnerability and will take any additional action if needed. [0] https://www.openwall.com/lists/oss-security/2026/05/07/8

2. 2026년 5월 8일 PM 3:35 UTCresolved

   This incident has been resolved.

1. 2026년 5월 8일 AM 1:05 UTCinvestigating

   Due to an underlying issue in AWS we are experiencing instability in a certain availability zone in us-east-1. We are investigating this issue and will follow up with recommendations shortly

2. 2026년 5월 8일 AM 1:39 UTCidentified

   We have blocked operations from provisioning into the specified availability zone

3. 2026년 5월 8일 PM 2:02 UTCmonitoring

   We have unblocked operations from the specified region as we have seen successful provisioning and maintenance of new instance in that availability zone. Instances that were affected have not yet seen recovery, and the upstream AWS alert indicates they may still take hours of recovery.

4. 2026년 5월 9일 AM 3:35 UTCresolved

   An upstream AWS issue affected infrastructure in a single Availability Zone in us-east-1. During the event, we blocked provisioning into the affected zone and monitored AWS’s recovery efforts. AWS has since recovered enough of the affected availability zone for the impacted Aptible infrastructure to recover. We have also completed recovery of any Aptible-managed resources that required follow-up action on our side. Any customers with directly affected resources have been notified of resolution. We are continuing normal platform monitoring, but we are marking this incident as resolved on our end.

1. 2026년 4월 29일 PM 10:40 UTCresolved

   Aptible is aware of CVE-2026-31431, also known as “Copy Fail,” a recently disclosed Linux kernel vulnerability that may allow local privilege escalation under certain conditions [0]. We have taken action across the Aptible platform to protect all customer workloads against this vulnerability. No customer action is required at this time. We will continue to monitor upstream kernel, distribution, and security vendor guidance related to this CVE and will take any additional action if needed. [0] https://copy.fail/

1. 2026년 4월 19일 AM 2:03 UTCinvestigating

   We are investigating an incident impacting api.aptible.com. Viewing current resources and running operations are currently impacted, however running applications and databases are unaffected

2. 2026년 4월 19일 AM 2:54 UTCmonitoring

   A fix has been implemented and operations that were failing to enqueue during the availability outage have been retried. We will continue monitoring the situation.

3. 2026년 4월 19일 AM 3:10 UTCresolved

   This incident has been resolved.

1. 2026년 3월 17일 PM 10:33 UTCinvestigating

   We are currently observing intermittent DNS resolution failures affecting infrastructure in us-east-1. These resolver errors may cause some Aptible Deploy operations (such as deploys, restarts, SSH, or container provisioning) to fail or time out. Applications running on Aptible Deploy that rely on network hostname resolution may also experience intermittent connection failures related to DNS resolution. We are actively working with our infrastructure providers to investigate the issue and will provide updates here as they are available.

2. 2026년 3월 17일 PM 11:02 UTCmonitoring

   We have observed DNS resolution error rates return to normal levels, and all dependent activities should be functioning normally. We will continue to monitor for an additional period to ensure the issue is fully resolved before closing the incident. Further updates will be posted here if we observe any recurrence.

3. 2026년 3월 17일 PM 11:33 UTCresolved

   This incident has been resolved.

4. 2026년 3월 18일 PM 4:42 UTCpostmortem

   Additional detail about timeline: we observed a relative increase in DNS resolver errors in us-east-1 beginning at approximately 2026-03-17 18:30 UTC. The elevated error rate was intermittent rather than constant, with the most visible clusters occurring from roughly 18:30-18:55 UTC and again from about 20:55-21:40 UTC. Error rates returned to normal shortly after 21:40 UTC.

1. 2025년 12월 31일 AM 10:48 UTCinvestigating

   We are aware of degraded performance affecting auth.aptible.com. This affects all actions that interact with our APIs including login, viewing and managing resources, etc.

2. 2025년 12월 31일 AM 11:33 UTCmonitoring

   Performance has improved since our mitigations. We are still monitoring the situation for continued degradation.

3. 2025년 12월 31일 PM 2:49 UTCresolved

   This incident has been resolved.

1. 2025년 12월 30일 AM 2:14 UTCmonitoring

   The Aptible Security Team is aware of the recently disclosed vulnerability CVE-2025-14847 (https://github.com/advisories/GHSA-4742-mr57-2r9j) affected all MongoDB versions. The vulnerability could allow an attacker with network access to a MongoDB database to exfiltrate data, including sensitive data and/or credentials, without authenticating to the database. In response to the vulnerability, we have updated our supported MongoDB versions to prevent exploitations of this vulnerability. In addition, we have proactively restarted databases matching either of the following criteria, to ensure they are running on the latest protected versions: * All databases in shared-tenancy stacks, and * All databases with customer-created public endpoints that do not have access restricted to an IP allow list Since Aptible databases run on private networks by default, most Aptible managed MongoDB databases are not accessible from the internet or by other Aptible customers, and so are not vulnerable to CVE-2025-14847. As such, we did not proactively restart these databases. Customers may restart their databases at any time to update to the latest protected versions.

2. 2025년 12월 30일 PM 5:15 UTCresolved

   This incident has been resolved.

1. 2025년 12월 5일 PM 2:39 UTCinvestigating

   We are aware of the recently disclosed critical vulnerability CVE-2025-55182 (https://react.dev/blog/2025/12/03/critical-security-vulnerability-in-react-server-components) affecting React Server Components, which could allow remote code execution under certain conditions. This vulnerability affects certain React 19.x and frameworks that implement React Server Components, including Next.js. *Aptible Platform Impact:* Aptible has reviewed all infrastructure components that we manage and has confirmed that all are unaffected by this vulnerability. *Customer Application Impact:* If you are running applications on Aptible that use React Server Components, you may be affected. We have seen active exploitation of this vulnerability, and we recommend upgrading to the patched versions immediately. Specifically, applications using: - React 19.0.0, 19.1.0, 19.1.1, or 19.2.0 with react-server-dom-webpack, react-server-dom-turbopack, or react-server-dom-parcel - Next.js versions 14.3.0-canary.77 and all subsequent 14.3.x canary releases, 15.0.0, 15.0.1, 15.0.2, 15.0.3, 15.0.4, 15.1.0, 15.1.1, 15.1.2, 15.1.3, 15.1.4, 15.1.5, 15.1.6, 15.1.7, 15.1.8, 15.2.0, 15.2.1, 15.2.2, 15.2.3, 15.2.4, 15.2.5, 15.3.0, 15.3.1, 15.3.2, 15.3.3, 15.3.4, 15.3.5, 15.4.0, 15.4.1, 15.4.2, 15.4.3, 15.4.4, 15.4.5, 15.4.6, 15.4.7, 15.5.0, 15.5.1, 15.5.2, 15.5.3, 15.5.4, 15.5.5, 15.5.6, 15.6.0-canary.0 through 15.6.0-canary.57, 16.0.0, 16.0.1, 16.0.2, 16.0.3, 16.0.4, 16.0.5, 16.0.6 - Other frameworks implementing React Server Components including Vite, Parcel, React Router, RedwoodSDK, Waku *Additional Resources:* React Security Advisory: https://github.com/facebook/react/security/advisories/GHSA-fv66-9v8q-g76r Next.js Security Advisory: https://github.com/vercel/next.js/security/advisories/GHSA-9qr9-h5gf-34mp We will continue to update this incident page as more information becomes available.

2. 2025년 12월 8일 PM 4:52 UTCresolved

   This incident has been resolved.

1. 2025년 11월 5일 PM 3:59 UTCidentified

   We are investigating an issue causing degraded performance for the Aptible Platform API. As part of the mitigation, a temporary operation block across all regions is in place which prevents new operations from starting. This includes deployments, scaling actions, SSH sessions, and database tunnels. Running apps and databases are not affected. Next update We will provide an update by 11:30 ET or sooner.

2. 2025년 11월 5일 PM 4:05 UTCmonitoring

   A fix has been implemented and we are monitoring the results. The operation block has been lifted.

3. 2025년 11월 5일 PM 4:07 UTCresolved

   This incident has been resolved.

1. 2025년 10월 29일 PM 9:17 UTCinvestigating

   We are currently experiencing failures when provisioning new hosts across all regions. This impacts operations that require new capacity, including but not limited to: app deploys, scaling events, and new database creation. Existing running workloads are not affected. Our team is actively investigating and working to restore normal operation performance. We will provide an update by no later than 5:45pm EST.

2. 2025년 10월 29일 PM 9:33 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

3. 2025년 10월 29일 PM 9:50 UTCresolved

   This incident has been resolved.

1. 2025년 10월 28일 PM 5:09 UTCinvestigating

   We are seeing significantly increased provisioning times in the AWS us-east-1 region. This is affecting operations that require new capacity, including deployments, scaling, and creation of new resources. These operations may time out and fail. Our team is actively investigating and working to mitigate the impact. We will provide our next update by 2 pm EST.

2. 2025년 10월 28일 PM 5:32 UTCmonitoring

   AWS has confirmed the issue with EC2 provisioning issues in us-east-1. Meanwhile, we have been able to implement a mitigation for affected customers. We will continue to leave this issue open and monitor its progress, and will provide any relevant updates as needed.

3. 2025년 10월 28일 PM 5:53 UTCidentified

   We are now seeing additional broader EC2 provisioning issues in the AWS us-east-1 region that are outside the scope of our mitigation. Additional delays/failures on operations that require new capacity in us-east-1 may still occur at this time. We will provide our next update by 2:30 pm EST.

4. 2025년 10월 28일 PM 6:35 UTCmonitoring

   We are not currently seeing provisioning issues in AWS us-east-1. We are modifying the incident status to Monitoring, and will provide any relevant updates as needed.

5. 2025년 10월 28일 PM 9:08 UTCresolved

   This incident has been resolved.

1. 2025년 10월 20일 AM 11:53 UTCidentified

   AWS is experiencing an issue with their APIs in us-east-1, and many services are impacted at this time. This may lead to failed operations. There is currently no impact to availability for apps and databases running on Aptible at this time.

2. 2025년 10월 20일 PM 1:56 UTCidentified

   We continue to see fall out from the AWS outage this morning. Operations that result in scaling capacity such as service scales, deployments, etc. may be impacted, resulting in failing operations. We recommend periodically retrying such operations.

3. 2025년 10월 20일 PM 3:16 UTCidentified

   We’re continuing to see widespread impact from the ongoing AWS outage in us-east-1. Most requests for new EC2 instances in this region are currently failing. As a result, any operations that require additional instance capacity, such as service scales, deployments, and other scaling-related actions, are likely to fail. This remains an evolving AWS issue, and we’ll provide updates as more information becomes available. At this time, apps and databases operating with existing capacity appear to be running normally without impact.

4. 2025년 10월 20일 PM 3:44 UTCidentified

   We are seeing continued widespread impact from the AWS outage in us-east-1. AWS has confirmed ongoing network and API failures within the EC2 internal network that are preventing new instance launches across multiple Availability Zones. To avoid any operation failures that could leave resources in a transitional state, we have implemented an operation block for all new operations in the us-east-1 region. Apps and databases already running on existing capacity continue to operate normally at this time. We are actively monitoring AWS recovery efforts and will lift the block once we confirm increased stability in the us-east-1 region. Further updates will be posted as they become available.

5. 2025년 10월 20일 PM 6:04 UTCidentified

   We are starting to see EC2 provisioning operations in us-east-1 succeed again as AWS recovery efforts progress. We’re continuing to monitor closely, and if this trend holds, we expect to lift the operational block in the region shortly. We will provide another update as soon as the block is lifted or if conditions change.

6. 2025년 10월 20일 PM 6:55 UTCmonitoring

   We have lifted the operation block in us-east-1 as we are now seeing a sustained increase in successful EC2-related operations. While some operations may still fail as AWS recovery continues, the overall success rate has improved significantly. We are observing that EC2-related actions are taking longer to complete than usual, so users may experience increased delays in operation completion times. Given this progress, new operations have been re-enabled. We’ll continue to monitor closely and will update the incident status once failure rates and operation times have fully stabilized.

7. 2025년 10월 20일 PM 8:03 UTCmonitoring

   Platform operations in us-east-1 are fully operational at this time. We continue to actively monitor EC2-related operations for any elevated error rates. We’ll keep this incident open while AWS’s own outage remains active and will close it once AWS has resolved their incident and we’ve confirmed continued stability across all operations in us-east-1.

8. 2025년 10월 20일 PM 11:04 UTCresolved

   AWS has marked the underlying issue in us-east-1 as resolved, and we’ve confirmed continued stability across all EC2-related operations. All Aptible operations are functioning normally, and this incident is now closed.

1. 2025년 10월 20일 AM 7:47 UTCidentified

   AWS is experiencing an issue with their APIs in us-east-1, and many services are impacted at this time. This may lead to failed operations, but there appears to be no impact to availability for apps and databases running on Aptible at this time.

2. 2025년 10월 20일 AM 8:31 UTCmonitoring

   Since the AWS issues seem serious, and to avoid any operation failures causing resources to get stuck in a transitional state, we are taking action to block all new operations on the platform at this time. We have identified no impact to acceibilty or preformance of running apps and database containers at this time.

3. 2025년 10월 20일 AM 9:30 UTCmonitoring

   AWS has indicated a fix has been applied, and we are re-enabling operations. We will continue to monitor for additional impact, as both Dockerhub and Quay.io were offline due to the AWS issue.

4. 2025년 10월 20일 AM 9:46 UTCresolved

   This incident has been resolved.

1. 2025년 10월 12일 AM 2:30 UTCinvestigating

   One of our internal queuing systems observed a failure that has affected several operations enqueued in us-east-1. Our Reliability Team is restoring the queueing system and investigating the root cause of the failure.

2. 2025년 10월 12일 AM 2:42 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

3. 2025년 10월 12일 AM 3:40 UTCmonitoring

   No operation queueing issues have been observed since the last update. If you observed any operations that failed or were stuck in a "pending" state, please retry the operations and they should run without errors.

4. 2025년 10월 12일 AM 3:40 UTCresolved

   This incident has been resolved.

1. 2025년 10월 10일 PM 3:53 UTCresolved

   Aptible has reviewed and addressed the Redis vulnerability described in https://redis.io/blog/security-advisory-cve-2025-49844/ . We have confirmed that all internal Redis instances used in delivering the Aptible platform are secure and have no exploitation path related to this CVE. For Aptible customers who do not have their Redis databases publicly exposed, a path to exploitation is similarly unlikely. Additionally, for customers using Redis 6.2 databases on Aptible, a patched version will be available by Oct 10th at 5pm EST, and additional releases (7.2) are upcoming; see aptible.com/changelog for ongoing release updates. To ensure you are running the latest minor version when available, please run the cli command: aptible db:reload for each Redis database. For more information on using this command, visit https://www.aptible.com/docs/reference/aptible-cli/cli-commands/cli-db-reload . If you have any specific questions or concerns related to this CVE, please contact us .

1. 2025년 8월 7일 PM 1:43 UTCinvestigating

   Our online documentation at aptible.com/docs is temporarily unavailable. We are working with our upstream provider to resolve the issue and will update this incident when it is resolved.

2. 2025년 8월 7일 PM 2:21 UTCresolved

   This incident has been resolved.