Storj 장애 기록

1. 2026년 10월 20일 오후 4:25 UTCidentified

   문제는 식별되고 수정이 구현됩니다.

2. 2026년 10월 20일 오후 7:00 UTCresolved

   이 사건은 해결되었습니다.

공식 인시던트 업데이트를 자동 번역했습니다.

1. 2026년 7월 4일 오후 4:39 UTCinvestigating

   We're currently investigating reports of errors that occur when attempting to access resources, including objects and buckets.

2. 2026년 7월 4일 오후 5:54 UTCmonitoring

   We have implemented a fix and are monitoring the results.

3. 2026년 7월 4일 오후 5:54 UTCmonitoring

   We have applied a fix for the previously mentioned performance impact and are continuing to monitor the system.

4. 2026년 7월 4일 오후 5:56 UTCmonitoring

   We are continuing to monitor for any further issues.

5. 2026년 7월 4일 오후 6:10 UTCresolved

   The previously identified issue affecting performance has been resolved.

1. 2026년 7월 2일 오후 8:00 UTCinvestigating

   We are investigating reports of EU1 login errors.

2. 2026년 7월 2일 오후 8:01 UTCinvestigating

   We are continuing to investigate this issue.

3. 2026년 7월 2일 오후 8:03 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

4. 2026년 7월 2일 오후 8:05 UTCresolved

   This incident has been resolved.

1. 2026년 6월 3일 오후 7:21 UTCinvestigating

   We are investigating reports of elevated error rates from our API services.

2. 2026년 6월 3일 오후 7:21 UTCinvestigating

   We are continuing to investigate this issue.

3. 2026년 6월 3일 오후 7:29 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

4. 2026년 6월 3일 오후 7:53 UTCresolved

   This incident has been resolved.

1. 2026년 3월 18일 오후 7:44 UTCinvestigating

   We are investigating reports of elevated error rates from our API services.

2. 2026년 3월 18일 오후 8:12 UTCidentified

   The issue has been identified and a fix is being implemented.

3. 2026년 3월 18일 오후 8:59 UTCidentified

   We are continuing to work on a fix for this issue.

4. 2026년 3월 18일 오후 9:25 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

5. 2026년 3월 18일 오후 10:35 UTCresolved

   This incident has been resolved.

1. 2026년 3월 6일 오전 10:10 UTCinvestigating

   We are currently investigating this issue.

2. 2026년 3월 6일 오전 10:48 UTCmonitoring

   We have applied a fix for the previously mentioned performance impact and are continuing to monitor the system.

3. 2026년 3월 6일 오후 12:31 UTCresolved

   The previously identified issue affecting performance has been resolved.

1. 2026년 2월 24일 오후 6:26 UTCinvestigating

   We are investigating the issue

2. 2026년 2월 24일 오후 6:27 UTCinvestigating

   We are continuing to investigate this issue.

3. 2026년 2월 24일 오후 6:58 UTCidentified

   The issue has been identified and a fix is being implemented.

4. 2026년 2월 24일 오후 6:58 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

5. 2026년 2월 24일 오후 7:32 UTCresolved

   This incident has been resolved.

1. 2025년 12월 15일 오후 12:31 UTCresolved

   The issue with elevated error rates for uploads has been resolved for all affected users. The incident was limited to a region-specific storage tier in the United States.

1. 2025년 12월 10일 오후 11:46 UTCinvestigating

   We are experiencing an issue with elevated error rates for uploads. The incident is limited to a region-specific storage tier in the United States. Our engineering team is actively investigating. We apologize to all who are affected by the disruption.

2. 2025년 12월 10일 오후 11:56 UTCresolved

   The issue with elevated error rates for uploads has been resolved for all affected users. The incident was limited to a region-specific storage tier in the United States. We thank you for your patience while we worked on resolving the issue.

1. 2025년 9월 18일 오후 6:41 UTCinvestigating

   The main storj landing page is returning 500 errors. All Storj services remain unaffected. Use https://www.storj.io/login to login to the web console and use your accounts until this is resolved.

2. 2025년 9월 18일 오후 7:34 UTCresolved

   This incident has been resolved.

1. 2025년 9월 16일 오후 4:09 UTCinvestigating

   We are investigating reports of elevated error rates from our API services.

2. 2025년 9월 16일 오후 4:30 UTCidentified

   The issue has been identified and a fix is being implemented.

3. 2025년 9월 16일 오후 4:36 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

4. 2025년 9월 16일 오후 4:51 UTCresolved

   This incident has been resolved.

1. 2025년 9월 13일 오전 7:42 UTCinvestigating

   We have been alerted to an issue causing access keys in the EU1 region to stop working.

2. 2025년 9월 13일 오전 8:14 UTCinvestigating

   We are continuing to investigate this issue.

3. 2025년 9월 13일 오전 8:22 UTCresolved

   This incident has been resolved.

4. 2025년 9월 19일 오후 3:30 UTCpostmortem

   On September 13th, 2025, we experienced an issue that caused access keys to stop working for some customers in the EU1 region. This problem was isolated to a small subset of our infrastructure. We identified the root cause as a database configuration mismatch that occurred during a software update performed the previous day at approximately 21:12 UTC. Our team quickly redirected traffic away from the affected server to minimize service interruption. After implementing the necessary configuration changes, we were able to restore full functionality. The service has fully recovered, and we're taking steps to prevent similar issues in the future. We apologize for any inconvenience this may have caused.

1. 2025년 9월 3일 오후 8:13 UTCinvestigating

   We are investigating reports of elevated error rates during uploads

2. 2025년 9월 3일 오후 8:14 UTCidentified

   The issue has been identified and a fix is being applied

3. 2025년 9월 3일 오후 9:09 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

4. 2025년 9월 3일 오후 9:49 UTCresolved

   This incident has been resolved

1. 2025년 8월 27일 오후 9:25 UTCidentified

   Email sending on AP1 is disabled due to a bug which caused excessive amounts of bounced emails in a short period of time. The root cause of the bounced emails has been fixed, and we are working with our mail service provider to re-enable sending from ap1.storj.io as soon as possible.

2. 2025년 8월 28일 오후 6:37 UTCresolved

   This incident has been resolved.

3. 2025년 9월 19일 오후 9:14 UTCpostmortem

   On 2025-08-04, we enabled a “trial expiration escalation” job, which runs every day to automatically mark accounts who have completed their free trial without upgrading as “pending deletion”, 90 days after trial expiration. When accounts are marked as “pending deletion”, they can no longer be accessed, but the data still needs to be deleted. On 2025-08-26, we executed a batch of these deletions, which fully deactivated 50k\+ users, deleting their data, and moving them from a “pending deletion” to “deactivated” state. Due to a bug in the “trial expiration escalation” job, a large portion of these deleted users were moved backwards from a “deactivated” status to a “pending deletion” status, resulting in a large batch of associated notification emails being sent to the already-deactivated account emails. A fully deactivated account has an email which looks like `deactivated- @storj.io`, and prior to this incident, `[email protected]` was not a registered email address. In other words, on 2025-08-25, we attempted to send 50k\+ emails that bounced, to `[email protected]`. Our email service provider temporarily disabled our [ap1.storj.io](http://ap1.storj.io) sending domain as a result, due to “poor mailing list quality”. Once we became aware of the issue, we quickly registered `[email protected]` so that future emails sent to that address would not bounce. Additionally, we disabled our “trial expiration escalation” job to give us time to investigate and identify the issue described above. Finally, we corresponded with the support team for our email service provider to get email sending for AP1 reinstated as soon as possible. The full range of time that email service for AP1 was down spanned from roughly 2025-08-27 at 18:46 UTC, to 2025-08-28 at 16:35 UTC.

1. 2025년 8월 26일 오전 5:32 UTCidentified

   We have identified the cause of the elevated API error rate and will be applying a fix shortly.

2. 2025년 8월 26일 오전 6:19 UTCmonitoring

   A fix has been applied and we will continue to monitor the system.

3. 2025년 8월 26일 오전 6:41 UTCresolved

   The previously identified issue has been resolved.

1. 2025년 8월 7일 오후 9:25 UTCinvestigating

   We are investigating upload/download failures on US1

2. 2025년 8월 7일 오후 9:30 UTCidentified

   The issue has been identified and a fix is being implemented.

3. 2025년 8월 7일 오후 9:55 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

4. 2025년 8월 7일 오후 10:31 UTCmonitoring

   We are investigating another increase in error rates.

5. 2025년 8월 7일 오후 10:45 UTCmonitoring

   Error rate are normal and we are continuing to monitor for further issues.

6. 2025년 8월 7일 오후 11:05 UTCresolved

   This incident has been resolved.

7. 2025년 8월 22일 오후 7:24 UTCpostmortem

   ### Summary of the Storj US1 Service Disruption on August 7, 2025 #### Overview: On August 7, 2025, at approximately 21:04 UTC, the Storj US1 satellite experienced a performance degradation due to an unusually high volume of concurrent uploads to the same objects. The incident affected only the US1 satellite, while the AP1 and EU1 satellites remained fully operational throughout the event. #### Root Cause: The primary root cause was an exceptionally high volume of simultaneous uploads targeting the same objects, which triggered a bottleneck in the database due to transaction contention. This led to a cascade of issues, including database locking, request timeouts, and connection pool exhaustion. Database transactions can cause issues because they hold locks on the affected data, preventing other operations from accessing or modifying that data until the transactions are committed or rolled back.  The increased level of long running transactions eventually leads to request timeouts. #### Impact: The incident affected customers and applications relying on the Storj US1 satellite for storage and retrieval operations. The AP1 and EU1 satellites were not affected by this incident and continued to operate normally. #### Timeline: 21:04 UTC: Database transaction locks started to increase. 21:14 UTC: The on-call team received a page and started investigating the issue. 21:55 UTC: A fix was implemented and the on-call team started monitoring the results. 22:31 UTC: The on-call team started investigating another increase in error rates. 22:45 UTC: Error rates trended back down to normal and the on-call team continued to monitor for further issues. 23:05 UTC: Operations were fully restored. #### Remediation and Prevention: To address the issue, we periodically reset the state of connections thus avoiding a cascading growth of contention related errors. To prevent similar incidents from occurring in the future, we implemented the following measures: 1. Identified and resolved sources of contentions between database transactions by implementing appropriate code updates. 2. Implemented additional monitoring and alerting mechanisms to detect and notify the team of increased levels of database locks. 3. Conducted thorough post-mortem analysis to identify any other potential improvements to our processes and systems, including building tools that ease or eliminate the need for manual maintenance. We apologize for any impact this service disruption may have caused our customers and users. We are committed to learning from this incident and continuing to improve the reliability and resilience of our platform.

1. 2025년 7월 22일 오후 8:57 UTCinvestigating

   We are investigating reports of elevated error rates from our API services.

2. 2025년 7월 22일 오후 10:30 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

3. 2025년 7월 22일 오후 10:53 UTCresolved

   This incident has been resolved.

1. 2025년 7월 22일 오후 8:56 UTCresolved

   We are investigating reports of elevated error rates from our API services.

1. 2025년 7월 9일 오전 2:19 UTCidentified

   We have identified the cause of the elevated API error rate and will be applying a fix shortly.

2. 2025년 7월 9일 오전 3:23 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

3. 2025년 7월 9일 오전 5:15 UTCresolved

   The previously identified issue has been resolved.

1. 2025년 6월 18일 오전 7:59 UTCinvestigating

   Some requests to Storj US1 are failing with error 500. We are investigating the issue.

2. 2025년 6월 18일 오전 9:55 UTCmonitoring

   We identified high lock time in the database and we scaled it up. We are already seeing reduced error levels as we continue to monitor.

3. 2025년 6월 18일 오후 1:58 UTCmonitoring

   Our additional analysis spotted a problem with the Redis cache. We restarted Redis and saw the remaining errors clearing.

4. 2025년 6월 18일 오후 2:27 UTCresolved

   No more errors have been observed since the last intervention.