51 Storj incidents · czerwiec 2024 — official updates, affected components, duration and resolution details.
Problem został zidentyfikowany i jest w trakcie wdrażania.
Ten incydent został rozwiązany.
Automatyczne tłumaczenie oficjalnej aktualizacji incydentu.
We're currently investigating reports of errors that occur when attempting to access resources, including objects and buckets.
We have implemented a fix and are monitoring the results.
We have applied a fix for the previously mentioned performance impact and are continuing to monitor the system.
We are continuing to monitor for any further issues.
The previously identified issue affecting performance has been resolved.
We are investigating reports of EU1 login errors.
We are continuing to investigate this issue.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We are investigating reports of elevated error rates from our API services.
We are continuing to investigate this issue.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We are investigating reports of elevated error rates from our API services.
The issue has been identified and a fix is being implemented.
We are continuing to work on a fix for this issue.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We are currently investigating this issue.
We have applied a fix for the previously mentioned performance impact and are continuing to monitor the system.
The previously identified issue affecting performance has been resolved.
We are investigating the issue
We are continuing to investigate this issue.
The issue has been identified and a fix is being implemented.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
The issue with elevated error rates for uploads has been resolved for all affected users. The incident was limited to a region-specific storage tier in the United States.
We are experiencing an issue with elevated error rates for uploads. The incident is limited to a region-specific storage tier in the United States. Our engineering team is actively investigating. We apologize to all who are affected by the disruption.
The issue with elevated error rates for uploads has been resolved for all affected users. The incident was limited to a region-specific storage tier in the United States. We thank you for your patience while we worked on resolving the issue.
The main storj landing page is returning 500 errors. All Storj services remain unaffected. Use https://www.storj.io/login to login to the web console and use your accounts until this is resolved.
This incident has been resolved.
We are investigating reports of elevated error rates from our API services.
The issue has been identified and a fix is being implemented.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We have been alerted to an issue causing access keys in the EU1 region to stop working.
We are continuing to investigate this issue.
This incident has been resolved.
On September 13th, 2025, we experienced an issue that caused access keys to stop working for some customers in the EU1 region. This problem was isolated to a small subset of our infrastructure. We identified the root cause as a database configuration mismatch that occurred during a software update performed the previous day at approximately 21:12 UTC. Our team quickly redirected traffic away from the affected server to minimize service interruption. After implementing the necessary configuration changes, we were able to restore full functionality. The service has fully recovered, and we're taking steps to prevent similar issues in the future. We apologize for any inconvenience this may have caused.
We are investigating reports of elevated error rates during uploads
The issue has been identified and a fix is being applied
A fix has been implemented and we are monitoring the results.
This incident has been resolved
Email sending on AP1 is disabled due to a bug which caused excessive amounts of bounced emails in a short period of time. The root cause of the bounced emails has been fixed, and we are working with our mail service provider to re-enable sending from ap1.storj.io as soon as possible.
This incident has been resolved.
On 2025-08-04, we enabled a “trial expiration escalation” job, which runs every day to automatically mark accounts who have completed their free trial without upgrading as “pending deletion”, 90 days after trial expiration. When accounts are marked as “pending deletion”, they can no longer be accessed, but the data still needs to be deleted. On 2025-08-26, we executed a batch of these deletions, which fully deactivated 50k\+ users, deleting their data, and moving them from a “pending deletion” to “deactivated” state. Due to a bug in the “trial expiration escalation” job, a large portion of these deleted users were moved backwards from a “deactivated” status to a “pending deletion” status, resulting in a large batch of associated notification emails being sent to the already-deactivated account emails. A fully deactivated account has an email which looks like `deactivated- @storj.io`, and prior to this incident, `[email protected]` was not a registered email address. In other words, on 2025-08-25, we attempted to send 50k\+ emails that bounced, to `[email protected]`. Our email service provider temporarily disabled our [ap1.storj.io](http://ap1.storj.io) sending domain as a result, due to “poor mailing list quality”. Once we became aware of the issue, we quickly registered `[email protected]` so that future emails sent to that address would not bounce. Additionally, we disabled our “trial expiration escalation” job to give us time to investigate and identify the issue described above. Finally, we corresponded with the support team for our email service provider to get email sending for AP1 reinstated as soon as possible. The full range of time that email service for AP1 was down spanned from roughly 2025-08-27 at 18:46 UTC, to 2025-08-28 at 16:35 UTC.
We have identified the cause of the elevated API error rate and will be applying a fix shortly.
A fix has been applied and we will continue to monitor the system.
The previously identified issue has been resolved.
We are investigating upload/download failures on US1
The issue has been identified and a fix is being implemented.
A fix has been implemented and we are monitoring the results.
We are investigating another increase in error rates.
Error rate are normal and we are continuing to monitor for further issues.
This incident has been resolved.
### Summary of the Storj US1 Service Disruption on August 7, 2025 #### Overview: On August 7, 2025, at approximately 21:04 UTC, the Storj US1 satellite experienced a performance degradation due to an unusually high volume of concurrent uploads to the same objects. The incident affected only the US1 satellite, while the AP1 and EU1 satellites remained fully operational throughout the event. #### Root Cause: The primary root cause was an exceptionally high volume of simultaneous uploads targeting the same objects, which triggered a bottleneck in the database due to transaction contention. This led to a cascade of issues, including database locking, request timeouts, and connection pool exhaustion. Database transactions can cause issues because they hold locks on the affected data, preventing other operations from accessing or modifying that data until the transactions are committed or rolled back. The increased level of long running transactions eventually leads to request timeouts. #### Impact: The incident affected customers and applications relying on the Storj US1 satellite for storage and retrieval operations. The AP1 and EU1 satellites were not affected by this incident and continued to operate normally. #### Timeline: 21:04 UTC: Database transaction locks started to increase. 21:14 UTC: The on-call team received a page and started investigating the issue. 21:55 UTC: A fix was implemented and the on-call team started monitoring the results. 22:31 UTC: The on-call team started investigating another increase in error rates. 22:45 UTC: Error rates trended back down to normal and the on-call team continued to monitor for further issues. 23:05 UTC: Operations were fully restored. #### Remediation and Prevention: To address the issue, we periodically reset the state of connections thus avoiding a cascading growth of contention related errors. To prevent similar incidents from occurring in the future, we implemented the following measures: 1. Identified and resolved sources of contentions between database transactions by implementing appropriate code updates. 2. Implemented additional monitoring and alerting mechanisms to detect and notify the team of increased levels of database locks. 3. Conducted thorough post-mortem analysis to identify any other potential improvements to our processes and systems, including building tools that ease or eliminate the need for manual maintenance. We apologize for any impact this service disruption may have caused our customers and users. We are committed to learning from this incident and continuing to improve the reliability and resilience of our platform.
We are investigating reports of elevated error rates from our API services.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We are investigating reports of elevated error rates from our API services.
We have identified the cause of the elevated API error rate and will be applying a fix shortly.
A fix has been implemented and we are monitoring the results.
The previously identified issue has been resolved.
Some requests to Storj US1 are failing with error 500. We are investigating the issue.
We identified high lock time in the database and we scaled it up. We are already seeing reduced error levels as we continue to monitor.
Our additional analysis spotted a problem with the Redis cache. We restarted Redis and saw the remaining errors clearing.
No more errors have been observed since the last intervention.
