Readme の障害履歴

1. 2026年10月23日 15:46 UTCinvestigating

   ドキュメンテーションページをロードする際に、503のエラーやタイムアウトが増加したという問題について、現在調査中です.

2. 2026年10月23日 15:46 UTCinvestigating

   今後もこの問題について調査を続けてまいります.

3. 2026年10月23日 15:47 UTCidentified

   問題が特定され、修正が実装されています.

4. 2026年10月23日 16:07 UTCmonitoring

   修正を実装し、結果を監視しています.

5. 2026年10月23日 16:09 UTCresolved

   この事件は解決しました.

公式のインシデント更新を自動翻訳しています。

1. 2026年10月14日 22:44 UTCinvestigating

   現在、404ページが表示されている理由を調査しています

2. 2026年10月14日 22:51 UTCidentified

   問題を特定し、修正に取り組んでいます

3. 2026年10月14日 22:52 UTCmonitoring

   修正が展開され、現在は運用サーバーを展開しています。 404をキャッシュする可能性があるので、問題にまだ実行されているかどうかをお知らせください

4. 2026年10月14日 22:55 UTCresolved

   プロジェクトは再び表示されます。 404 エラーが発生した場合は、[email protected] までお問い合わせください.

公式のインシデント更新を自動翻訳しています。

1. 2026年9月22日 14:22 UTCinvestigating

   現在、ハブに影響を与える高レイテンシと低応答時間を調査しています。 根本原因を特定し、正常なサービスを回復するために活動的に働く.

2. 2026年9月22日 14:41 UTCmonitoring

   修正が実装され、現在結果の監視をしています。 パフォーマンスは正常に戻ります.

3. 2026年9月22日 15:03 UTCmonitoring

   さらなる貢献因子を特定し、それらを修正するために積極的に取り組んでいます.

4. 2026年9月22日 15:16 UTCmonitoring

   今後の問題がないか引き続き監視しています。

5. 2026年9月22日 16:44 UTCresolved

   この事件は解決しました.

公式のインシデント更新を自動翻訳しています。

1. 2026年9月8日 16:04 UTCinvestigating

   私たちの管理者インターフェイスは、停電を経験しています。 ReadMe 開発者ハブにアクセスするエンドユーザの能力は影響を受けません.

2. 2026年9月8日 16:15 UTCinvestigating

   状況を解決することができました。 次の数分間、いくつかの断続的な悲しみがあるかもしれません.

3. 2026年9月8日 16:16 UTCresolved

   この事件は解決しました.

公式のインシデント更新を自動翻訳しています。

1. 2026年8月21日 16:36 UTCidentified

   現在、顧客ハブ全体で遅い検索結果を経験しています

2. 2026年8月21日 18:21 UTCresolved

   検索結果とパフォーマンスを再びインデックス化しました。 検索に問題が見つからない場合は、[email protected]までメールでお問い合わせください

公式のインシデント更新を自動翻訳しています。

1. 2026年7月24日 12:22 UTCmonitoring

   サードパーティのインフラの問題による劣化した性能の検討 私たちは、この状況を監視し、私たちの最後に安定性を維持するために働いています.

2. 2026年7月24日 13:37 UTCresolved

   この事件は解決しました.

公式のインシデント更新を自動翻訳しています。

1. 2026年6月17日 15:28 UTCinvestigating

   We are currently investigating an incident related to API Reference pages.

2. 2026年6月17日 18:10 UTCresolved

   This incident was resolved and normal site function has been resolved

1. 2026年5月28日 13:40 UTCinvestigating

   We're actively investigating this issue.

2. 2026年5月28日 13:43 UTCidentified

   The issue has been identified and a fix is being implemented.

3. 2026年5月28日 14:35 UTCidentified

   Systems are getting back up. We're continuing to work on a permanent fix for this issue.

4. 2026年5月28日 15:27 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

5. 2026年5月28日 15:33 UTCmonitoring

   We are continuing to monitor for any further issues.

6. 2026年5月28日 17:53 UTCmonitoring

   We are receiving reports of degraded performance and are actively investigating. We will provide updates as more information becomes available.

7. 2026年5月28日 20:04 UTCmonitoring

   Systems appear stable and we are actively rolling out a series of fixes. We will continue to monitor and provide updates

8. 2026年5月29日 1:49 UTCmonitoring

   Systems remain stable. Improvements and fixes have been rolled out and we are continuing to monitor.

9. 2026年5月29日 17:03 UTCmonitoring

   Systems remain stable, and we are continuing to monitor the situation. Bi-directional sync was on a maintenance job from 9:50 - 11:59 AM (ET) and is now operational.

10. 2026年5月29日 18:36 UTCmonitoring

    Systems remain stable and our team continues to deploy targeted fixes. Work is ongoing to address degraded read performance across the system, including serving all 404s from cache. We’ll be actively monitoring over the weekend, and we will provide a full RCA next week.

11. 2026年6月1日 18:16 UTCresolved

    Our systems have been stable since Friday, May 29th. We're monitoring closely and improving spike detection to prevent recurrence and adding proactive capacity monitoring to detect admin dashboard degradation before it impacts customers. Full root cause analysis coming soon. Thanks for your patience and understanding.

1. 2026年5月27日 13:51 UTCinvestigating

   We are currently investigating this issue.

2. 2026年5月27日 14:52 UTCidentified

   The issue has been identified and a fix is being implemented.

3. 2026年5月27日 15:57 UTCmonitoring

   We're slowly recovering and monitoring the situation.

4. 2026年5月27日 17:08 UTCmonitoring

   We are monitoring the issue.

5. 2026年5月27日 17:17 UTCmonitoring

   Updated 6/3: A routine configuration update coincided with the downtime window, which led us to initially identify it as the cause. Further investigation confirmed the two were unrelated. See the postmortem for full details.

6. 2026年5月27日 17:43 UTCinvestigating

   We are seeing some slow performance, possibly 500 errors. investigating

7. 2026年5月27日 17:47 UTCinvestigating

   We are continuing to investigate this issue.

8. 2026年5月27日 18:02 UTCmonitoring

   Performance has improved, but we're still investigating the issue.

9. 2026年5月27日 20:54 UTCmonitoring

   Performance and loading is still being affected, and we're in the process of rolling out fixes

10. 2026年5月27日 22:37 UTCresolved

    This incident has been resolved.

11. 2026年6月3日 15:28 UTCpostmortem

    ### What Happened Beginning Tuesday, May 26, 2026, customers were experiencing slow loading and 503 errors across ReadMe-hosted docs and the admin dashboard. The outage was intermittent but recurring, with the worst periods hitting during business hours when traffic spiked. ### Root Cause An internal data backup process was generating excessive I/O on our storage layer. Under normal traffic conditions, this additional load was manageable. But when it coincided with peak customer traffic and elevated bot activity, total I/O demand exceeded system capacity, causing cascading request timeouts. The maintenance process ran on a recurring schedule, which is why the degradation followed a predictable pattern of spikes throughout each day. Separately, a surge in bot traffic to non-existent pages \(404s\) amplified the problem because those requests were not being served from cache. ### Resolution **Immediate fix:** We identified and disabled the maintenance process causing the excess I/O load. Within three hours, storage utilization returned to normal levels and remained stable. **Additional improvements shipped during the incident:** * Expanded caching across multiple layers, significantly reducing load on backend storage * Hardened 404 handling to serve error pages from cache instead of hitting the backend * Implemented rate limiting and IP-based protections against abusive bot traffic * Optimized several high-traffic API endpoints to reduce redundant backend calls * Added new monitoring and alerting for storage I/O thresholds ### Timeline May 26 - June 1, 2026 | Time | Status | Details | | --- | --- | --- | | Mon 5/26, 6:39 AM PDT | Investigating | Issue reported | | Mon 5/26, 7:26 AM PDT | Monitoring | Fix implemented, monitoring results | | Mon 5/26, 8:37 AM PDT | Resolved | Admin hub incident resolved | | Mon 5/26, 10:41 AM PDT | Investigating | Slow performance across customer hubs | | Mon 5/26, 10:49 AM PDT | Monitoring | Quick fix applied, investigating thorough fix | | Mon 5/26, 8:54 PM PDT | Resolved | Customer hub incident resolved | | Tue 5/27, 6:51 AM PDT | Investigating | Issue reported | | Tue 5/27, 7:52 AM PDT | Identified | Fix in progress | | Tue 5/27, 8:57 AM PDT | Monitoring | Slowly recovering | | Tue 5/27, 10:17 AM PDT | Update | Updated 6/3: A routine configuration update coincided with the downtime window, which led us to initially identify it as the cause. Further investigation confirmed the two were unrelated. See root cause and resolution above. | | Tue 5/27, 1:54 PM PDT | Update | Performance and loading still affected, rolling out fixes | | Tue 5/27, 3:37 PM PDT | Resolved | Incident resolved | | Wed 5/28, 6:40 AM PDT | Investigating | Issue reported | | Wed 5/28, 6:43 AM PDT | Identified | Fix being implemented | | Wed 5/28, 7:35 AM PDT | Update | Systems coming back up, working on permanent fix | | Wed 5/28, 8:27 AM PDT | Monitoring | Fix implemented, monitoring results | | Wed 5/28, 10:53 AM PDT | Update | Reports of degraded performance, actively investigating | | Wed 5/28, 1:04 PM PDT | Update | Systems appear stable, rolling out fixes | | Wed 5/28, 6:49 PM PDT | Monitoring | Improvements and fixes deployed, continuing to monitor | | Thu 5/29, 10:03 AM PDT | Monitoring | Systems stable. Bi-directional sync maintenance 9:50–11:59 AM ET. | | Thu 5/29, 11:36 AM PDT | Monitoring | Deploying targeted fixes. Serving all 404s from cache. Degraded read performance. | | Fri 5/30 | Monitoring | Monitoring continues | | Sat 5/31 | Monitoring | Weekend monitoring | | Mon 6/1, 11:16 AM PDT | Resolved | Root cause identified and resolved. Systems stable since Thursday, May 29. | ### Path Forward We are using this incident to make lasting improvements to reliability and incident response: * **Storage capacity and isolation:** Restructuring how background processes interact with production storage to eliminate contention under load. * **Caching and performance:** The caching improvements shipped during the incident are permanent. We are continuing to expand cache coverage across additional endpoints and page types. * **Bot and traffic protection:** Strengthening rate limiting and abuse detection to prevent bot traffic from contributing to backend load. * **Monitoring and alerting:** Adding proactive capacity monitoring with earlier thresholds so the team can intervene before customers are affected. * **Incident response:** Improving our internal processes for faster escalation and more frequent status page updates during multi-day incidents. ### Final Note During the incident, we posted an update referencing a platform update. That was our initial hypothesis based on timing. Further investigation confirmed it was unrelated. The change in question was a routine, isolated configuration update and had no impact on the outage or any other customers. We should have waited for confirmation before publishing it, and we're tightening our internal process for status page updates as a result. We know how critical your documentation is to your customers, and this level of disruption is not acceptable. We have already shipped meaningful improvements to prevent recurrence, and the work outlined above will continue through the coming weeks. If you have questions, reach out to your account team or contact [[email protected]](mailto:[email protected]).

1. 2026年5月26日 17:41 UTCinvestigating

   We are experiencing slow performance across customer hubs

2. 2026年5月26日 17:49 UTCmonitoring

   We have identified and quickly fixed the issue and are currently investigating a more thorough fix

3. 2026年5月27日 3:54 UTCresolved

   This incident has been resolved.

1. 2026年5月26日 13:39 UTCinvestigating

   We are currently investigating this issue.

2. 2026年5月26日 14:26 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

3. 2026年5月26日 15:37 UTCresolved

   This incident has been resolved.

1. 2026年5月12日 14:15 UTCidentified

   We are investigating slowness across hubs.

2. 2026年5月12日 19:38 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

3. 2026年5月12日 21:10 UTCresolved

   This incident has been resolved.

1. 2026年5月11日 15:43 UTCinvestigating

   We are currently investigating this issue.

2. 2026年5月11日 16:03 UTCidentified

   The issue has been identified and a fix is being implemented.

3. 2026年5月11日 17:08 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

4. 2026年5月11日 18:34 UTCresolved

   This incident has been resolved.

1. 2026年5月7日 23:43 UTCinvestigating

   We are currently investigating an issue creating changelogs

2. 2026年5月8日 0:40 UTCidentified

   We have identified the issue preventing the changelog creation page from rendering correctly, and we are working on a fix

3. 2026年5月8日 3:02 UTCmonitoring

   We have deployed the fix and are monitoring the results

4. 2026年5月8日 3:18 UTCresolved

   This incident has been resolved.

1. 2026年4月28日 18:38 UTCinvestigating

   We are currently investigating an issue where customers are unable to create branches

2. 2026年4月28日 19:02 UTCidentified

   We have identified the issue and are currently looking into a fix

3. 2026年4月28日 19:31 UTCresolved

   Branches and versions can now be created successfully

1. 2026年4月24日 12:17 UTCinvestigating

   We are currently investigating this issue.

2. 2026年4月24日 12:29 UTCinvestigating

   We are continuing to investigate this issue.

3. 2026年4月24日 13:42 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

4. 2026年4月24日 15:11 UTCresolved

   This incident has been resolved.

1. 2026年4月22日 14:35 UTCinvestigating

   We are currently investigating downtime impacting admin users.

2. 2026年4月22日 14:40 UTCidentified

   An issue has been identified with a third-party provider. A fix is being implemented.

3. 2026年4月22日 14:53 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

4. 2026年4月22日 14:56 UTCmonitoring

   Admin dashboard is recovering and we are continuing to monitor for further issues.

5. 2026年4月22日 14:59 UTCmonitoring

   Fix has been applied and admin requests have recovered. We are continuing to monitor for any further issues.

6. 2026年4月22日 16:36 UTCresolved

   Issues with admin requests have been resolved.

1. 2026年4月22日 2:59 UTCinvestigating

   We are currently investigating performance issues across the platform.

2. 2026年4月22日 3:08 UTCidentified

   We've discovered the issue and have implemented a quick fix

3. 2026年4月22日 3:43 UTCresolved

   Fix has been implemented and performance has stabilized.

1. 2026年4月20日 20:05 UTCidentified

   The admin interface is experiencing slower than expected response times.

2. 2026年4月20日 21:16 UTCmonitoring

   Performance has returned to normal, and we are continuing to monitor.

3. 2026年4月20日 21:20 UTCresolved

   This incident has been resolved.

1. 2026年4月16日 16:07 UTCinvestigating

   We are getting reports of slow performance and 500 errors when loading pages

2. 2026年4月16日 16:57 UTCmonitoring

   We've stabilized our platform and our response times have improved. We are currently monitoring the situation.

3. 2026年4月16日 19:25 UTCresolved

   Performance has stabilized and we are no longer returning errors upon loading pages.