503 错误和文档可用性
Outage重大事件
受影响的组件
Admin DashboardReadMe Hubs
- investigating
我们正在调查一个问题 造成503个错误 和超时的加载文件页.
- investigating
我们正在继续调查这一问题.
- identified
这一问题已经确定,一个解决办法正在实施之中.
- monitoring
一项措施已经执行,我们正在监测结果.
- resolved
这一事件已经得到解决.
自动翻译自官方事件更新。
57 Readme incidents · 2024年11月 — official updates, affected components, duration and resolution details.
我们正在调查一个问题 造成503个错误 和超时的加载文件页.
我们正在继续调查这一问题.
这一问题已经确定,一个解决办法正在实施之中.
一项措施已经执行,我们正在监测结果.
这一事件已经得到解决.
自动翻译自官方事件更新。
我们正在调查为什么404页
我们已查明问题,正在设法解决问题
修复装置已经部署,我们目前正在部署操作服务器。 可能有些被缓存的404, 所以请通知支持您是否还遇到问题
各项目应再次展示。 如果您还遇到任何404个错误, 请联系支持@ readme.com .
自动翻译自官方事件更新。
我们正在调查影响中心点的 高纬度和缓慢反应时间 积极开展工作,查明根源,恢复正常服务.
已经实施了一个固定,我们正在监测结果。 性能应该恢复正常.
我们已经找出了其他的促成因素,并正在积极努力加以纠正.
我们正在继续监测任何其他问题.
这一事件已经得到解决.
自动翻译自官方事件更新。
我们的管理员界面 正在经历一个中断。 最终用户访问您的 ReadMe 开发器枢纽的能力没有受到影响 .
我们解决了这种情况。 接下来的几分钟里可能会有间歇性迟缓.
这一事件已经得到解决.
自动翻译自官方事件更新。
我们目前在整个客户中心搜索结果缓慢
我们重新索引了搜索结果,业绩有所改善。 请提供电子邮件支持@readme.com 如果您还在搜索中看到问题
自动翻译自官方事件更新。
由于第三方基础设施问题,业绩下降。 我们正在监测这一局势并努力维持我们最终的稳定.
这一事件已经得到解决.
自动翻译自官方事件更新。
We are currently investigating an incident related to API Reference pages.
This incident was resolved and normal site function has been resolved
We're actively investigating this issue.
The issue has been identified and a fix is being implemented.
Systems are getting back up. We're continuing to work on a permanent fix for this issue.
A fix has been implemented and we are monitoring the results.
We are continuing to monitor for any further issues.
We are receiving reports of degraded performance and are actively investigating. We will provide updates as more information becomes available.
Systems appear stable and we are actively rolling out a series of fixes. We will continue to monitor and provide updates
Systems remain stable. Improvements and fixes have been rolled out and we are continuing to monitor.
Systems remain stable, and we are continuing to monitor the situation. Bi-directional sync was on a maintenance job from 9:50 - 11:59 AM (ET) and is now operational.
Systems remain stable and our team continues to deploy targeted fixes. Work is ongoing to address degraded read performance across the system, including serving all 404s from cache. We’ll be actively monitoring over the weekend, and we will provide a full RCA next week.
Our systems have been stable since Friday, May 29th. We're monitoring closely and improving spike detection to prevent recurrence and adding proactive capacity monitoring to detect admin dashboard degradation before it impacts customers. Full root cause analysis coming soon. Thanks for your patience and understanding.
We are currently investigating this issue.
The issue has been identified and a fix is being implemented.
We're slowly recovering and monitoring the situation.
We are monitoring the issue.
Updated 6/3: A routine configuration update coincided with the downtime window, which led us to initially identify it as the cause. Further investigation confirmed the two were unrelated. See the postmortem for full details.
We are seeing some slow performance, possibly 500 errors. investigating
We are continuing to investigate this issue.
Performance has improved, but we're still investigating the issue.
Performance and loading is still being affected, and we're in the process of rolling out fixes
This incident has been resolved.
### What Happened Beginning Tuesday, May 26, 2026, customers were experiencing slow loading and 503 errors across ReadMe-hosted docs and the admin dashboard. The outage was intermittent but recurring, with the worst periods hitting during business hours when traffic spiked. ### Root Cause An internal data backup process was generating excessive I/O on our storage layer. Under normal traffic conditions, this additional load was manageable. But when it coincided with peak customer traffic and elevated bot activity, total I/O demand exceeded system capacity, causing cascading request timeouts. The maintenance process ran on a recurring schedule, which is why the degradation followed a predictable pattern of spikes throughout each day. Separately, a surge in bot traffic to non-existent pages \(404s\) amplified the problem because those requests were not being served from cache. ### Resolution **Immediate fix:** We identified and disabled the maintenance process causing the excess I/O load. Within three hours, storage utilization returned to normal levels and remained stable. **Additional improvements shipped during the incident:** * Expanded caching across multiple layers, significantly reducing load on backend storage * Hardened 404 handling to serve error pages from cache instead of hitting the backend * Implemented rate limiting and IP-based protections against abusive bot traffic * Optimized several high-traffic API endpoints to reduce redundant backend calls * Added new monitoring and alerting for storage I/O thresholds ### Timeline May 26 - June 1, 2026 | Time | Status | Details | | --- | --- | --- | | Mon 5/26, 6:39 AM PDT | Investigating | Issue reported | | Mon 5/26, 7:26 AM PDT | Monitoring | Fix implemented, monitoring results | | Mon 5/26, 8:37 AM PDT | Resolved | Admin hub incident resolved | | Mon 5/26, 10:41 AM PDT | Investigating | Slow performance across customer hubs | | Mon 5/26, 10:49 AM PDT | Monitoring | Quick fix applied, investigating thorough fix | | Mon 5/26, 8:54 PM PDT | Resolved | Customer hub incident resolved | | Tue 5/27, 6:51 AM PDT | Investigating | Issue reported | | Tue 5/27, 7:52 AM PDT | Identified | Fix in progress | | Tue 5/27, 8:57 AM PDT | Monitoring | Slowly recovering | | Tue 5/27, 10:17 AM PDT | Update | Updated 6/3: A routine configuration update coincided with the downtime window, which led us to initially identify it as the cause. Further investigation confirmed the two were unrelated. See root cause and resolution above. | | Tue 5/27, 1:54 PM PDT | Update | Performance and loading still affected, rolling out fixes | | Tue 5/27, 3:37 PM PDT | Resolved | Incident resolved | | Wed 5/28, 6:40 AM PDT | Investigating | Issue reported | | Wed 5/28, 6:43 AM PDT | Identified | Fix being implemented | | Wed 5/28, 7:35 AM PDT | Update | Systems coming back up, working on permanent fix | | Wed 5/28, 8:27 AM PDT | Monitoring | Fix implemented, monitoring results | | Wed 5/28, 10:53 AM PDT | Update | Reports of degraded performance, actively investigating | | Wed 5/28, 1:04 PM PDT | Update | Systems appear stable, rolling out fixes | | Wed 5/28, 6:49 PM PDT | Monitoring | Improvements and fixes deployed, continuing to monitor | | Thu 5/29, 10:03 AM PDT | Monitoring | Systems stable. Bi-directional sync maintenance 9:50–11:59 AM ET. | | Thu 5/29, 11:36 AM PDT | Monitoring | Deploying targeted fixes. Serving all 404s from cache. Degraded read performance. | | Fri 5/30 | Monitoring | Monitoring continues | | Sat 5/31 | Monitoring | Weekend monitoring | | Mon 6/1, 11:16 AM PDT | Resolved | Root cause identified and resolved. Systems stable since Thursday, May 29. | ### Path Forward We are using this incident to make lasting improvements to reliability and incident response: * **Storage capacity and isolation:** Restructuring how background processes interact with production storage to eliminate contention under load. * **Caching and performance:** The caching improvements shipped during the incident are permanent. We are continuing to expand cache coverage across additional endpoints and page types. * **Bot and traffic protection:** Strengthening rate limiting and abuse detection to prevent bot traffic from contributing to backend load. * **Monitoring and alerting:** Adding proactive capacity monitoring with earlier thresholds so the team can intervene before customers are affected. * **Incident response:** Improving our internal processes for faster escalation and more frequent status page updates during multi-day incidents. ### Final Note During the incident, we posted an update referencing a platform update. That was our initial hypothesis based on timing. Further investigation confirmed it was unrelated. The change in question was a routine, isolated configuration update and had no impact on the outage or any other customers. We should have waited for confirmation before publishing it, and we're tightening our internal process for status page updates as a result. We know how critical your documentation is to your customers, and this level of disruption is not acceptable. We have already shipped meaningful improvements to prevent recurrence, and the work outlined above will continue through the coming weeks. If you have questions, reach out to your account team or contact [[email protected]](mailto:[email protected]).
We are experiencing slow performance across customer hubs
We have identified and quickly fixed the issue and are currently investigating a more thorough fix
This incident has been resolved.
We are currently investigating this issue.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We are investigating slowness across hubs.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We are currently investigating this issue.
The issue has been identified and a fix is being implemented.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We are currently investigating an issue creating changelogs
We have identified the issue preventing the changelog creation page from rendering correctly, and we are working on a fix
We have deployed the fix and are monitoring the results
This incident has been resolved.
We are currently investigating an issue where customers are unable to create branches
We have identified the issue and are currently looking into a fix
Branches and versions can now be created successfully
We are currently investigating this issue.
We are continuing to investigate this issue.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We are currently investigating downtime impacting admin users.
An issue has been identified with a third-party provider. A fix is being implemented.
A fix has been implemented and we are monitoring the results.
Admin dashboard is recovering and we are continuing to monitor for further issues.
Fix has been applied and admin requests have recovered. We are continuing to monitor for any further issues.
Issues with admin requests have been resolved.
We are currently investigating performance issues across the platform.
We've discovered the issue and have implemented a quick fix
Fix has been implemented and performance has stabilized.
The admin interface is experiencing slower than expected response times.
Performance has returned to normal, and we are continuing to monitor.
This incident has been resolved.
We are getting reports of slow performance and 500 errors when loading pages
We've stabilized our platform and our response times have improved. We are currently monitoring the situation.
Performance has stabilized and we are no longer returning errors upon loading pages.