13 Crowdin incidents · 2022年11月 — official updates, affected components, duration and resolution details.
决议 我们解决了一个状态页报告问题 并删除了一个事件记录 错误地显示Crowdin Translate的停机时间为7小时53分 服务一直全面运作——没有实际故障时间或用户影响. 发生了什么事? 假"主出道"状态被自动化通知故障所触发. 两个相冲突的更新(DOWN和UP)正好同时到达. 由于赛事条件,系统处理下行提示但忽略了同时UP通知,使得状态页面被卡入了错误状态.
自动翻译自官方事件更新。
We noticed unexpected service behavior, mainly crowdin.com is impacted. Our developers have already started investigating the case, and you will be posted on further updates.
The issue has been identified, and a solution is being implemented. As a temporary measure, users may encounter a CAPTCHA when navigating between pages.
All services are back to the normal state. We will keep the incident open for the next 10 minutes to provide you with a final confirmation.
After thorough monitoring, we can confirm the service is fully back to normal. The incident has now been marked as "Resolved". We will continue to monitor Crowdin's health 24/7/365. Thank you!
We identified degraded performance among organizations hosted in the EU data center. All other Crowdin components are fully accessible. We will keep you posted on any further updates.
The solution was delivered several minutes after the incident was reported, we were monitoring the performance of EU data center organizations and confirmed that it got back to normal
Between 13:40 and 14:10 UTC, a subset of Crowdin Enterprise users experienced server errors when attempting to access projects via the web interface (UI). The issue affected users with specific project configurations. Access via the API remained operational. The service was fully available for translators, though. Our team identified and deployed a fix for the underlying server issue. The problem is resolved, and project access is fully restored. If you were affected by this issue, please refresh your browser page to ensure full functionality. We apologize for the inconvenience.
We have noticed that builds are failing or taking longer than usual due to an issue with the cloud provider. This issue is not global; we are continuing to monitor the status. We will update the status once we confirm that the builds are working properly.
Almost right after the incident was raised, the project builds started working as expected. After conducting various tests, we can confirm that the builds have returned to normal, and the incident can be considered resolved.
As you may be aware, Amazon AWS has experienced a major service disruption. The initial reports were logged at 8:11 am UTC, and within an hour, we noticed that translations export via UI and API started to fail almost instantly. Other components of Crowdin (dashboard, translation editor) were not impacted. We hope that it did not affect your work significantly. Our ticketing system got impacted too, and we couldn't react to the live chat efficiently. Sorry about that. All services are operational currently. We will keep monitoring the situation and update the incident once we confirm that every Crowdin component works as usual.
After nearly 3 hours of monitoring, we confirm that translations are exported on time, and all connectors are fully accessible. We consider the case as resolved.
Our team has spotted a performance downgrade at Crowdin Enterprise. We are continuing to monitor the situation and work on a resolution.
Everything appears to be working properly. We will continue to monitor the system.
After additional monitoring, we can confirm that the service is fully operational
We noticed unusual behavior on the project dashboard, and the API is working more slowly than usual. It's being investigated further by our development team.
All services are fully operational
We experienced a temporary downtime of our Downloads and redirect service, which impacted the ability of some users to download translations.
We noticed that most of the apps are malfunctioning right now, other services (e.g. translation editor, API) function as usual. We are investigating it already
A fix has been implemented, we will monitor the status for a while
The store is fully functioning again, along with all its apps 🙌
We noticed unusual performance on Crowdin, already started investigating it
The service got back to normal, but we keep monitoring the results.
This accident has been fully resolved, after monitoring we confirmed that all functionality returned to normal
Between 13:19 and 13:36 UTC, we experienced performance degradation in crowdin.com and Crowdin Enterprise. The issue has been resolved, and the services are now operating normally.
When your product or service isn’t functioning as expected, let your customers know by creating an incident. Communicate early, even if you don’t know exactly what’s going on.
As you continue to work through the incident, update your customers frequently.
Let your users know once a fix is in place, and keep communication clear and precise.
Empathize with those affected and let them know everything is operating as normal.