49 Hcaptcha incidents · Mart 2020 — official updates, affected components, duration and resolution details.
Bazı bölgelerdeki kenarımızda beklenen gecikmeden daha yüksek gözlemliyoruz, öncelikle siteverify taleplerinin% 0,075'ini etkiliyoruz. Etkilenen trafik şimdi yeniden yönlendiriliyor ve kısa bir süre sonra gecikme bekliyoruz.
Bir düzeltme uygulandı ve sonuçları takip ediyoruz.
Bu olay çözüldü.
Resmî olay güncellemesinden otomatik olarak çevrilmiştir.
Birkaç ağ üzerindeki 429 oranları yaklaşık 60 saniye arttı. Bu konu şimdi çözüldü.
Resmî olay güncellemesinden otomatik olarak çevrilmiştir.
Between 14:15 UTC and 14:22 UTC, some loads of API assets returned 403s in some regions. This incident has now been resolved.
We are monitoring sporadic latency impact related to network-level issues at an upstream WAF provider: https://www.cloudflarestatus.com/incidents/dz87vqwvl8wr This appeared to primarily affect some LATAM and EUR region POPs.
This notice is purely informational. There was zero impact to hCaptcha from the AWS us-east-1 outage that took down many other online services this week.
Passkey support is being updated; error rates for logins or 2FA verification on accounts using passkeys may increase for approximately 10 minutes.
Maintenance is now complete.
Due to an issue at an upstream edge CDN, P99 times in several regions, primarily GIG and GRU, were elevated between 14:15 UTC and 14:45 UTC.
An upstream WAF provider is experiencing increased latency in the IAD region, which appears to be primarily affecting connections from AWS us-east to our API endpoints. Most traffic has been rerouted to nearby regions, so siteverify endpoints from servers connecting from AWS us-east regions that do not use our PrivateLink or Direct Routing options may see 20-50ms of increased latency until IAD is restored. Impact appears to be less than 0.005% of total requests, but may be concentrated on some network links.
We are no longer seeing elevated latency in IAD.
Due to an incident at Google, we are observing increased error rates on login authentication. Use GitHub SSO or contact [email protected] if you need assistance with alternate methods. You may track status here: https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1SsW#2c2sBHWU84yPDJ8y1ar4
Error rates for Google SSO appear to have returned to normal.
Due to an issue at an upstream mail gateway, some inbound support emails may have been delayed or rejected earlier today. This did not affect outbound email, and the issue has now been resolved.
A fix has been implemented and we are monitoring the results.
This incident has been resolved. As a reminder, please confirm that you are loading the JS SDK only via the recommended host: https://js.hcaptcha.com/1/api.js
Due to an incident at an upstream CDN provider starting at 18:00 UTC, we are observing elevated P99 latency (+100-200ms) on a minority of traffic in the Eastern US and Western Europe regions, caused by some traffic being rerouted to more distant POPs. We have put overrides in place and will continue to monitor performance; there is no impact to availability and no action is required.
P99 latency has returned to norm as of 20:05 UTC.
At Mar 22 01:21:56 2024 GMT, some users of dedicated mainland China region endpoints experienced an expired certificate error on some endpoints. The root cause has been identified and addressed. Global traffic was unaffected.
hCaptcha APIs use several SSL certificate authorities, maintaining both primary and backup certificates; our CAA record is authoritative. We also automatically rotate certificates every three months as part of our security best practices. We received several reports today from customers running servers with outdated root CA entries. They either needed to update these after our most recent automatic certificate rotation, or had locked their validation for our endpoints to a specific certificate chain rather than relying on CA validation and our CAA records. Please ensure your servers calling the siteverify endpoint have an updated root CA store. This is an important security practice, as root CAs are occasionally compromised and removed from OS vendors' stores. Similarly, if you would like to enforce additional restrictions on validating our TLS certificates, please rely on the CAA record rather than hard-coding a specific intermediate chain.
Some sites consuming hCaptcha risk scores briefly experienced an elevated score distribution, affecting approximately 3% of traffic. This incident was fully resolved by 01:28 UTC. This issue and its root cause have now been addressed, and adjustments scheduled to prevent any similar incidents in the future.
Token verification endpoint error rates increased for several minutes on approximately 0.1% of sites during a token-related update. This was most commonly visible in an increased probability of a sitekey mismatch error.
Error rates for some users located in mainland China and using dedicated regional routing endpoints were elevated during a rerouting event triggered by failures at an upstream provider. Mainland China users accessing standard global and first-party endpoints were unaffected.
Due to an upstream caching issue, some endpoints for mainland China users failed to update during certificate rotation and briefly served requests using an expired TLS certificate. This affected only users of dedicated China endpoints; China traffic using global endpoints was unaffected.
Challenge rates were briefly elevated for some traffic segments, with a duration of between 0 and 20 minutes depending on region. This issue has now been resolved.
A behavior change in an upstream deployment workflow caused some paths related to WAF customers' asset host paths to temporarily de-route. While most requests (>99%) were re-routed automatically with no user-visible impact, some WAF customer sites using custom asset hosts failed to load images for visual challenges. This issue has now been resolved.
