52 Globalsign incidents · 2023年12月 — official updates, affected components, duration and resolution details.
アトラス API への接続の中断を想定しています。 スコープを確認した直後に更新を投稿します
全てのサービスが運用されていることを確認しました。 外部監視では、停電は検知されませんが、一部の地理的にはパケットの損失が短いことがあります.
公式のインシデント更新を自動翻訳しています。
We’re currently investigating reports of a service interruption with GCC Certificate Issuance. The root cause is currently unknown. We understand the issues this may be causing and are working hard to resolve the problem as soon as possible. Occurred at: Aug 13, 08:21 UTC ■Scope of impact - GCC User Portal - Certificate Issuance - SSL API Certificate Issuance - SSL (Access to Domain Validation URL) - ePKI API Certificate Issuance - EPKI (Access to certificate acquisition URL) - JCAN User Portal - JCAN Certificate Issuance - JCAN API Certificate Issuance - JCAN (Access to certificate acquisition URL) [Note] There is no impact on the use of issued certificates or revocation confirmation.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
One of our upstream ISPs experienced an issue in their network serving our London data centre at approximately 19:34 UTC today which caused a brief interruption to connectivity to the Atlas APIs (DSS, Certificate Issuance, TSA). Some customers may have noticed timeouts or connection errors for a period of 1-2 minutes. This issue has been resolved.
We are experiencing problems with the logging cluster that is used for usage reporting in Atlas - directly via the APIs, and in the portal for DSS users. Users may receive errors when checking the APIs or may receive responses with missing usage. The team are working to resolve the issue ASAP.
The core issue has been resolved however we are continuing to closely monitor the situation and ensure the usage reporting cluster remains stable.
This incident has been resolved.
We are aware of an issue starting at around 5:17 UTC on Saturday 7th June which caused timeouts and errors for connections to the Atlas HVCA and DSS APIs with total disruption lasting around 12 minutes. This appears to have been related to upstream connectivity problems, however the Infrastructure team are investigating and will be planning appropriate resilience improvements.
We are aware of some further issues relating to the display of usage in the Atlas portal, affecting a small number of customers. The development team are actively working on a fix for affected customers. Certificate issuance and use of the APIs themselves is not affected.
Our development team are currently testing a fix for the issue. We will provide further updates as this is confirmed ready for production release.
The team have successfully deployed the fix and we are closely monitoring to ensure that the issue is resolved for all affected customers.
Following an extended period of close monitoring, the team have confirmed that the issue is resolved.
We just experienced a high spike in errors on the Atlas APIs (HVCA, DSS, TSA). The issue lasted for a few minutes between roughly 12:25 and 12:35 UTC. The issue is no longer occurring but the Infrastructure team are investigating the root cause.
The Infrastructure team have identified the root cause and are monitoring the situation.
The issue has been confirmed resolved
We are aware of incident which caused some intermittent periods of errors and timeouts between 04:00 and 04:45 UTC this morning. The Infrastructure team have identified and resolved the issue.
We are aware that following a portal update, some customers may see incorrect usage data displayed. Our development team are currently validating the fix and we expect this to be deployed on Monday. Certificate issuance and use of the APIs themselves is not affected.
The development team are deploying a fix which should restore proper visualisation of usage for some sets of customers. A fix for remaining affected customers is expected to be deployed tomorrow.
The development team are still working on resolving the issue. We will continue to post updates as they become available
The issue affecting usage display in the Atlas portal has been resolved. All fixes have been successfully deployed, and the service is operating normally.
We have seen a brief issue with issuance on GCC causing some certificate requests to fail or be delayed over the last 15-20 minutes. The team are working to ensure the problem is resolved.
The issue has been confirmed as resolved.
We identified an issue that affected the processing of some OCSP requests. Mitigation steps have been implemented, and the service is currently operating normally. We are continuing to monitor the situation to ensure ongoing stability.
We are continuing to work with our CDN providers to troubleshoot the issue before restoring full CDN redundancy. To provide some more details - the first reports of the issue were received on 4th March, with further reports on 9th-12th March. We were able to recreate and identify the issue on 16th March, with a mitigation put in place around 12:25 UTC on 16th March.
Our third party CDN provider has confirmed that a fix has been deployed and we are in the process of testing this before planning to restore full CDN redundancy tomorrow at around 10:00 UTC. Our team will be monitoring closely throughout this period.
Full CDN redundancy has been restored as planned, and behaviour is under close monitoring. If no issues are encountered, we will close this issue.
No further issues have been encountered
The team have seen an increase in error rate on the Atlas (HVCA and DSS) APIs. This seems to be related to an internal failover event but showed a brief failure rate of around 4% on /login calls. The team are confirming the root cause and are monitoring the situation, although it appears to be stable.
The issue has been confirmed related to a failover event on one of our database clusters and has been confirmed as resolved.
We are aware of an issue which is causing sporadic failures when calling the /stats endpoint on the Atlas HVCA API. The team is working on this and will restore full functionality as soon as we can. Certificate issuance and other features are unaffected.
The issue was resolved last night UK time, and full functionality has now been confirmed restored.
We are experiencing an issue with Japan certified timestamps. This relates to a problem at our upstream provider. We are monitoring the situation and will provide updates as we get them. This also affects customers obtaining Japan certified timestamps through the Atlas TSA and DSS services
The issue looks to have been resolved, however we are waiting for an update from our provider to confirm that everything is stable.
Our supplier has confirmed the issue is resolved.
We are seeing a higher than normal failure rate of calls to the /stats endpoints on the HVCA (Atlas Certificate Management API). This relates to problems in the logging cluster which the team are currently working to resolve. Issuance and other core services are not affected.
The team are continuing to work on resolving the issues with the logging cluster and hope to have the problem resolved soon.
The issue has been resolved and we've not seen further errors.
We are aware of an issue affecting TLS certificates issued on 1st December which can cause them to not be trusted by some browsers. This is caused by the trust status of some of the 2027 CT Logs. We are urgently removing affected logs from our configuration. Re-issuing affected certificates will resolve the problem.
The issue affecting TLS certificates issued on or after December 1 has been resolved. All affected CT logs have been removed from our configuration, and reissuing certificates restores trust in all major browsers. If you have not yet reissued your certificate, please do so to ensure proper functionality. Services are operating normally.
We have detected an issue with the Atlas portal which may cause errors when trying to access some sections. This is caused by the browser caching stale code - if you experience this issue, please do a hard reload ("Ctrl-F5" or "Ctrl-Shift-R") or clear your browser cache, which should resolve the problem.
We are no longer seeing any failed requests in our logs relating to this issue. If you do experience any problems with the portal, please try the actions mentioned above which should resolve it.
We are investigating an issue with the Atlas portal which is causing it to hang after login. We will post further information as we find the root cause.
The issue has been resolved.
We’re currently investigating reports of a potential service interruption with GCC Certificate Issuance. The root cause is currently unknown. We understand the issues this may be causing and are working hard to resolve the problem as soon as possible.
Service for GCC Certificate Issuance has been restored, and customers should now be able to issue certificates normally. We are closely monitoring the system to ensure stability. The root cause is still under investigation, and we will provide updates as soon as more information is available.
The GCC Certificate Issuance service has been fully restored and is operating normally.
We are currently investigating a slightly increased error rate on the Atlas TSA service, resulting in customers receiving HTTP 503 errors in a small number of cases. Retrying failed requests should result in a successful response. We will provide updates as we identify the cause.
We are continuing to investigate this issue. The problem is limited to customers using Atlas Timestamping with the Japanese Qualified Timestamp service provided by our partner in Japan. However, we are yet to identify the root cause for the high failure rate of these timestamps.
We have found a way to prevent the errors for affected customers and have implemented this while we confirm and resolve the root cause. We will keep a close eye on error levels on the service over the next 24 hours.
No further errors have been recorded.