Datacite 장애 기록

Requests to the MDS API are returning errors. We've identified the issue and are working on a fix.

The MDS API is now operational.

We are investigating an issue where new DOI records and DOI record updates may be taking longer than usual to appear in services like Fabrica, Commons, and the REST API. API requests are still being received and processed successfully.

We have identified and resolved the issue. New records and updates should now appear as expected in services like Fabrica, Commons, and the REST API, and queued changes have been processed.

We are investigating degraded performance across our services. This is manifesting as occasional error messages or slow response times for API requests and/or frontend services.

Performance has stabilized. We are continuing to monitor.

Performance of all services has returned to normal levels.

We are investigating intermittent performance issues affecting our services, which may result in slow response times for API requests and frontend applications.

Intermittent performance issues have stabilized. We will continue to monitor.

We are investigating degraded performance across our services. This is manifesting as occasional error messages or slow response times for API requests and/or frontend services.

Our service autoscaling increased resources to handle a higher amount of traffic. This has improved performance, and we are continuing to monitor the situation.

Performance of all services has returned to normal levels.

We're experiencing some timeouts adding to our search index cluster, this will mostly be noticeable in DOIs that are registered might not be findable via certain API endpoints and search/management interfaces. We believe this is a disk space issue due to increased demand and are investigating a remedy. Note no actual data loss as this is just the indexing side, data storage is unaffected and therefore DOI registration is operational.

A temporary fix has been put in place and things should be working now, but we may still have a period of time where some things have not been indexed. We are working on a longer term fix for increased demand and also add any missing items (i.e. DOIs) to our search indices.

We've added a further fix and all search indexing and related services should be operational. We are now monitoring and looking to clean up any missing data in the indices.

All systems are operational and data unavailable in search index has been restored.

We've identified an issue that may affect access to the DataCite homepage at https://datacite.org and the Make Data Count homepage at https://makedatacount.org. Other DataCite services and APIs are not affected. The issue has been isolated, and we are working on a resolution.

The DataCite homepage at https://datacite.org and the Make Data Count homepage at https://makedatacount.org are now available. We will continue to monitor.

The test services that are powered by our OpenSearch cluster are currently not accepting new index writes. This means data is still being saved to the test database, but new records and record updates (for DOIs, members, repositories, etc.) are not appearing in REST API list responses, Fabrica list views, and other places that rely on the OpenSearch index We believe we already know the issue but are currently discussing about how best to resolve. Production services are not affected and are operating as normal.

Test services are restored.

Around 09:30 UTC to 10:00 UTC we had a minor outage within part of our infrastructure that processes various messages on queues (AWS SQS). This mostly meant that messages that usually processes tasks like indexing into our search cluster may have gone missing, this namely would affect things like DOI updates appearing in Commons/Fabrica public APIs. No data loss occurred as all data is saved to our database before background messages are sent to queues for processing. Users may have noticed http 500 (Internal Server Error) responses from our API endpoints during this period of time, but again the data was saved for any DOI create/update requests, it is also safe to re-run any requests that received these errors. The main issue that has arisen is some inconsistencies for a small amount of DOIs in our search index and possibly some URL updates to the global handle service that haven't gone through yet. We are working to clean up any inconsistencies, this should be complete within the next few hours.

We're currently experiencing increased load on our services and are investigating the underlying cause and mitigation that we can take. This mostly appears to be affecting our search cluster, and therefore has a knock-on effect to queries and retrieval through DataCite Commons and some functionality in Fabrica. DOI Registration services should be unaffected, although may experience timeouts viewing content after registration.

Mitigation strategies have been deployed to help with some anonymous traffic that was causing increased load on our search cluster. Services appear to be resuming normal function but with potentially still some degraded performance, we will continue monitoring.

This incident has been resolved.

We're currently experiencing increased load on our services, we're investigating mitigation strategies. This mostly appears to be affecting our search cluster. Registration services should be unaffected, though may experience timeouts viewing content through frontend services / api search interfaces.

We are continuing to investigate this issue.

Services appear stable, we will continue monitoring.

This incident has been resolved.

We've noticed an increase load on our services which is having an affect on returning search results in different places. We are investigating possible causes and attempting to compensate.

Yesterday we made some changes to how traffic was handled by our services, this combined with overall load reducing, everything should be working as normal now.

We are currently experiencing problems with our database services. This is affecting all DOI registration services and some search/indexing service responses. We are investigating as a matter of priority.

We are continuing to investigate this issue.

We have restored connectivity to the database, DOI registration and Search/Index lookups are now operational but marked as degraded while we establish further incident root causes, we will update further once we have a clear picture.

DataCite experienced issues with our database services earlier today. Systems are now operational and we continue to monitor closely. We are investigating whether DOI registration and/or metadata updates were impacted and if so, will contact affected members directly.

All systems are operational and working. We continue to investigate whether there are any data issues and we will contact impacted members if necessary.

We are investigating degraded service across our services. This is manifesting as occasional error messages or slow response times for API requests and/or frontend services. This appears to be due to increased traffic across but we are investigating the source and what capacity we can increase.

The issues currently seen are related to increased load across our services, some specific problems have been identified and resolved. Services are not down, but you may experience intermittent timeouts/errors depending on which service. we are looking at various options to further improve this and increase our capacity.

Yesterday evening (UTC) we introduced a few extra changes to increase support for load in certain areas of our services. We will continue to monitor the situation, services are still slightly degraded but overall should be functional.

Services are operational. We continue to investigate and consider further long term improvements.

We are aware of an issue where individual DOI pages on DataCite commons are returning a 404 Error and are investigating the root cause. Individual DOI metadata and related information can still be accessed via the REST and GraphQL API endpoints.

We are continuing to investigate the underlying cause of the errors appearing on individual DOI pages on the DataCite Commons service. Individual DOI metadata can be accessed via REST and GraphQL - for more information, please see our Support Pages: https://support.datacite.org/docs/api and https://support.datacite.org/docs/datacite-graphql-api-guide

We have identified a problem and individual DOI detail pages should be available again within commons. We however have been receiving and influx in high usage from some automated sources, so we will continue to monitor for additional effects.

All systems remain operational, overall health of services appears normal. Incident Resolved.

We are investigating instability affecting DOI registration and other services. This appears to be related to increased service load and we are scaling services to compensate.

We believe we have identified the source of the increased load, and are working on further mitigation

Mitigation has been applied, and the services are starting to stabilise. We will continue to monitor the situation

The applied mitigation has allowed services to stabilise back to normal operating conditions. Services are still scaled up to prevent further issue.

We are investigating an issue with response times being slow for DOI registration. This appears to be related to increased service load and we are scaling services to compensate.

Services have been scaled to cope with load, we are monitoring the situation.

All services appear normal now, closing incident. Services are still marginally scaled up to prevent issues.

We've received reports and internal monitoring that is showing errors when attempting to register DOIs, this appears to not affect all registrations but instead intermittent performance issues. We are currently investigating.

We increased our capacity to cope with an increased load on our services. We are still monitoring at this stage but affected systems appear operational.

All services appear normal now, closing incident. Services are still marginally scaled up to prevent issues.

increased load has been creeping up for past couple of hours, some automatic and manual mitigation has already taken place, but we are investigating further.

The initial capacity increases looks to have provided the needed support for services, there may be some slowness for indexing of DOIs which may take longer than the usual few minutes, a backlog queue is currently being processed after the load spike. We are monitoring for further problems.

All services appear normal.

We have identified that our services are experiencing high load. It is primarily affecting MDS however REST API for DOI registrations may also be affected, consequently our frontend application Fabrica is potentially affected. We've already compensated with higher number of servers running in our production cluster and our now monitoring the situation.

Service appears stable for few hours now, incident resolved but will continue monitoring.