IQSS/6497 migrate dataset api

doc/release-notes/6497-migrate-api.md

@@ -0,0 +1,17 @@
+# Release Highlights
+
+### Dataset Migration API (Experimental)
+
+Datasets can now imported/updated following the format of an OAI-ORE export (RDA-conformant Bags), allowing for not only easier migration from one Dataverse installation to another, but also for better support of import from other systems. This experimental endpoint also allows keeping the existing persistent identifier (where the authority and shoulder match those for which the software is configured) and publication dates. This endpoint also allows for the update of terms metadata (#5899).
+
+This development was supported by the [Research Data Alliance](https://rd-alliance.org) and follows the recommendations from the [Research Data Repository Interoperability Working Group](http://dx.doi.org/10.15497/RDA00025).
+
+### Additional Upgrade Steps
+
+Update Solr Schema
+
+- copy schema_dv_mdb_fields.xml and schema_dv_mdb_copies.xml to solr server, for example into /usr/local/solr/solr-8.8.1/server/solr/collection1/conf/ directory
+
+- Restart Solr, or tell Solr to reload its configuration:
+
+   `curl "http://localhost:8983/solr/admin/cores?action=RELOAD&core=collection1"`

doc/sphinx-guides/source/_static/api/dataset-migrate.jsonld

@@ -0,0 +1,42 @@
+{
+"citation:Depositor": "Admin, Dataverse",
+"Title": "Test Dataset",
+"Subject": "Computer and Information Science",
+"Creator": {
+  "author:Name": "Admin, Dataverse",
+  "author:Affiliation": "GDCC"
+},
+"Deposit Date": "2020-10-08",
+"citation:Distributor": {
+  "distributor:Name": "Demo Dataverse Repository",
+  "distributor:Affiliation": "Dataverse Community",
+  "distributor:Abbreviation": "GDCC",
+  "distributor:URL": "https://dataverse.org/global-dataverse-community-consortium"
+},
+"citation:Contact": {
+"datasetContact:Name": "Admin, Dataverse",
+"datasetContact:Affiliation": "GDCC",
+"datasetContact:E-mail": "admin@demo.dataverse.org"
+},
+"citation:Description": {
+  "dsDescription:Text": "A short description"
+},
+"@id": "doi:10.33564/FK27U7YBV",
+"schema:version": "1.0",
+"schema:license": "https://creativecommons.org/publicdomain/zero/1.0/",
+"dvcore:fileTermsOfAccess": {
+"dvcore:fileRequestAccess": false
+},
+"@context": {
+  "Creator": "http://purl.org/dc/terms/creator",
+  "Deposit Date": "http://purl.org/dc/terms/dateSubmitted",
+  "Subject": "http://purl.org/dc/terms/subject",
+  "Title": "http://purl.org/dc/terms/title",
+  "author": "https://dataverse.org/schema/citation/author#",
+  "citation": "https://dataverse.org/schema/citation/",
+  "datasetContact": "https://dataverse.org/schema/citation/datasetContact#",
+  "distributor": "https://dataverse.org/schema/citation/distributor#",
+  "dsDescription": "https://dataverse.org/schema/citation/dsDescription#",
+  "dvcore": "https://dataverse.org/schema/core#",
+  "schema": "http://schema.org/"
+}}

doc/sphinx-guides/source/admin/dashboard.rst

@@ -22,7 +22,7 @@ This dashboard tool allows you to define sets of local datasets to make availabl
 Metadata Export
 ---------------
 
-This part of the Dashboard is simply a reminder message that metadata export happens through the Dataverse Software API. See the :doc:`metadataexport` section and the :doc:`/api/native-api` section of the API Guide for more details.
+This part of the Dashboard is simply a reminder message that metadata export happens through the Dataverse Software API. See the :doc:`/admin/metadataexport` section and the :doc:`/api/native-api` section of the API Guide for more details.
 
 Users
 -----

doc/sphinx-guides/source/developers/dataset-migration-api.rst

@@ -0,0 +1,51 @@
+Dataset Migration API
+=====================
+
+The Dataverse software includes several ways to add Datasets originally created elsewhere (not to mention Harvesting capabilities). These include the Sword API (see the :doc:`/api/sword` guide) and the /dataverses/{id}/datasets/:import methods (json and ddi) (see the :doc:`/api/native-api` guide).
+
+This experimental migration API offers an additional option with some potential advantages:
+
+* metadata can be specified using the json-ld format used in the OAI-ORE metadata export
+* existing PIDs can be maintained (currently limited to the case where the PID can be managed by the Dataverse software, e.g. where the authority and shoulder match those the software is configured for)
+* adding files can be done via the standard APIs, including using direct-upload to S3
+* the dataset can be published keeping the original publication date
+
+This API consists of 2 calls: one to create an initial Dataset version, and one to publish the version with a specified publication date. 
+These calls can be used in concert with other API calls to add files, update metadata for additional versions, etc.   
+
+
+Start Migrating a Dataset into a Dataverse Collection
+-----------------------------------------------------
+
+.. note:: This action requires a Dataverse installation account with superuser permissions.
+
+To import a dataset with an existing persistent identifier (PID), the provided json-ld metadata should include it.
+
+.. code-block:: bash
+
+  export API_TOKEN=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
+  export SERVER_URL=https://demo.dataverse.org
+  export DATAVERSE_ID=root
+  export PERSISTENT_IDENTIFIER=doi:10.5072/FK27U7YBV
+
+  curl -H X-Dataverse-key:$API_TOKEN -X POST $SERVER_URL/api/dataverses/$DATAVERSE_ID/datasets/:startmigration --upload-file dataset-migrate.jsonld
+
+An example jsonld file is available at :download:`dataset-migrate.jsonld <../_static/api/dataset-migrate.jsonld>` 
+
+
+Publish a Migrated Dataset
+--------------------------
+
+The call above creates a Dataset. Once it is created, other APIs can be used to add files, add additional metadata, etc. When a version is complete, the following call can be used to publish it with its original publication date.
+
+.. note:: This action requires a Dataverse installation account with superuser permissions.
+
+.. code-block:: bash
+
+  export API_TOKEN=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
+  export PERSISTENT_IDENTIFIER=doi:10.5072/FK27U7YBV
+  export SERVER_URL=https://demo.dataverse.org
+
+  curl -H 'Content-Type: application/jsonld' -H X-Dataverse-key:$API_TOKEN -X POST -d '{"schema:datePublished": "2020-10-26","@context":{ "schema":"http://schema.org/"}}' "$SERVER_URL/api/datasets/{id}/actions/:releasemigrated"
+
+datePublished is the only metadata supported in this call.

doc/sphinx-guides/source/developers/index.rst

@@ -36,4 +36,5 @@ Developer Guide
    aux-file-support
    s3-direct-upload-api
    dataset-semantic-metadata-api
+   dataset-migration-api 
    workflows

src/main/java/edu/harvard/iq/dataverse/api/Datasets.java

@@ -51,6 +51,7 @@
 import edu.harvard.iq.dataverse.engine.command.impl.DeleteDatasetLinkingDataverseCommand;
 import edu.harvard.iq.dataverse.engine.command.impl.DeletePrivateUrlCommand;
 import edu.harvard.iq.dataverse.engine.command.impl.DestroyDatasetCommand;
+import edu.harvard.iq.dataverse.engine.command.impl.FinalizeDatasetPublicationCommand;
 import edu.harvard.iq.dataverse.engine.command.impl.GetDatasetCommand;
 import edu.harvard.iq.dataverse.engine.command.impl.GetSpecificPublishedDatasetVersionCommand;
 import edu.harvard.iq.dataverse.engine.command.impl.GetDraftDatasetVersionCommand;
@@ -77,7 +78,6 @@
 import edu.harvard.iq.dataverse.ingest.IngestServiceBean;
 import edu.harvard.iq.dataverse.privateurl.PrivateUrl;
 import edu.harvard.iq.dataverse.S3PackageImporter;
-import edu.harvard.iq.dataverse.api.AbstractApiBean.WrappedResponse;
 import edu.harvard.iq.dataverse.api.dto.RoleAssignmentDTO;
 import edu.harvard.iq.dataverse.batch.util.LoggingUtil;
 import edu.harvard.iq.dataverse.dataaccess.DataAccess;
@@ -104,16 +104,23 @@
 import edu.harvard.iq.dataverse.util.SystemConfig;
 import edu.harvard.iq.dataverse.util.bagit.OREMap;
 import edu.harvard.iq.dataverse.util.json.JSONLDUtil;
+import edu.harvard.iq.dataverse.util.json.JsonLDTerm;
 import edu.harvard.iq.dataverse.util.json.JsonParseException;
 import edu.harvard.iq.dataverse.search.IndexServiceBean;
 import static edu.harvard.iq.dataverse.util.json.JsonPrinter.*;
 import static edu.harvard.iq.dataverse.util.json.NullSafeJsonBuilder.jsonObjectBuilder;
 import edu.harvard.iq.dataverse.util.json.NullSafeJsonBuilder;
+import edu.harvard.iq.dataverse.workflow.Workflow;
+import edu.harvard.iq.dataverse.workflow.WorkflowContext;
+import edu.harvard.iq.dataverse.workflow.WorkflowServiceBean;
+import edu.harvard.iq.dataverse.workflow.WorkflowContext.TriggerType;
+
 import java.io.IOException;
 import java.io.InputStream;
 import java.io.StringReader;
 import java.sql.Timestamp;
 import java.text.MessageFormat;
+import java.time.LocalDateTime;
 import java.time.ZoneId;
 import java.time.format.DateTimeFormatter;
 import java.util.ArrayList;
@@ -123,6 +130,7 @@
 import java.util.LinkedList;
 import java.util.List;
 import java.util.Map;
+import java.util.Optional;
 import java.util.Map.Entry;
 import java.util.Set;
 import java.util.logging.Level;
@@ -140,6 +148,7 @@
 import javax.json.stream.JsonParsingException;
 import javax.servlet.http.HttpServletRequest;
 import javax.servlet.http.HttpServletResponse;
+import javax.ws.rs.BadRequestException;
 import javax.ws.rs.Consumes;
 import javax.ws.rs.DELETE;
 import javax.ws.rs.DefaultValue;
@@ -158,13 +167,14 @@
 import javax.ws.rs.core.Response.Status;
 import static javax.ws.rs.core.Response.Status.BAD_REQUEST;
 import javax.ws.rs.core.UriInfo;
+
+import org.apache.commons.lang3.StringUtils;
 import org.apache.solr.client.solrj.SolrServerException;
 import org.glassfish.jersey.media.multipart.FormDataBodyPart;
 import org.glassfish.jersey.media.multipart.FormDataContentDisposition;
 import org.glassfish.jersey.media.multipart.FormDataParam;
 
 import com.amazonaws.services.s3.model.PartETag;
-import edu.harvard.iq.dataverse.FileMetadata;
 
 @Path("datasets")
 public class Datasets extends AbstractApiBean {
@@ -224,6 +234,9 @@ public class Datasets extends AbstractApiBean {
 
     @Inject
     DataverseRequestServiceBean dvRequestService;
+
+    @Inject
+    WorkflowServiceBean wfService;
 
     /**
      * Used to consolidate the way we parse and handle dataset versions.
@@ -1190,6 +1203,97 @@ public Response publishDataset(@PathParam("id") String id, @QueryParam("type") S
         }
     }
 
+    @POST
+    @Path("{id}/actions/:releasemigrated")
+    @Consumes("application/ld+json, application/json-ld")
+    public Response publishMigratedDataset(String jsonldBody, @PathParam("id") String id) {
+        try {
+            AuthenticatedUser user = findAuthenticatedUserOrDie();
+            if (!user.isSuperuser()) {
+                return error(Response.Status.FORBIDDEN, "Only superusers can release migrated datasets");
+            }
+
+            Dataset ds = findDatasetOrDie(id);
+            try {
+                JsonObject metadata = JSONLDUtil.decontextualizeJsonLD(jsonldBody);
+                String pubDate = metadata.getString(JsonLDTerm.schemaOrg("datePublished").getUrl());
+                logger.fine("Submitted date: " + pubDate);
+                LocalDateTime dateTime = null;
+                if(!StringUtils.isEmpty(pubDate)) {
+                    dateTime = JSONLDUtil.getDateTimeFrom(pubDate);
+                    final Timestamp time = Timestamp.valueOf(dateTime);
+                    //Set version release date
+                    ds.getLatestVersion().setReleaseTime(new Date(time.getTime()));
+                }
+                // dataset.getPublicationDateFormattedYYYYMMDD())
+                // Assign a version number if not set
+                if (ds.getLatestVersion().getVersionNumber() == null) {
+
+                    if (ds.getVersions().size() == 1) {
+                        // First Release
+                        ds.getLatestVersion().setVersionNumber(Long.valueOf(1));
+                        ds.getLatestVersion().setMinorVersionNumber(Long.valueOf(0));
+                        //Also set publication date if this is the first 
+                        if(dateTime != null) {
+                          ds.setPublicationDate(Timestamp.valueOf(dateTime));
+                        }
+                        // Release User is only set in FinalizeDatasetPublicationCommand if the pub date
+                        // is null, so set it here.
+                        ds.setReleaseUser((AuthenticatedUser) user);
+
+                    } else if (ds.getLatestVersion().isMinorUpdate()) {
+                        ds.getLatestVersion().setVersionNumber(Long.valueOf(ds.getVersionNumber()));
+                        ds.getLatestVersion().setMinorVersionNumber(Long.valueOf(ds.getMinorVersionNumber() + 1));
+
+                    } else {
+                        // major, non-first release
+                        ds.getLatestVersion().setVersionNumber(Long.valueOf(ds.getVersionNumber() + 1));
+                        ds.getLatestVersion().setMinorVersionNumber(Long.valueOf(0));
+                    }
+
+                }
+            } catch (Exception e) {
+                logger.fine(e.getMessage());
+                throw new BadRequestException("Unable to set publication date ("
+                        + JsonLDTerm.schemaOrg("datePublished").getUrl() + "): " + e.getMessage());
+            }
+            /*
+             * Note: The code here mirrors that in the
+             * edu.harvard.iq.dataverse.DatasetPage:updateCurrentVersion method. Any changes
+             * to the core logic (i.e. beyond updating the messaging about results) should
+             * be applied to the code there as well.
+             */
+            String errorMsg = null;
+            Optional<Workflow> prePubWf = wfService.getDefaultWorkflow(TriggerType.PrePublishDataset);
+
+            try {
+                // ToDo - should this be in onSuccess()? May relate to todo above
+                if (prePubWf.isPresent()) {
+                    // Start the workflow, the workflow will call FinalizeDatasetPublication later
+                    wfService.start(prePubWf.get(),
+                            new WorkflowContext(createDataverseRequest(user), ds, TriggerType.PrePublishDataset, true),
+                            false);
+                } else {
+                    FinalizeDatasetPublicationCommand cmd = new FinalizeDatasetPublicationCommand(ds,
+                            createDataverseRequest(user), true);
+                    ds = commandEngine.submit(cmd);
+                }
+            } catch (CommandException ex) {
+                errorMsg = BundleUtil.getStringFromBundle("datasetversion.update.failure") + " - " + ex.toString();
+                logger.severe(ex.getMessage());
+            }
+
+            if (errorMsg != null) {
+                return error(Response.Status.INTERNAL_SERVER_ERROR, errorMsg);
+            } else {
+                return prePubWf.isPresent() ? accepted(json(ds)) : ok(json(ds));
+            }
+
+        } catch (WrappedResponse ex) {
+            return ex.getResponse();
+        }
+    }
+
     @POST
     @Path("{id}/move/{targetDataverseAlias}")
     public Response moveDataset(@PathParam("id") String id, @PathParam("targetDataverseAlias") String targetDataverseAlias, @QueryParam("forceMove") Boolean force) {

src/main/java/edu/harvard/iq/dataverse/api/Dataverses.java

@@ -88,6 +88,7 @@
 import javax.json.stream.JsonParsingException;
 import javax.validation.ConstraintViolation;
 import javax.validation.ConstraintViolationException;
+import javax.ws.rs.BadRequestException;
 import javax.ws.rs.Consumes;
 import javax.ws.rs.DELETE;
 import javax.ws.rs.GET;
@@ -450,6 +451,59 @@ public Response importDatasetDdi(String xml, @PathParam("identifier") String par
         }
     }
 
+    @POST
+    @Path("{identifier}/datasets/:startmigration")
+    @Consumes("application/ld+json, application/json-ld")
+    public Response recreateDataset(String jsonLDBody, @PathParam("identifier") String parentIdtf) {
+        try {
+            User u = findUserOrDie();
+            if (!u.isSuperuser()) {
+                return error(Status.FORBIDDEN, "Not a superuser");
+            }
+            Dataverse owner = findDataverseOrDie(parentIdtf);
+
+            Dataset ds = new Dataset();
+
+            ds.setOwner(owner);
+            ds = JSONLDUtil.updateDatasetMDFromJsonLD(ds, jsonLDBody, metadataBlockSvc, datasetFieldSvc, false, true); 
+          //ToDo - verify PID is one Dataverse can manage (protocol/authority/shoulder match)
+            if(!
+            (ds.getAuthority().equals(settingsService.getValueForKey(SettingsServiceBean.Key.Authority))&& 
+            ds.getProtocol().equals(settingsService.getValueForKey(SettingsServiceBean.Key.Protocol))&&
+            ds.getIdentifier().startsWith(settingsService.getValueForKey(SettingsServiceBean.Key.Shoulder)))) {
+                throw new BadRequestException("Cannot recreate a dataset that has a PID that doesn't match the server's settings");
+            }
+            if(!datasetSvc.isIdentifierLocallyUnique(ds)) {
+                throw new BadRequestException("Cannot recreate a dataset whose PID is already in use");
+            }
+
+
+
+            if (ds.getVersions().isEmpty()) {
+                return badRequest("Supplied json must contain a single dataset version.");
+            }
+
+            DatasetVersion version = ds.getVersions().get(0);
+            if (!version.isPublished()) {
+                throw new BadRequestException("Cannot recreate a dataset that hasn't been published.");
+            }
+            //While the datasetversion whose metadata we're importing has been published, we consider it in draft until the API caller adds files and then completes the migration
+            version.setVersionState(DatasetVersion.VersionState.DRAFT);
+
+            DataverseRequest request = createDataverseRequest(u);
+
+            Dataset managedDs = execCommand(new ImportDatasetCommand(ds, request));
+            JsonObjectBuilder responseBld = Json.createObjectBuilder()
+                    .add("id", managedDs.getId())
+                    .add("persistentId", managedDs.getGlobalId().toString());
+
+            return created("/datasets/" + managedDs.getId(), responseBld);
+
+        } catch (WrappedResponse ex) {
+            return ex.getResponse();
+        }
+    }
+
     private Dataset parseDataset(String datasetJson) throws WrappedResponse {
         try (StringReader rdr = new StringReader(datasetJson)) {
             return jsonParser().parseDataset(Json.createReader(rdr).readObject());

src/main/java/edu/harvard/iq/dataverse/engine/command/impl/FinalizeDatasetPublicationCommand.java

@@ -115,12 +115,16 @@ public Dataset execute(CommandContext ctxt) throws CommandException {
         } 
 
         // update metadata
-        theDataset.getLatestVersion().setReleaseTime(getTimestamp());
+        if (theDataset.getLatestVersion().getReleaseTime() == null) {
+            // Allow migrated versions to keep original release dates
+            theDataset.getLatestVersion().setReleaseTime(getTimestamp());
+        }
         theDataset.getLatestVersion().setLastUpdateTime(getTimestamp());
         theDataset.setModificationTime(getTimestamp());
         theDataset.setFileAccessRequest(theDataset.getLatestVersion().getTermsOfUseAndAccess().isFileAccessRequest());
 
-        updateFiles(getTimestamp(), ctxt);
+        //Use dataset pub date (which may not be the current date for migrated datasets)
+        updateFiles(new Timestamp(theDataset.getLatestVersion().getReleaseTime().getTime()), ctxt);
 
         // 
         // TODO: Not sure if this .merge() is necessary here - ?