diff --git a/conf/docker-aio/readme.md b/conf/docker-aio/readme.md index 1a388618ce5..f3031a5bb6e 100644 --- a/conf/docker-aio/readme.md +++ b/conf/docker-aio/readme.md @@ -1,7 +1,7 @@ # Docker All-In-One > :information_source: **NOTE: Sunsetting of this module is imminent.** There is no schedule yet, but expect it to go away. -> Please let the [Dataverse Containerization Working Group](https://dc.wgs.gdcc.io) know if you are a user and +> Please let the [Dataverse Containerization Working Group](https://ct.gdcc.io) know if you are a user and > what should be preserved. First pass docker all-in-one image, intended for running integration tests against. diff --git a/doc/release-notes/8889-2-filepids-in-collections-changes.md b/doc/release-notes/8889-2-filepids-in-collections-changes.md new file mode 100644 index 00000000000..a3693d9008e --- /dev/null +++ b/doc/release-notes/8889-2-filepids-in-collections-changes.md @@ -0,0 +1,15 @@ +The default for whether PIDs are registered for files or not is now false. + +Installations where file PIDs were enabled by default will have to add the :FilePIDsEnabled = true setting to maintain the existing functionality. + +Add step to install: + + If your installation did not have :FilePIDsEnabled set, you will need to set it to true to keep file PIDs enabled: + + curl -X PUT -d 'true' http://localhost:8080/api/admin/settings/:FilePIDsEnabled + + + +It is now possible to allow File PIDs to be enabled/disabled per collection. See the [:AllowEnablingFilePIDsPerCollection](https://guides.dataverse.org/en/latest/installation/config.html#allowenablingfilepidspercollection) section of the Configuration guide for details. + +For example, registration of PIDs for files can now be enabled in a specific collection when it is disabled instance-wide. Or it can be disabled in specific collections where it is enabled by default. \ No newline at end of file diff --git a/doc/release-notes/9704-GeotifShp.md b/doc/release-notes/9704-GeotifShp.md new file mode 100644 index 00000000000..3a663ec3944 --- /dev/null +++ b/doc/release-notes/9704-GeotifShp.md @@ -0,0 +1 @@ +Two file previewers for GeoTIFF and Shapefiles are now available for visualizing geotiff image files and zipped Shapefiles on a map. diff --git a/doc/sphinx-guides/source/_static/admin/dataverse-external-tools.tsv b/doc/sphinx-guides/source/_static/admin/dataverse-external-tools.tsv index 6e0eb810b27..826989643ba 100644 --- a/doc/sphinx-guides/source/_static/admin/dataverse-external-tools.tsv +++ b/doc/sphinx-guides/source/_static/admin/dataverse-external-tools.tsv @@ -2,5 +2,5 @@ Tool Type Scope Description Data Explorer explore file A GUI which lists the variables in a tabular data file allowing searching, charting and cross tabulation analysis. See the README.md file at https://github.com/scholarsportal/dataverse-data-explorer-v2 for the instructions on adding Data Explorer to your Dataverse. Whole Tale explore dataset A platform for the creation of reproducible research packages that allows users to launch containerized interactive analysis environments based on popular tools such as Jupyter and RStudio. Using this integration, Dataverse users can launch Jupyter and RStudio environments to analyze published datasets. For more information, see the `Whole Tale User Guide `_. Binder explore dataset Binder allows you to spin up custom computing environments in the cloud (including Jupyter notebooks) with the files from your dataset. `Installation instructions `_ are in the Data Exploration Lab girder_ythub project. See also :ref:`binder`. -File Previewers explore file A set of tools that display the content of files - including audio, html, `Hypothes.is `_ annotations, images, PDF, text, video, tabular data, spreadsheets, GeoJSON, zip, HDF5, NetCDF, and NcML files - allowing them to be viewed without downloading the file. The previewers can be run directly from github.io, so the only required step is using the Dataverse API to register the ones you want to use. Documentation, including how to optionally brand the previewers, and an invitation to contribute through github are in the README.md file. Initial development was led by the Qualitative Data Repository and the spreasdheet previewer was added by the Social Sciences and Humanities Open Cloud (SSHOC) project. https://github.com/gdcc/dataverse-previewers +File Previewers explore file A set of tools that display the content of files - including audio, html, `Hypothes.is `_ annotations, images, PDF, text, video, tabular data, spreadsheets, GeoJSON, GeoTIFF, Shapefile, zip, HDF5, NetCDF, and NcML files - allowing them to be viewed without downloading the file. The previewers can be run directly from github.io, so the only required step is using the Dataverse API to register the ones you want to use. Documentation, including how to optionally brand the previewers, and an invitation to contribute through github are in the README.md file. Initial development was led by the Qualitative Data Repository and the spreasdheet previewer was added by the Social Sciences and Humanities Open Cloud (SSHOC) project. https://github.com/gdcc/dataverse-previewers Data Curation Tool configure file A GUI for curating data by adding labels, groups, weights and other details to assist with informed reuse. See the README.md file at https://github.com/scholarsportal/Dataverse-Data-Curation-Tool for the installation instructions. diff --git a/doc/sphinx-guides/source/admin/dataverses-datasets.rst b/doc/sphinx-guides/source/admin/dataverses-datasets.rst index 92e01578f71..170807d3d67 100644 --- a/doc/sphinx-guides/source/admin/dataverses-datasets.rst +++ b/doc/sphinx-guides/source/admin/dataverses-datasets.rst @@ -154,6 +154,8 @@ In the following example, the database id of the file is 42:: export FILE_ID=42 curl "http://localhost:8080/api/admin/$FILE_ID/registerDataFile" + +This method will return a FORBIDDEN response if minting of file PIDs is not enabled for the collection the file is in. (Note that it is possible to have file PIDs enabled for a specific collection, even when it is disabled for the Dataverse installation as a whole. See :ref:`collection-attributes-api` in the Native API Guide.) Mint PIDs for all unregistered published files in the specified collection ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -162,7 +164,8 @@ The following API will register the PIDs for all the yet unregistered published curl "http://localhost:8080/api/admin/registerDataFiles/{collection_alias}" -It will not attempt to register the datafiles in its sub-collections, so this call will need to be repeated on any sub-collections where files need to be registered as well. File-level PID registration must be enabled on the collection. (Note that it is possible to have it enabled for a specific collection, even when it is disabled for the Dataverse installation as a whole. See :ref:`collection-attributes-api` in the Native API Guide.) +It will not attempt to register the datafiles in its sub-collections, so this call will need to be repeated on any sub-collections where files need to be registered as well. +File-level PID registration must be enabled on the collection. (Note that it is possible to have it enabled for a specific collection, even when it is disabled for the Dataverse installation as a whole. See :ref:`collection-attributes-api` in the Native API Guide.) This API will sleep for 1 second between registration calls by default. A longer sleep interval can be specified with an optional ``sleep=`` parameter:: @@ -171,7 +174,7 @@ This API will sleep for 1 second between registration calls by default. A longer Mint PIDs for ALL unregistered files in the database ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -The following API will attempt to register the PIDs for all the published files in your instance that do not yet have them:: +The following API will attempt to register the PIDs for all the published files in your instance, in collections that allow file PIDs, that do not yet have them:: curl http://localhost:8080/api/admin/registerDataFileAll diff --git a/doc/sphinx-guides/source/admin/make-data-count.rst b/doc/sphinx-guides/source/admin/make-data-count.rst index 521ee4f97ca..ec6ff9a685b 100644 --- a/doc/sphinx-guides/source/admin/make-data-count.rst +++ b/doc/sphinx-guides/source/admin/make-data-count.rst @@ -72,6 +72,8 @@ Enable or Disable Display of Make Data Count Metrics By default, when MDC logging is enabled (when ``:MDCLogPath`` is set), your Dataverse installation will display MDC metrics instead of it's internal (legacy) metrics. You can avoid this (e.g. to collect MDC metrics for some period of time before starting to display them) by setting ``:DisplayMDCMetrics`` to false. +You can also decide to display MDC metrics along with Dataverse's traditional download counts from the time before MDC was enabled. To do this, set the :ref:`:MDCStartDate` to when you started MDC logging. + The following discussion assumes ``:MDCLogPath`` has been set to ``/usr/local/payara5/glassfish/domains/domain1/logs/mdc`` Configure Counter Processor diff --git a/doc/sphinx-guides/source/api/native-api.rst b/doc/sphinx-guides/source/api/native-api.rst index ccf172eb84e..2bbcf108cad 100644 --- a/doc/sphinx-guides/source/api/native-api.rst +++ b/doc/sphinx-guides/source/api/native-api.rst @@ -298,6 +298,8 @@ The fully expanded example above (without environment variables) looks like this curl -H X-Dataverse-key:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx -X POST -H "Content-type:application/json" https://demo.dataverse.org/api/dataverses/root/metadatablockfacets/isRoot -d 'true' +.. _create-role-in-collection: + Create a New Role in a Dataverse Collection ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -317,16 +319,7 @@ The fully expanded example above (without environment variables) looks like this curl -H X-Dataverse-key:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx -X POST -H "Content-type:application/json" https://demo.dataverse.org/api/dataverses/root/roles --upload-file roles.json -Where ``roles.json`` looks like this:: - - { - "alias": "sys1", - "name": “Restricted System Role”, - "description": “A person who may only add datasets.”, - "permissions": [ - "AddDataset" - ] - } +For ``roles.json`` see :ref:`json-representation-of-a-role` .. note:: Only a Dataverse installation account with superuser permissions is allowed to create roles in a Dataverse Collection. @@ -753,7 +746,7 @@ The following attributes are supported: * ``name`` Name * ``description`` Description * ``affiliation`` Affiliation -* ``filePIDsEnabled`` ("true" or "false") Enables or disables registration of file-level PIDs in datasets within the collection (overriding the instance-wide setting). +* ``filePIDsEnabled`` ("true" or "false") Restricted to use by superusers and only when the :ref:`:AllowEnablingFilePIDsPerCollection <:AllowEnablingFilePIDsPerCollection>` setting is true. Enables or disables registration of file-level PIDs in datasets within the collection (overriding the instance-wide setting). Datasets @@ -3094,26 +3087,14 @@ Optionally, you may use a third query parameter "sendEmailNotification=false" to Roles ----- -Create a New Role in a Dataverse Collection -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Creates a new role under Dataverse collection ``id``. Needs a json file with the role description: - -.. code-block:: bash - - export API_TOKEN=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx - export SERVER_URL=https://demo.dataverse.org - export ID=root - - curl -H X-Dataverse-key:$API_TOKEN -X POST -H "Content-type:application/json" $SERVER_URL/api/dataverses/$ID/roles --upload-file roles.json +A role is a set of permissions. -The fully expanded example above (without environment variables) looks like this: - -.. code-block:: bash +.. _json-representation-of-a-role: - curl -H X-Dataverse-key:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx -X POST -H "Content-type:application/json" https://demo.dataverse.org/api/dataverses/root/roles --upload-file roles.json +JSON Representation of a Role +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Where ``roles.json`` looks like this:: +The JSON representation of a role (``roles.json``) looks like this:: { "alias": "sys1", @@ -3124,8 +3105,12 @@ Where ``roles.json`` looks like this:: ] } -.. note:: Only a Dataverse installation account with superuser permissions is allowed to create roles in a Dataverse Collection. +.. note:: alias is constrained to a length of 16 characters + +Create Role +~~~~~~~~~~~ +Roles can be created globally (:ref:`create-global-role`) or for individual Dataverse collections (:ref:`create-role-in-collection`). Show Role ~~~~~~~~~ @@ -3986,12 +3971,24 @@ List all global roles in the system. :: GET http://$SERVER/api/admin/roles +.. _create-global-role: + Create Global Role ~~~~~~~~~~~~~~~~~~ Creates a global role in the Dataverse installation. The data POSTed are assumed to be a role JSON. :: POST http://$SERVER/api/admin/roles + +.. code-block:: bash + + export API_TOKEN=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx + export SERVER_URL=https://demo.dataverse.org + export ID=root + + curl -H X-Dataverse-key:$API_TOKEN -X POST $SERVER_URL/api/admin/roles --upload-file roles.json + +``roles.json`` see :ref:`json-representation-of-a-role` Delete Global Role ~~~~~~~~~~~~~~~~~~ diff --git a/doc/sphinx-guides/source/developers/testing.rst b/doc/sphinx-guides/source/developers/testing.rst index 55351b851bb..c228d8e20ca 100755 --- a/doc/sphinx-guides/source/developers/testing.rst +++ b/doc/sphinx-guides/source/developers/testing.rst @@ -175,7 +175,7 @@ Running the full API test suite using Docker .. note:: Sunsetting of this module is imminent.** There is no schedule yet, but expect it to go away. - Please let the `Dataverse Containerization Working Group `_ know if you are a user and + Please let the `Dataverse Containerization Working Group `_ know if you are a user and what should be preserved. To run the full suite of integration tests on your laptop, we recommend using the "all in one" Docker configuration described in ``conf/docker-aio/readme.md`` in the root of the repo. diff --git a/doc/sphinx-guides/source/installation/config.rst b/doc/sphinx-guides/source/installation/config.rst index 13f858c6538..8493702406b 100644 --- a/doc/sphinx-guides/source/installation/config.rst +++ b/doc/sphinx-guides/source/installation/config.rst @@ -34,6 +34,12 @@ It is very important to keep the block in place for the "admin" endpoint, and to It's also possible to prevent file uploads via API by adjusting the :ref:`:UploadMethods` database setting. +If you are using a load balancer or a reverse proxy, there are some additional considerations. If no additional configurations are made and the upstream is configured to redirect to localhost, the API will be accessible from the outside, as your installation will register as origin the localhost for any requests to the endpoints "admin" and "builtin-users". To prevent this, you have two options: + +- If your upstream is configured to redirect to localhost, you will need to set the :ref:`JVM option ` to one of the following values ``%client.name% %datetime% %request% %status% %response.length% %header.referer% %header.x-forwarded-for%`` and configure from the load balancer side the chosen header to populate with the client IP address. + +- Another solution is to set the upstream to the client IP address. In this case no further configuration is needed. + Forcing HTTPS +++++++++++++ @@ -248,7 +254,7 @@ this provider. - :ref:`:Shoulder <:Shoulder>` - :ref:`:IdentifierGenerationStyle <:IdentifierGenerationStyle>` (optional) - :ref:`:DataFilePIDFormat <:DataFilePIDFormat>` (optional) -- :ref:`:FilePIDsEnabled <:FilePIDsEnabled>` (optional, defaults to true) +- :ref:`:FilePIDsEnabled <:FilePIDsEnabled>` (optional, defaults to false) .. _pids-handle-configuration: @@ -297,7 +303,7 @@ Here are the configuration options for PermaLinks: - :ref:`:Shoulder <:Shoulder>` - :ref:`:IdentifierGenerationStyle <:IdentifierGenerationStyle>` (optional) - :ref:`:DataFilePIDFormat <:DataFilePIDFormat>` (optional) -- :ref:`:FilePIDsEnabled <:FilePIDsEnabled>` (optional, defaults to true) +- :ref:`:FilePIDsEnabled <:FilePIDsEnabled>` (optional, defaults to false) .. _auth-modes: @@ -2775,14 +2781,35 @@ timestamps. :FilePIDsEnabled ++++++++++++++++ -Toggles publishing of file-level PIDs for the entire installation. By default this setting is absent and Dataverse Software assumes it to be true. If enabled, the registration will be performed asynchronously (in the background) during publishing of a dataset. +Toggles publishing of file-level PIDs for the entire installation. By default this setting is absent and Dataverse Software assumes it to be false. If enabled, the registration will be performed asynchronously (in the background) during publishing of a dataset. + +It is possible to override the installation-wide setting for specific collections, see :ref:`:AllowEnablingFilePIDsPerCollection <:AllowEnablingFilePIDsPerCollection>`. For example, registration of PIDs for files can be enabled in a specific collection when it is disabled instance-wide. Or it can be disabled in specific collections where it is enabled by default. See :ref:`collection-attributes-api` for details. + +To enable file-level PIDs for the entire installation:: + +``curl -X PUT -d 'true' http://localhost:8080/api/admin/settings/:FilePIDsEnabled`` -If you don't want to register file-based PIDs for your installation, set: + +If you don't want to register file-based PIDs for your entire installation:: ``curl -X PUT -d 'false' http://localhost:8080/api/admin/settings/:FilePIDsEnabled`` +.. _:AllowEnablingFilePIDsPerCollection: + +:AllowEnablingFilePIDsPerCollection ++++++++++++++++++++++++++++++++++++ + +Toggles whether superusers can change the File PIDs policy per collection. By default this setting is absent and Dataverse Software assumes it to be false. + +For example, if this setting is true, registration of PIDs for files can be enabled in a specific collection when it is disabled instance-wide. Or it can be disabled in specific collections where it is enabled by default. See :ref:`collection-attributes-api` for details. + +To enable setting file-level PIDs per collection:: + +``curl -X PUT -d 'true' http://localhost:8080/api/admin/settings/:AllowEnablingFilePIDsPerCollection`` + + +When :AllowEnablingFilePIDsPerCollection is true, setting File PIDs to be enabled/disabled for a given collection can be done via the Native API - see :ref:`collection-attributes-api` in the Native API Guide. -It is possible to override the installation-wide setting for specific collections. For example, registration of PIDs for files can be enabled in a specific collection when it is disabled instance-wide. Or it can be disabled in specific collections where it is enabled by default. See :ref:`collection-attributes-api` for details. .. _:IndependentHandleService: @@ -3469,6 +3496,20 @@ Sets the path where the raw Make Data Count logs are stored before being process ``curl -X PUT -d 'false' http://localhost:8080/api/admin/settings/:DisplayMDCMetrics`` +.. _:MDCStartDate: + +:MDCStartDate ++++++++++++++ + +It is possible to display MDC metrics (as of the start date of MDC logging) along with legacy download counts, generated before MDC was enabled. +This is enabled via the new setting `:MDCStartDate` that specifies the cut-over date. If a dataset has any legacy access counts collected prior to that date, those numbers will be displayed in addition to the MDC views and downloads recorded since then. +(Nominally, this date should be when your installation started logging MDC metrics but it can be any date after that if desired.) + + +``curl -X PUT -d '2019-10-01' http://localhost:8080/api/admin/settings/:MDCStartDate`` + + + .. _:Languages: :Languages diff --git a/doc/sphinx-guides/source/user/dataset-management.rst b/doc/sphinx-guides/source/user/dataset-management.rst index 648d81c1076..3b5b4ec6ba8 100755 --- a/doc/sphinx-guides/source/user/dataset-management.rst +++ b/doc/sphinx-guides/source/user/dataset-management.rst @@ -208,6 +208,8 @@ Previewers are available for the following file types: - Zip (preview and extract/download) - HTML - GeoJSON +- GeoTIFF +- Shapefile - NetCDF/HDF5 - Hypothes.is @@ -343,6 +345,20 @@ GeoJSON A map will be shown as a preview of GeoJSON files when the previewer has been enabled (see :ref:`file-previews`). See also a `video demo `_ of the GeoJSON previewer by its author, Kaitlin Newson. +.. _geotiff: + +GeoTIFF +------- + +A map is also displayed as a preview of GeoTiFF image files, whose previewer must be enabled (see :ref:`file-previews`). Since GeoTIFFs do not have their own mimetype, it is advisable to use this previewer only when GeoTIFFs are used (and not "normal" TIFs). For performance reasons, this previewer has a file size limit of 15 MB and a row/column limit of 50,000 so that larger files are not loaded. + +.. _shapefile: + +Shapefile +--------- + +Another previewer can be enabled for shapefiles (see :ref:`file-previews`). This previewer only works with zipped shapefiles (see :doc:`/developers/geospatial`). A file size limit of 20 MB is set for this previewer (also because of performance reasons). + .. _netcdf-and-hdf5: NetCDF and HDF5 diff --git a/src/main/java/edu/harvard/iq/dataverse/DatasetPage.java b/src/main/java/edu/harvard/iq/dataverse/DatasetPage.java index 343868e81e2..35d235146f4 100644 --- a/src/main/java/edu/harvard/iq/dataverse/DatasetPage.java +++ b/src/main/java/edu/harvard/iq/dataverse/DatasetPage.java @@ -346,7 +346,7 @@ public void setSelectedHostDataverse(Dataverse selectedHostDataverse) { private Boolean hasRsyncScript = false; - private Boolean hasTabular = false; + /*private Boolean hasTabular = false;*/ /** @@ -355,6 +355,12 @@ public void setSelectedHostDataverse(Dataverse selectedHostDataverse) { * sometimes you want to know about the current version ("no tabular files * currently"). Like all files, tabular files can be deleted. */ + /** + * There doesn't seem to be an actual real life case where we need to know + * if this dataset "has ever had a tabular file" - for all practical purposes + * only the versionHasTabular appears to be in use. I'm going to remove the + * other boolean. + */ private boolean versionHasTabular = false; private boolean showIngestSuccess; @@ -1881,30 +1887,58 @@ private String init(boolean initFull) { if (persistentId != null) { setIdByPersistentId(); } + if (this.getId() != null) { // Set Working Version and Dataset by Datasaet Id and Version - dataset = datasetService.findDeep(this.getId()); + + // We are only performing these lookups to obtain the database id + // of the version that we are displaying, and then we will use it + // to perform a .findDeep(versionId); see below. + + // TODO: replace the code block below, the combination of + // datasetService.find(id) and datasetVersionService.selectRequestedVersion() + // with some optimized, direct query-based way of obtaining + // the numeric id of the requested DatasetVersion (and that's + // all we need, we are not using any of the entities produced + // below. + + dataset = datasetService.find(this.getId()); + if (dataset == null) { logger.warning("No such dataset: "+dataset); return permissionsWrapper.notFound(); } //retrieveDatasetVersionResponse = datasetVersionService.retrieveDatasetVersionById(dataset.getId(), version); retrieveDatasetVersionResponse = datasetVersionService.selectRequestedVersion(dataset.getVersions(), version); + if (retrieveDatasetVersionResponse == null) { + return permissionsWrapper.notFound(); + } this.workingVersion = retrieveDatasetVersionResponse.getDatasetVersion(); logger.fine("retrieved version: id: " + workingVersion.getId() + ", state: " + this.workingVersion.getVersionState()); - - } else if (versionId != null) { - // TODO: 4.2.1 - this method is broken as of now! - // Set Working Version and Dataset by DatasaetVersion Id - //retrieveDatasetVersionResponse = datasetVersionService.retrieveDatasetVersionByVersionId(versionId); - + + versionId = workingVersion.getId(); + + this.workingVersion = null; + this.dataset = null; + + } + + // ... And now the "real" working version lookup: + + if (versionId != null) { + this.workingVersion = datasetVersionService.findDeep(versionId); + dataset = workingVersion.getDataset(); + } + + if (workingVersion == null) { + logger.warning("Failed to retrieve version"); + return permissionsWrapper.notFound(); } + this.maxFileUploadSizeInBytes = systemConfig.getMaxFileUploadSizeForStore(dataset.getEffectiveStorageDriverId()); - if (retrieveDatasetVersionResponse == null) { - return permissionsWrapper.notFound(); - } + switch (selectTab){ case "dataFilesTab": @@ -1921,16 +1955,6 @@ private String init(boolean initFull) { break; } - //this.dataset = this.workingVersion.getDataset(); - - // end: Set the workingVersion and Dataset - // --------------------------------------- - // Is the DatasetVersion or Dataset null? - // - if (workingVersion == null || this.dataset == null) { - return permissionsWrapper.notFound(); - } - // Is the Dataset harvested? if (dataset.isHarvested()) { @@ -1958,7 +1982,7 @@ private String init(boolean initFull) { return permissionsWrapper.notAuthorized(); } - if (!retrieveDatasetVersionResponse.wasRequestedVersionRetrieved()) { + if (retrieveDatasetVersionResponse != null && !retrieveDatasetVersionResponse.wasRequestedVersionRetrieved()) { //msg("checkit " + retrieveDatasetVersionResponse.getDifferentVersionMessage()); JsfHelper.addWarningMessage(retrieveDatasetVersionResponse.getDifferentVersionMessage());//BundleUtil.getStringFromBundle("dataset.message.metadataSuccess")); } @@ -2105,23 +2129,18 @@ private String init(boolean initFull) { displayLockInfo(dataset); displayPublishMessage(); + // TODO: replace this loop, and the loop in the method that calculates + // the total "originals" size of the dataset with direct custom queries; + // then we'll be able to drop the lookup hint for DataTable from the + // findDeep() method for the version and further speed up the lookup + // a little bit. for (FileMetadata fmd : workingVersion.getFileMetadatas()) { if (fmd.getDataFile().isTabularData()) { versionHasTabular = true; break; } } - for(DataFile f : dataset.getFiles()) { - // TODO: Consider uncommenting this optimization. -// if (versionHasTabular) { -// hasTabular = true; -// break; -// } - if(f.isTabularData()) { - hasTabular = true; - break; - } - } + //Show ingest success message if refresh forces a page reload after ingest success //This is needed to display the explore buttons (the fileDownloadHelper needs to be reloaded via page if (showIngestSuccess) { @@ -2168,14 +2187,19 @@ private void displayPublishMessage(){ } } + Boolean valid = null; + public boolean isValid() { - DatasetVersion version = dataset.getLatestVersion(); - if (!version.isDraft()) { - return true; + if (valid == null) { + DatasetVersion version = dataset.getLatestVersion(); + if (!version.isDraft()) { + valid = true; + } + DatasetVersion newVersion = version.cloneDatasetVersion(); + newVersion.setDatasetFields(newVersion.initDatasetFields()); + valid = newVersion.isValid(); } - DatasetVersion newVersion = version.cloneDatasetVersion(); - newVersion.setDatasetFields(newVersion.initDatasetFields()); - return newVersion.isValid(); + return valid; } public boolean isValidOrCanReviewIncomplete() { @@ -2400,9 +2424,9 @@ private DefaultTreeNode createFileTreeNode(FileMetadata fileMetadata, TreeNode p return fileNode; } - public boolean isHasTabular() { + /*public boolean isHasTabular() { return hasTabular; - } + }*/ public boolean isVersionHasTabular() { return versionHasTabular; @@ -2839,48 +2863,29 @@ public String refresh() { //dataset = datasetService.find(dataset.getId()); dataset = null; + workingVersion = null; logger.fine("refreshing working version"); DatasetVersionServiceBean.RetrieveDatasetVersionResponse retrieveDatasetVersionResponse = null; - if (persistentId != null) { - setIdByPersistentId(); - if (this.getId() == null) { - logger.warning("No such dataset: "+persistentId); - return permissionsWrapper.notFound(); - } - dataset = datasetService.findDeep(this.getId()); - if (dataset == null) { - logger.warning("No such dataset: "+persistentId); - return permissionsWrapper.notFound(); - } - retrieveDatasetVersionResponse = datasetVersionService.selectRequestedVersion(dataset.getVersions(), version); - } else if (versionId != null) { - retrieveDatasetVersionResponse = datasetVersionService.retrieveDatasetVersionByVersionId(versionId); - } + if (versionId != null) { + // versionId must have been set by now, in the init() method, + // regardless of how the page was originally called - by the dataset + // database id, by the persistent identifier, or by the db id of + // the version. + this.workingVersion = datasetVersionService.findDeep(versionId); + dataset = workingVersion.getDataset(); + } + - if (retrieveDatasetVersionResponse == null) { + if (this.workingVersion == null) { // TODO: // should probably redirect to the 404 page, if we can't find // this version anymore. // -- L.A. 4.2.3 return ""; } - this.workingVersion = retrieveDatasetVersionResponse.getDatasetVersion(); - - if (this.workingVersion == null) { - // TODO: - // same as the above - - return ""; - } - - if (dataset == null) { - // this would be the case if we were retrieving the version by - // the versionId, above. - this.dataset = this.workingVersion.getDataset(); - } fileMetadatasSearch = selectFileMetadatasForDisplay(); @@ -3059,19 +3064,32 @@ public void setTooLargeToDownload(boolean tooLargeToDownload) { this.tooLargeToDownload = tooLargeToDownload; } + private Long sizeOfDatasetArchival = null; + private Long sizeOfDatasetOriginal = null; + + public Long getSizeOfDatasetNumeric() { - if (this.hasTabular){ + if (this.versionHasTabular){ return Math.min(getSizeOfDatasetOrigNumeric(), getSizeOfDatasetArchivalNumeric()); } return getSizeOfDatasetOrigNumeric(); } public Long getSizeOfDatasetOrigNumeric() { - return DatasetUtil.getDownloadSizeNumeric(workingVersion, true); + if (versionHasTabular) { + if (sizeOfDatasetOriginal == null) { + sizeOfDatasetOriginal = DatasetUtil.getDownloadSizeNumeric(workingVersion, true); + } + return sizeOfDatasetOriginal; + } + return getSizeOfDatasetArchivalNumeric(); } public Long getSizeOfDatasetArchivalNumeric() { - return DatasetUtil.getDownloadSizeNumeric(workingVersion, false); + if (sizeOfDatasetArchival == null) { + sizeOfDatasetArchival = DatasetUtil.getDownloadSizeNumeric(workingVersion, false); + } + return sizeOfDatasetArchival; } public String getSizeOfSelectedAsString(){ @@ -6100,16 +6118,23 @@ public String getWebloaderUrlForDataset(Dataset d) { /** * Add Signposting + * * @return String */ + + String signpostingLinkHeader = null; + public String getSignpostingLinkHeader() { if (!workingVersion.isReleased()) { return null; } - SignpostingResources sr = new SignpostingResources(systemConfig, workingVersion, - JvmSettings.SIGNPOSTING_LEVEL1_AUTHOR_LIMIT.lookupOptional().orElse(""), - JvmSettings.SIGNPOSTING_LEVEL1_ITEM_LIMIT.lookupOptional().orElse("")); - return sr.getLinks(); + if (signpostingLinkHeader == null) { + SignpostingResources sr = new SignpostingResources(systemConfig, workingVersion, + JvmSettings.SIGNPOSTING_LEVEL1_AUTHOR_LIMIT.lookupOptional().orElse(""), + JvmSettings.SIGNPOSTING_LEVEL1_ITEM_LIMIT.lookupOptional().orElse("")); + signpostingLinkHeader = sr.getLinks(); + } + return signpostingLinkHeader; } } diff --git a/src/main/java/edu/harvard/iq/dataverse/DatasetServiceBean.java b/src/main/java/edu/harvard/iq/dataverse/DatasetServiceBean.java index 6ff3270808f..c93236f347b 100644 --- a/src/main/java/edu/harvard/iq/dataverse/DatasetServiceBean.java +++ b/src/main/java/edu/harvard/iq/dataverse/DatasetServiceBean.java @@ -128,7 +128,7 @@ public Dataset findDeep(Object pk) { .setHint("eclipselink.left-join-fetch", "o.files.dataFileTags") .setHint("eclipselink.left-join-fetch", "o.files.fileMetadatas") .setHint("eclipselink.left-join-fetch", "o.files.fileMetadatas.fileCategories") - .setHint("eclipselink.left-join-fetch", "o.files.guestbookResponses") + //.setHint("eclipselink.left-join-fetch", "o.files.guestbookResponses") .setHint("eclipselink.left-join-fetch", "o.files.embargo") .setHint("eclipselink.left-join-fetch", "o.files.fileAccessRequests") .setHint("eclipselink.left-join-fetch", "o.files.owner") diff --git a/src/main/java/edu/harvard/iq/dataverse/DatasetVersion.java b/src/main/java/edu/harvard/iq/dataverse/DatasetVersion.java index 2873d2f2a22..9d5c27ae9fd 100644 --- a/src/main/java/edu/harvard/iq/dataverse/DatasetVersion.java +++ b/src/main/java/edu/harvard/iq/dataverse/DatasetVersion.java @@ -66,7 +66,9 @@ @NamedQueries({ @NamedQuery(name = "DatasetVersion.findUnarchivedReleasedVersion", query = "SELECT OBJECT(o) FROM DatasetVersion AS o WHERE o.dataset.harvestedFrom IS NULL and o.releaseTime IS NOT NULL and o.archivalCopyLocation IS NULL" - )}) + ), + @NamedQuery(name = "DatasetVersion.findById", + query = "SELECT o FROM DatasetVersion o LEFT JOIN FETCH o.fileMetadatas WHERE o.id=:id")}) @Entity diff --git a/src/main/java/edu/harvard/iq/dataverse/DatasetVersionServiceBean.java b/src/main/java/edu/harvard/iq/dataverse/DatasetVersionServiceBean.java index 9f272ec6877..607c46d3662 100644 --- a/src/main/java/edu/harvard/iq/dataverse/DatasetVersionServiceBean.java +++ b/src/main/java/edu/harvard/iq/dataverse/DatasetVersionServiceBean.java @@ -154,6 +154,21 @@ public DatasetVersion getDatasetVersion(){ public DatasetVersion find(Object pk) { return em.find(DatasetVersion.class, pk); } + + public DatasetVersion findDeep(Object pk) { + return (DatasetVersion) em.createNamedQuery("DatasetVersion.findById") + .setParameter("id", pk) + // Optimization hints: retrieve all data in one query; this prevents point queries when iterating over the files + .setHint("eclipselink.left-join-fetch", "o.fileMetadatas.dataFile.ingestRequest") + .setHint("eclipselink.left-join-fetch", "o.fileMetadatas.dataFile.thumbnailForDataset") + .setHint("eclipselink.left-join-fetch", "o.fileMetadatas.dataFile.dataTables") + .setHint("eclipselink.left-join-fetch", "o.fileMetadatas.fileCategories") + .setHint("eclipselink.left-join-fetch", "o.fileMetadatas.dataFile.embargo") + .setHint("eclipselink.left-join-fetch", "o.fileMetadatas.datasetVersion") + .setHint("eclipselink.left-join-fetch", "o.fileMetadatas.dataFile.releaseUser") + .setHint("eclipselink.left-join-fetch", "o.fileMetadatas.dataFile.creator") + .getSingleResult(); + } public DatasetVersion findByFriendlyVersionNumber(Long datasetId, String friendlyVersionNumber) { Long majorVersionNumber = null; diff --git a/src/main/java/edu/harvard/iq/dataverse/api/Admin.java b/src/main/java/edu/harvard/iq/dataverse/api/Admin.java index 45fb5480577..2e98388b4b6 100644 --- a/src/main/java/edu/harvard/iq/dataverse/api/Admin.java +++ b/src/main/java/edu/harvard/iq/dataverse/api/Admin.java @@ -1514,6 +1514,9 @@ public Response registerDataFile(@Context ContainerRequestContext crc, @PathPara User u = getRequestUser(crc); DataverseRequest r = createDataverseRequest(u); DataFile df = findDataFileOrDie(id); + if(!systemConfig.isFilePIDsEnabledForCollection(df.getOwner().getOwner())) { + return forbidden("PIDs are not enabled for this file's collection."); + } if (df.getIdentifier() == null || df.getIdentifier().isEmpty()) { execCommand(new RegisterDvObjectCommand(r, df)); } else { @@ -1537,41 +1540,56 @@ public Response registerDataFileAll(@Context ContainerRequestContext crc) { Integer alreadyRegistered = 0; Integer released = 0; Integer draft = 0; + Integer skipped = 0; logger.info("Starting to register: analyzing " + count + " files. " + new Date()); logger.info("Only unregistered, published files will be registered."); + User u = null; + try { + u = getRequestAuthenticatedUserOrDie(crc); + } catch (WrappedResponse e1) { + return error(Status.UNAUTHORIZED, "api key required"); + } + DataverseRequest r = createDataverseRequest(u); for (DataFile df : fileService.findAll()) { try { if ((df.getIdentifier() == null || df.getIdentifier().isEmpty())) { - if (df.isReleased()) { + if(!systemConfig.isFilePIDsEnabledForCollection(df.getOwner().getOwner())) { + skipped++; + if (skipped % 100 == 0) { + logger.info(skipped + " of " + count + " files not in collections that allow file PIDs. " + new Date()); + } + } else if (df.isReleased()) { released++; - User u = getRequestAuthenticatedUserOrDie(crc); - DataverseRequest r = createDataverseRequest(u); execCommand(new RegisterDvObjectCommand(r, df)); successes++; if (successes % 100 == 0) { logger.info(successes + " of " + count + " files registered successfully. " + new Date()); } + try { + Thread.sleep(1000); + } catch (InterruptedException ie) { + logger.warning("Interrupted Exception when attempting to execute Thread.sleep()!"); + } } else { draft++; - logger.info(draft + " of " + count + " files not yet published"); + if (draft % 100 == 0) { + logger.info(draft + " of " + count + " files not yet published"); + } } } else { alreadyRegistered++; - logger.info(alreadyRegistered + " of " + count + " files are already registered. " + new Date()); + if(alreadyRegistered % 100 == 0) { + logger.info(alreadyRegistered + " of " + count + " files are already registered. " + new Date()); + } } } catch (WrappedResponse ex) { - released++; logger.info("Failed to register file id: " + df.getId()); Logger.getLogger(Datasets.class.getName()).log(Level.SEVERE, null, ex); } catch (Exception e) { logger.info("Unexpected Exception: " + e.getMessage()); } - try { - Thread.sleep(1000); - } catch (InterruptedException ie) { - logger.warning("Interrupted Exception when attempting to execute Thread.sleep()!"); - } + } logger.info("Final Results:"); logger.info(alreadyRegistered + " of " + count + " files were already registered. " + new Date()); @@ -1579,6 +1597,7 @@ public Response registerDataFileAll(@Context ContainerRequestContext crc) { logger.info(released + " of " + count + " unregistered, published files to register. " + new Date()); logger.info(successes + " of " + released + " unregistered, published files registered successfully. " + new Date()); + logger.info(skipped + " of " + count + " files not in collections that allow file PIDs. " + new Date()); return ok("Datafile registration complete." + successes + " of " + released + " unregistered, published files registered successfully."); @@ -1633,6 +1652,11 @@ public Response registerDataFilesInCollection(@Context ContainerRequestContext c if (countSuccesses % 100 == 0) { logger.info(countSuccesses + " out of " + count + " files registered successfully. " + new Date()); } + try { + Thread.sleep(sleepInterval * 1000); + } catch (InterruptedException ie) { + logger.warning("Interrupted Exception when attempting to execute Thread.sleep()!"); + } } else { countDrafts++; logger.fine(countDrafts + " out of " + count + " files not yet published"); @@ -1648,12 +1672,6 @@ public Response registerDataFilesInCollection(@Context ContainerRequestContext c } catch (Exception e) { logger.info("Unexpected Exception: " + e.getMessage()); } - - try { - Thread.sleep(sleepInterval * 1000); - } catch (InterruptedException ie) { - logger.warning("Interrupted Exception when attempting to execute Thread.sleep()!"); - } } logger.info(countAlreadyRegistered + " out of " + count + " files were already registered. " + new Date()); diff --git a/src/main/java/edu/harvard/iq/dataverse/api/Dataverses.java b/src/main/java/edu/harvard/iq/dataverse/api/Dataverses.java index 70199c64fa4..5214e3fbdcb 100644 --- a/src/main/java/edu/harvard/iq/dataverse/api/Dataverses.java +++ b/src/main/java/edu/harvard/iq/dataverse/api/Dataverses.java @@ -636,6 +636,12 @@ public Response updateAttribute(@Context ContainerRequestContext crc, @PathParam break; */ case "filePIDsEnabled": + if(!user.isSuperuser()) { + return forbidden("You must be a superuser to change this setting"); + } + if(!settingsService.isTrueForKey(SettingsServiceBean.Key.AllowEnablingFilePIDsPerCollection, false)) { + return forbidden("Changing File PID policy per collection is not enabled on this server"); + } collection.setFilePIDsEnabled(parseBooleanOrDie(value)); break; default: diff --git a/src/main/java/edu/harvard/iq/dataverse/batch/jobs/importer/filesystem/FileRecordWriter.java b/src/main/java/edu/harvard/iq/dataverse/batch/jobs/importer/filesystem/FileRecordWriter.java index 291396b4d33..195ed57bd43 100644 --- a/src/main/java/edu/harvard/iq/dataverse/batch/jobs/importer/filesystem/FileRecordWriter.java +++ b/src/main/java/edu/harvard/iq/dataverse/batch/jobs/importer/filesystem/FileRecordWriter.java @@ -358,9 +358,8 @@ private DataFile createPackageDataFile(List files) { dataset.getLatestVersion().getFileMetadatas().add(fmd); fmd.setDatasetVersion(dataset.getLatestVersion()); - String isFilePIDsEnabled = commandEngine.getContext().settings().getValueForKey(SettingsServiceBean.Key.FilePIDsEnabled, "true"); //default value for file PIDs is 'true' - if ("true".contentEquals( isFilePIDsEnabled )) { - + if (commandEngine.getContext().systemConfig().isFilePIDsEnabledForCollection(dataset.getOwner())) { + GlobalIdServiceBean idServiceBean = GlobalIdServiceBean.getBean(packageFile.getProtocol(), commandEngine.getContext()); if (packageFile.getIdentifier() == null || packageFile.getIdentifier().isEmpty()) { packageFile.setIdentifier(idServiceBean.generateDataFileIdentifier(packageFile)); diff --git a/src/main/java/edu/harvard/iq/dataverse/dataset/DatasetUtil.java b/src/main/java/edu/harvard/iq/dataverse/dataset/DatasetUtil.java index a75775810d9..adbd132bce8 100644 --- a/src/main/java/edu/harvard/iq/dataverse/dataset/DatasetUtil.java +++ b/src/main/java/edu/harvard/iq/dataverse/dataset/DatasetUtil.java @@ -7,6 +7,7 @@ import edu.harvard.iq.dataverse.FileMetadata; import edu.harvard.iq.dataverse.TermsOfUseAndAccess; import edu.harvard.iq.dataverse.authorization.groups.impl.ipaddress.ip.IpAddress; +import edu.harvard.iq.dataverse.authorization.users.AuthenticatedUser; import edu.harvard.iq.dataverse.dataaccess.DataAccess; import static edu.harvard.iq.dataverse.dataaccess.DataAccess.getStorageIO; import edu.harvard.iq.dataverse.dataaccess.StorageIO; @@ -497,18 +498,34 @@ public static Long getDownloadSizeNumericBySelectedFiles(List file public static boolean validateDatasetMetadataExternally(Dataset ds, String executable, DataverseRequest request) { String sourceAddressLabel = "0.0.0.0"; + String userIdentifier = "guest"; if (request != null) { IpAddress sourceAddress = request.getSourceAddress(); if (sourceAddress != null) { sourceAddressLabel = sourceAddress.toString(); } + + AuthenticatedUser user = request.getAuthenticatedUser(); + + if (user != null) { + userIdentifier = user.getUserIdentifier(); + } } String jsonMetadata; + // We are sending the dataset metadata encoded in our standard json + // format, with a couple of extra elements added, such as the ids of + // the home collection and the user, in order to make it easier + // for the filter to whitelist by these attributes. + try { - jsonMetadata = json(ds).add("datasetVersion", json(ds.getLatestVersion())).add("sourceAddress", sourceAddressLabel).build().toString(); + jsonMetadata = json(ds).add("datasetVersion", json(ds.getLatestVersion())) + .add("sourceAddress", sourceAddressLabel) + .add("userIdentifier", userIdentifier) + .add("parentAlias", ds.getOwner().getAlias()) + .build().toString(); } catch (Exception ex) { logger.warning("Failed to export dataset metadata as json; "+ex.getMessage() == null ? "" : ex.getMessage()); return false; @@ -524,7 +541,7 @@ public static boolean validateDatasetMetadataExternally(Dataset ds, String execu try { File tempFile = File.createTempFile("datasetMetadataCheck", ".tmp"); FileUtils.writeStringToFile(tempFile, jsonMetadata); - + // run the external executable: String[] params = { executable, tempFile.getAbsolutePath() }; Process p = Runtime.getRuntime().exec(params); diff --git a/src/main/java/edu/harvard/iq/dataverse/dataverse/DataverseUtil.java b/src/main/java/edu/harvard/iq/dataverse/dataverse/DataverseUtil.java index 04e7d56fad6..52a9501aefb 100644 --- a/src/main/java/edu/harvard/iq/dataverse/dataverse/DataverseUtil.java +++ b/src/main/java/edu/harvard/iq/dataverse/dataverse/DataverseUtil.java @@ -4,6 +4,7 @@ import edu.harvard.iq.dataverse.Dataverse; import edu.harvard.iq.dataverse.DvObjectContainer; import edu.harvard.iq.dataverse.authorization.groups.impl.ipaddress.ip.IpAddress; +import edu.harvard.iq.dataverse.authorization.users.AuthenticatedUser; import edu.harvard.iq.dataverse.authorization.users.User; import edu.harvard.iq.dataverse.engine.command.DataverseRequest; import edu.harvard.iq.dataverse.util.BundleUtil; @@ -37,16 +38,32 @@ public static boolean validateDataverseMetadataExternally(Dataverse dv, String e String jsonMetadata; String sourceAddressLabel = "0.0.0.0"; + String userIdentifier = "guest"; + String parentAlias = dv.getOwner() == null ? "" : dv.getOwner().getAlias(); if (request != null) { IpAddress sourceAddress = request.getSourceAddress(); if (sourceAddress != null) { sourceAddressLabel = sourceAddress.toString(); } + AuthenticatedUser user = request.getAuthenticatedUser(); + + if (user != null) { + userIdentifier = user.getUserIdentifier(); + } } - + + // We are sending the collection metadata encoded in our standard json + // format, with a couple of extra elements added, such as the id of + // the user sending the request and the alias of the parent collection, + // in order to make it easier for the filter to manage whitelisting. + try { - jsonMetadata = json(dv).add("sourceAddress", sourceAddressLabel).build().toString(); + jsonMetadata = json(dv) + .add("sourceAddress", sourceAddressLabel) + .add("userIdentifier", userIdentifier) + .add("parentAlias", parentAlias) + .build().toString(); } catch (Exception ex) { logger.warning( "Failed to export dataverse metadata as json; " + ex.getMessage() == null ? "" : ex.getMessage()); diff --git a/src/main/java/edu/harvard/iq/dataverse/engine/command/impl/CreateDataverseCommand.java b/src/main/java/edu/harvard/iq/dataverse/engine/command/impl/CreateDataverseCommand.java index cece4230eed..3efefe90681 100644 --- a/src/main/java/edu/harvard/iq/dataverse/engine/command/impl/CreateDataverseCommand.java +++ b/src/main/java/edu/harvard/iq/dataverse/engine/command/impl/CreateDataverseCommand.java @@ -97,6 +97,10 @@ public Dataverse execute(CommandContext ctxt) throws CommandException { if (ctxt.dataverses().findByAlias(created.getAlias()) != null) { throw new IllegalCommandException("A dataverse with alias " + created.getAlias() + " already exists", this); } + + if(created.getFilePIDsEnabled()!=null && !ctxt.settings().isTrueForKey(SettingsServiceBean.Key.AllowEnablingFilePIDsPerCollection, false)) { + throw new IllegalCommandException("File PIDs cannot be enabled per collection", this); + } // Save the dataverse Dataverse managedDv = ctxt.dataverses().save(created); diff --git a/src/main/java/edu/harvard/iq/dataverse/settings/SettingsServiceBean.java b/src/main/java/edu/harvard/iq/dataverse/settings/SettingsServiceBean.java index d1ed3191556..ead0ac3bd39 100644 --- a/src/main/java/edu/harvard/iq/dataverse/settings/SettingsServiceBean.java +++ b/src/main/java/edu/harvard/iq/dataverse/settings/SettingsServiceBean.java @@ -595,7 +595,11 @@ Whether Harvesting (OAI) service is enabled * True/false(default) option deciding whether the dataset file table display should include checkboxes * allowing users to dynamically turn folder and category ordering on/off. */ - AllowUserManagementOfOrder + AllowUserManagementOfOrder, + /* + * True/false(default) option deciding whether file PIDs can be enabled per collection - using the Dataverse/collection set attribute API call. + */ + AllowEnablingFilePIDsPerCollection ; @Override diff --git a/src/main/java/edu/harvard/iq/dataverse/util/SystemConfig.java b/src/main/java/edu/harvard/iq/dataverse/util/SystemConfig.java index 45f7f396783..b9459e748bf 100644 --- a/src/main/java/edu/harvard/iq/dataverse/util/SystemConfig.java +++ b/src/main/java/edu/harvard/iq/dataverse/util/SystemConfig.java @@ -1011,7 +1011,7 @@ public boolean isFilePIDsEnabledForCollection(Dataverse collection) { // hasn't been explicitly enabled, therefore we presume that it is // subject to how the registration is configured for the // entire instance: - return settingsService.isTrueForKey(SettingsServiceBean.Key.FilePIDsEnabled, true); + return settingsService.isTrueForKey(SettingsServiceBean.Key.FilePIDsEnabled, false); } thisCollection = thisCollection.getOwner(); } diff --git a/src/main/webapp/dataset.xhtml b/src/main/webapp/dataset.xhtml index 4d0c1485335..67dcf89c380 100644 --- a/src/main/webapp/dataset.xhtml +++ b/src/main/webapp/dataset.xhtml @@ -315,13 +315,13 @@
- + - + - + diff --git a/src/main/webapp/filesFragment.xhtml b/src/main/webapp/filesFragment.xhtml index 896a7cae3f0..ddb5297ef7d 100644 --- a/src/main/webapp/filesFragment.xhtml +++ b/src/main/webapp/filesFragment.xhtml @@ -437,7 +437,7 @@
+ and !DatasetPage.isVersionHasTabular()}"> + and DatasetPage.isVersionHasTabular()}"> diff --git a/src/test/java/edu/harvard/iq/dataverse/api/FilesIT.java b/src/test/java/edu/harvard/iq/dataverse/api/FilesIT.java index 43dc1381221..5321159d1fb 100644 --- a/src/test/java/edu/harvard/iq/dataverse/api/FilesIT.java +++ b/src/test/java/edu/harvard/iq/dataverse/api/FilesIT.java @@ -2025,70 +2025,80 @@ public void testDeleteFile() { @Test public void testFilePIDsBehavior() { // Create user - String apiToken = createUserGetToken(); + Response createUser = UtilIT.createRandomUser(); + String apiToken = UtilIT.getApiTokenFromResponse(createUser); + String username = UtilIT.getUsernameFromResponse(createUser); + Response toggleSuperuser = UtilIT.makeSuperUser(username); + toggleSuperuser.then().assertThat() + .statusCode(OK.getStatusCode()); + try { + UtilIT.enableSetting(SettingsServiceBean.Key.FilePIDsEnabled); - // Create Dataverse - String collectionAlias = createDataverseGetAlias(apiToken); + // Create Dataverse + String collectionAlias = createDataverseGetAlias(apiToken); - // Create Initial Dataset with 1 file: - Integer datasetId = createDatasetGetId(collectionAlias, apiToken); - String pathToFile = "scripts/search/data/replace_test/003.txt"; - Response addResponse = UtilIT.uploadFileViaNative(datasetId.toString(), pathToFile, apiToken); + // Create Initial Dataset with 1 file: + Integer datasetId = createDatasetGetId(collectionAlias, apiToken); + String pathToFile = "scripts/search/data/replace_test/003.txt"; + Response addResponse = UtilIT.uploadFileViaNative(datasetId.toString(), pathToFile, apiToken); - addResponse.then().assertThat() - .body("data.files[0].dataFile.contentType", equalTo("text/plain")) - .body("data.files[0].label", equalTo("003.txt")) - .statusCode(OK.getStatusCode()); - - Long origFileId = JsonPath.from(addResponse.body().asString()).getLong("data.files[0].dataFile.id"); - - // ------------------------- - // Publish dataverse and dataset - // ------------------------- - msg("Publish dataverse and dataset"); - Response publishCollectionResp = UtilIT.publishDataverseViaSword(collectionAlias, apiToken); - publishCollectionResp.then().assertThat() - .statusCode(OK.getStatusCode()); - - Response publishDatasetResp = UtilIT.publishDatasetViaNativeApi(datasetId, "major", apiToken); - publishDatasetResp.then().assertThat() - .statusCode(OK.getStatusCode()); - - // The file in this dataset should have been assigned a PID when it was published: - Response fileInfoResponse = UtilIT.getFileData(origFileId.toString(), apiToken); - fileInfoResponse.then().assertThat().statusCode(OK.getStatusCode()); - String fileInfoResponseString = fileInfoResponse.body().asString(); - msg(fileInfoResponseString); - - String origFilePersistentId = JsonPath.from(fileInfoResponseString).getString("data.dataFile.persistentId"); - assertNotNull("The file did not get a persistent identifier assigned (check that file PIDs are enabled instance-wide!)", origFilePersistentId); + addResponse.then().assertThat().body("data.files[0].dataFile.contentType", equalTo("text/plain")) + .body("data.files[0].label", equalTo("003.txt")).statusCode(OK.getStatusCode()); - // Now change the file PIDs registration configuration for the collection: - - Response changeAttributeResp = UtilIT.setCollectionAttribute(collectionAlias, "filePIDsEnabled", "false", apiToken); - - // ... And do the whole thing with creating another dataset with a file: - - datasetId = createDatasetGetId(collectionAlias, apiToken); - addResponse = UtilIT.uploadFileViaNative(datasetId.toString(), pathToFile, apiToken); - addResponse.then().assertThat().statusCode(OK.getStatusCode()); - Long newFileId = JsonPath.from(addResponse.body().asString()).getLong("data.files[0].dataFile.id"); - - // And publish this dataset: - msg("Publish second dataset"); - - publishDatasetResp = UtilIT.publishDatasetViaNativeApi(datasetId, "major", apiToken); - publishDatasetResp.then().assertThat() - .statusCode(OK.getStatusCode()); - - // And confirm that the file didn't get a PID: - - fileInfoResponse = UtilIT.getFileData(newFileId.toString(), apiToken); - fileInfoResponse.then().assertThat().statusCode(OK.getStatusCode()); - fileInfoResponseString = fileInfoResponse.body().asString(); - msg(fileInfoResponseString); - - org.junit.Assert.assertEquals("The file was NOT supposed to be issued a PID", "", JsonPath.from(fileInfoResponseString).getString("data.dataFile.persistentId")); - + Long origFileId = JsonPath.from(addResponse.body().asString()).getLong("data.files[0].dataFile.id"); + + // ------------------------- + // Publish dataverse and dataset + // ------------------------- + msg("Publish dataverse and dataset"); + Response publishCollectionResp = UtilIT.publishDataverseViaSword(collectionAlias, apiToken); + publishCollectionResp.then().assertThat().statusCode(OK.getStatusCode()); + + Response publishDatasetResp = UtilIT.publishDatasetViaNativeApi(datasetId, "major", apiToken); + publishDatasetResp.then().assertThat().statusCode(OK.getStatusCode()); + + // The file in this dataset should have been assigned a PID when it was + // published: + Response fileInfoResponse = UtilIT.getFileData(origFileId.toString(), apiToken); + fileInfoResponse.then().assertThat().statusCode(OK.getStatusCode()); + String fileInfoResponseString = fileInfoResponse.body().asString(); + msg(fileInfoResponseString); + + String origFilePersistentId = JsonPath.from(fileInfoResponseString).getString("data.dataFile.persistentId"); + assertNotNull( + "The file did not get a persistent identifier assigned (check that file PIDs are enabled instance-wide!)", + origFilePersistentId); + + // Now change the file PIDs registration configuration for the collection: + UtilIT.enableSetting(SettingsServiceBean.Key.AllowEnablingFilePIDsPerCollection); + Response changeAttributeResp = UtilIT.setCollectionAttribute(collectionAlias, "filePIDsEnabled", "false", + apiToken); + + // ... And do the whole thing with creating another dataset with a file: + + datasetId = createDatasetGetId(collectionAlias, apiToken); + addResponse = UtilIT.uploadFileViaNative(datasetId.toString(), pathToFile, apiToken); + addResponse.then().assertThat().statusCode(OK.getStatusCode()); + Long newFileId = JsonPath.from(addResponse.body().asString()).getLong("data.files[0].dataFile.id"); + + // And publish this dataset: + msg("Publish second dataset"); + + publishDatasetResp = UtilIT.publishDatasetViaNativeApi(datasetId, "major", apiToken); + publishDatasetResp.then().assertThat().statusCode(OK.getStatusCode()); + + // And confirm that the file didn't get a PID: + + fileInfoResponse = UtilIT.getFileData(newFileId.toString(), apiToken); + fileInfoResponse.then().assertThat().statusCode(OK.getStatusCode()); + fileInfoResponseString = fileInfoResponse.body().asString(); + msg(fileInfoResponseString); + + org.junit.Assert.assertEquals("The file was NOT supposed to be issued a PID", "", + JsonPath.from(fileInfoResponseString).getString("data.dataFile.persistentId")); + } finally { + UtilIT.deleteSetting(SettingsServiceBean.Key.FilePIDsEnabled); + UtilIT.deleteSetting(SettingsServiceBean.Key.AllowEnablingFilePIDsPerCollection); + } } }