Merge branch 'develop' into 8227-verify-email #8227

doc/release-notes/8191-require-toa-or-request-access.md

@@ -0,0 +1,17 @@
+## Release Highlights
+
+### Terms of Access or Request Access Required for Restricted Files
+
+Beginning in this release, Restricted Files must have either Terms of Access or Request Access enabled. This change is to ensure that for each file in a Dataverse installation there is a clear path to get to the data, either through requesting access to the data or to provide context about why requesting access is not enabled.
+
+In the "Notes for Dataverse Installation Administrators" section, we have provided a query to help proactively identify datasets that need to be updated.
+
+## Notes for Dataverse Installation Administrators
+
+### Identifying Datasets Requiring Terms of Access or Request Access Changes
+
+In support of the change to require either Terms of Access or Request Access for all Restricted Files, we have provided a query to identify datasets in your installation where at least one file has neither Terms of Access or Request Access enabled:
+
+https://github.com/IQSS/dataverse/raw/develop/scripts/issues/8191/
+
+This will allow you to reach out to those dataset owners as appropriate.

doc/release-notes/8380-counter-processor-update.md

@@ -0,0 +1 @@
+This release includes support for counter-processor-0.1.04 for processing Make Data Count metrics. If you are running Make Data Counts support, you should reinstall/reconfigure counter-processor as described in the latest Guides. (For existing installations, note that counter-processor-0.1.04 requires a newer version of python so you will need to follow the full counter-processor install. Also note that if you configure the new version the same way, it will reprocess the days in the current month when it is first run. This is normal and will not affect the metrics in Dataverse.)

doc/sphinx-guides/source/_static/util/counter_daily.sh

@@ -1,6 +1,6 @@
 #! /bin/bash
 
-COUNTER_PROCESSOR_DIRECTORY="/usr/local/counter-processor-0.0.1"
+COUNTER_PROCESSOR_DIRECTORY="/usr/local/counter-processor-0.1.04"
 MDC_LOG_DIRECTORY="/usr/local/payara5/glassfish/domains/domain1/logs/mdc"
 
 # counter_daily.sh

doc/sphinx-guides/source/admin/dataverses-datasets.rst

@@ -107,7 +107,7 @@ Move a Dataset
 
 Superusers can move datasets using the dashboard. See also :doc:`dashboard`.
 
-Moves a dataset whose id is passed to a Dataverse collection whose alias is passed. If the moved dataset has a guestbook or a Dataverse collection link that is not compatible with the destination Dataverse collection, you will be informed and given the option to force the move (with ``forceMove=true`` as a query parameter) and remove the guestbook or link (or both). Only accessible to users with permission to publish the dataset in the original and destination Dataverse collection. ::
+Moves a dataset whose id is passed to a Dataverse collection whose alias is passed. If the moved dataset has a guestbook or a Dataverse collection link that is not compatible with the destination Dataverse collection, you will be informed and given the option to force the move (with ``forceMove=true`` as a query parameter) and remove the guestbook or link (or both). Only accessible to users with permission to publish the dataset in the original and destination Dataverse collection. Note: any roles granted to users on the dataset will continue to be in effect after the dataset has been moved. ::
 
     curl -H "X-Dataverse-key: $API_TOKEN" -X POST http://$SERVER/api/datasets/$id/move/$alias
 

doc/sphinx-guides/source/admin/make-data-count.rst

@@ -83,9 +83,9 @@ Configure Counter Processor
 
 * Change to the directory where you installed Counter Processor.
 
-  * ``cd /usr/local/counter-processor-0.0.1``
+  * ``cd /usr/local/counter-processor-0.1.04``
 
-* Download :download:`counter-processor-config.yaml <../_static/admin/counter-processor-config.yaml>` to ``/usr/local/counter-processor-0.0.1``.
+* Download :download:`counter-processor-config.yaml <../_static/admin/counter-processor-config.yaml>` to ``/usr/local/counter-processor-0.1.04``.
 
 * Edit the config file and pay particular attention to the FIXME lines.
 
@@ -98,7 +98,7 @@ Soon we will be setting up a cron job to run nightly but we start with a single
 
 * Change to the directory where you installed Counter Processor.
 
-  * ``cd /usr/local/counter-processor-0.0.1``
+  * ``cd /usr/local/counter-processor-0.1.04``
 
 * If you are running Counter Processor for the first time in the middle of a month, you will need create blank log files for the previous days. e.g.:
 
@@ -112,7 +112,7 @@ Soon we will be setting up a cron job to run nightly but we start with a single
 
 * Run Counter Processor.
 
-  * ``CONFIG_FILE=counter-processor-config.yaml python36 main.py``
+  * ``CONFIG_FILE=counter-processor-config.yaml python39 main.py``
 
   * A JSON file in SUSHI format will be created in the directory you specified under "output_file" in the config file.
 

doc/sphinx-guides/source/developers/make-data-count.rst

@@ -51,9 +51,9 @@ Once you are done with your configuration, you can run Counter Processor like th
 
 ``su - counter``
 
-``cd /usr/local/counter-processor-0.0.1``
+``cd /usr/local/counter-processor-0.1.04``
 
-``CONFIG_FILE=counter-processor-config.yaml python36 main.py``
+``CONFIG_FILE=counter-processor-config.yaml python39 main.py``
 
 (Please note that the Counter Processor README says you can also pass in values like ``START_DATE``, ``END_DATE`` etc. at the command line if you find this to be more convenient.)
 
@@ -84,7 +84,7 @@ Second, if you are also sending your SUSHI report to Make Data Count, you will n
 
 ``curl -H "Authorization: Bearer $JSON_WEB_TOKEN" -X DELETE https://$MDC_SERVER/reports/$REPORT_ID``
 
-To get the ``REPORT_ID``, look at the logs generated in ``/usr/local/counter-processor-0.0.1/tmp/datacite_response_body.txt``
+To get the ``REPORT_ID``, look at the logs generated in ``/usr/local/counter-processor-0.1.04/tmp/datacite_response_body.txt``
 
 To read more about the Make Data Count api, see https://github.com/datacite/sashimi
 

doc/sphinx-guides/source/installation/prerequisites.rst

@@ -96,7 +96,7 @@ PostgreSQL
 ----------
 
 Installing PostgreSQL
-=======================
+=====================
 
 The application has been tested with PostgreSQL versions up to 13 and version 10+ is required. We recommend installing the latest version that is available for your OS distribution. *For example*, to install PostgreSQL 13 under RHEL7/derivative::
 
@@ -416,43 +416,51 @@ Counter Processor is required to enable Make Data Count metrics in a Dataverse i
 Installing Counter Processor
 ============================
 
-Counter Processor has only been tested on el7 (see "Linux" above). Please note that a scripted installation using Ansible is mentioned in the :doc:`/developers/make-data-count` section of the Developer Guide.
+A scripted installation using Ansible is mentioned in the :doc:`/developers/make-data-count` section of the Developer Guide.
 
 As root, download and install Counter Processor::
 
         cd /usr/local
-        wget https://github.com/CDLUC3/counter-processor/archive/v0.0.1.tar.gz
-        tar xvfz v0.0.1.tar.gz
+        wget https://github.com/CDLUC3/counter-processor/archive/v0.1.04.tar.gz
+        tar xvfz v0.1.04.tar.gz
+        cd /usr/local/counter-processor-0.1.04
+
+Installing GeoLite Country Database
+===================================
+
+Counter Processor can report per country results if the optional GeoLite Country Database is installed. At present, this database is free but to use it one must signing an agreement (EULA) with MaxMind. 
+(The primary concern appears to be that individuals can opt-out of having their location tracked via IP address and, due to various privacy laws, MaxMind needs a way to comply with that for products it has "sold" (for no cost in this case). Their agreement requires you to either configure automatic updates to the GeoLite Country database or be responsible on your own for managing take down notices.)
+The process required to sign up, download the database, and to configure automated updating is described at https://blog.maxmind.com/2019/12/18/significant-changes-to-accessing-and-using-geolite2-databases/ and the links from that page.
 
-As root, change to the Counter Processor directory you just created, download the GeoLite2-Country tarball, untar it, and copy the geoip database into place::
+As root, change to the Counter Processor directory you just created, download the GeoLite2-Country tarball from MaxMind, untar it, and copy the geoip database into place::
 
-        cd /usr/local/counter-processor-0.0.1
-        wget https://geolite.maxmind.com/download/geoip/database/GeoLite2-Country.tar.gz
+        <download or move the GeoLite2-Country.tar.gz to the /usr/local/counter-processor-0.1.04 directory>
         tar xvfz GeoLite2-Country.tar.gz
         cp GeoLite2-Country_*/GeoLite2-Country.mmdb maxmind_geoip
 
+Creating a counter User
+=======================
+
 As root, create a "counter" user and change ownership of Counter Processor directory to this new user::
 
         useradd counter
-        chown -R counter:counter /usr/local/counter-processor-0.0.1
+        chown -R counter:counter /usr/local/counter-processor-0.1.04
 
 Installing Counter Processor Python Requirements
 ================================================
 
-Counter Processor requires Python 3.6.4 or higher. The following commands are intended to be run as root but we are aware that Pythonistas might prefer fancy virtualenv or similar setups. Pull requests are welcome to improve these steps!
-
-Enable the EPEL repo if you haven't already::
+Counter Processor version 0.1.04 requires Python 3.7 or higher. This version of Python is available in many operating systems, and is purportedly available for RHEL7 or CentOS 7 via Red Hat Software Collections. Alternately, one may compile it from source.
 
-        yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
+The following commands are intended to be run as root but we are aware that Pythonistas might prefer fancy virtualenv or similar setups. Pull requests are welcome to improve these steps!
 
-Install Python 3.6::
+Install Python 3.9::
 
-        yum install python36
+        yum install python39
 
 Install Counter Processor Python requirements::
 
-        python3.6 -m ensurepip
-        cd /usr/local/counter-processor-0.0.1
+        python3.9 -m ensurepip
+        cd /usr/local/counter-processor-0.1.04
         pip3 install -r requirements.txt
 
 See the :doc:`/admin/make-data-count` section of the Admin Guide for how to configure and run Counter Processor.

scripts/issues/8191/datasets_without_toa_or_request_access

@@ -0,0 +1,28 @@
+-- this query will identify datasets where at least one file does not have either terms of access or request access enabled, and will include owner information for those datasets
+
+select au.email, 
+concat(au.firstname, ' ', au.lastname),
+concat('$SERVERNAME/dataset.xhtml?persistentId=doi:' , dvo.authority , '/' ,  dvo.identifier)
+from roleassignment ra, dataverserole dvr,
+authenticateduser au, dvobject dvo
+where
+au.useridentifier = rtrim(substring(ra.assigneeidentifier, 2, 100))
+and dvo.id = ra.definitionpoint_id
+and
+ra.role_id = dvr.id and
+dvr.alias in (
+	'fullContributor',
+'dsContributor',
+'contributor',
+'admin',
+'curator'
+) and
+ra.definitionpoint_id in (
+	select dvo.id  from datasetversion v
+join termsofuseandaccess ua on ua.id = v.termsofuseandaccess_id
+join filemetadata fm on v.id = fm.datasetversion_id
+join datafile f on f.id = fm.datafile_id
+join dvobject dvo on v.dataset_id = dvo.id
+where ua.fileaccessrequest = false and ua.termsofaccess isnull
+and f.restricted = true
+)

scripts/search/tests/data/dataset-finch1.jsonld

@@ -20,7 +20,7 @@
   "http://schema.org/version": "DRAFT",
   "http://schema.org/name": "Darwin's Finches",
     "https://dataverse.org/schema/core#fileTermsOfAccess": {
-        "https://dataverse.org/schema/core#fileRequestAccess": false
+        "https://dataverse.org/schema/core#fileRequestAccess": true
     },
     "http://schema.org/includedInDataCatalog": "Root"
 }

scripts/vagrant/counter-processor-config.yaml

@@ -44,7 +44,7 @@ output_file: /dataverse/sushi_sample_logs
 output_format: json
 
 # the name of the platform that goes into your reports
-platform: Dash
+platform: Dataverse
 
 # Don't put your api token in here if you're going to commit it, but put in separate secrets.yaml in same
 # directory as the config or else set a environment variable when starting up in order to override the key.

scripts/vagrant/setup-counter-processor.sh

@@ -11,7 +11,7 @@ python3.6 -m ensurepip
 COUNTER_USER=counter
 echo "Ensuring Unix user '$COUNTER_USER' exists"
 useradd $COUNTER_USER || :
-COMMIT='a73dbced06f0ac2f0d85231e4d9dd4f21bee8487'
+COMMIT='7974dad259465ba196ef639f48dea007cae8f9ac'
 UNZIPPED_DIR="counter-processor-$COMMIT"
 if [ ! -e $UNZIPPED_DIR ]; then
   ZIP_FILE="${COMMIT}.zip"
@@ -20,19 +20,7 @@ if [ ! -e $UNZIPPED_DIR ]; then
   unzip $ZIP_FILE
 fi
 cd $UNZIPPED_DIR
-GEOIP_DIR='maxmind_geoip'
-GEOIP_FILE='GeoLite2-Country.mmdb'
-GEOIP_PATH_TO_FILE="$GEOIP_DIR/$GEOIP_FILE"
-if [ ! -e $GEOIP_PATH_TO_FILE ]; then
-  echo "let's do this thing"
-  TARBALL='GeoLite2-Country.tar.gz'
-  wget https://geolite.maxmind.com/download/geoip/database/$TARBALL
-  tar xfz GeoLite2-Country.tar.gz
-  # Glob (*) below because of directories like "GeoLite2-Country_20181204".
-  GEOIP_UNTARRED_DIR='GeoLite2-Country_*'
-  mv $GEOIP_UNTARRED_DIR/$GEOIP_FILE $GEOIP_PATH_TO_FILE
-  rm -rf $TARBALL $GEOIP_UNTARRED_DIR
-fi
+echo "Installation of the GeoLite2 country database for counter-processor can no longer be automated. See the Installation Guide for the manual installation process."
 pip3 install -r requirements.txt
 # For now, parsing sample_logs/counter_2018-05-08.log
 for i in `echo {00..31}`; do

src/main/java/edu/harvard/iq/dataverse/Dataset.java

@@ -294,6 +294,7 @@ public void setVersions(List<DatasetVersion> versions) {
     }
 
     private DatasetVersion createNewDatasetVersion(Template template, FileMetadata fmVarMet) {
+
         DatasetVersion dsv = new DatasetVersion();
         dsv.setVersionState(DatasetVersion.VersionState.DRAFT);
         dsv.setFileMetadatas(new ArrayList<>());
@@ -313,11 +314,11 @@ private DatasetVersion createNewDatasetVersion(Template template, FileMetadata f
             if (latestVersion.getDatasetFields() != null && !latestVersion.getDatasetFields().isEmpty()) {
                 dsv.setDatasetFields(dsv.copyDatasetFields(latestVersion.getDatasetFields()));
             }
-
-            if (latestVersion.getTermsOfUseAndAccess()!= null){
-                dsv.setTermsOfUseAndAccess(latestVersion.getTermsOfUseAndAccess().copyTermsOfUseAndAccess());
-            }
-
+            /*
+            adding file metadatas here and updating terms
+            because the terms need to know about the files
+            in a pre-save validation SEK 12/6/2021
+            */
             for (FileMetadata fm : latestVersion.getFileMetadatas()) {
                 FileMetadata newFm = new FileMetadata();
                 // TODO: 
@@ -348,6 +349,18 @@ private DatasetVersion createNewDatasetVersion(Template template, FileMetadata f
 
                 dsv.getFileMetadatas().add(newFm);
             }
+
+            if (latestVersion.getTermsOfUseAndAccess()!= null){
+                TermsOfUseAndAccess terms = latestVersion.getTermsOfUseAndAccess().copyTermsOfUseAndAccess();
+                terms.setDatasetVersion(dsv);
+                dsv.setTermsOfUseAndAccess(terms);
+            } else {
+                TermsOfUseAndAccess terms = new TermsOfUseAndAccess();
+                terms.setDatasetVersion(dsv);
+                terms.setLicense(null);
+                terms.setFileAccessRequest(true);
+                dsv.setTermsOfUseAndAccess(terms);
+            }
         }
 
         // I'm adding the version to the list so it will be persisted when