Merge pull request #8046 from IQSS/develop

v5.6

.travis.yml.future → .github/workflows/cypress_ui.yml.future

@@ -1,16 +1,17 @@
+###############################################################################
+#
+#   THIS IS AN OLD TRAVIS-CI.ORG JOB FILE
+#   To be used with Github Actions, it would be necessary to refactor it.
+#   Keeping it as the future example it has been before.
+#   See also #5846
+#
+###############################################################################
+
 services:
   - docker
 
 jobs:
   include:
-    # Execute java unit- and integration tests
-    - stage: test
-      language: java
-      jdk:
-        - oraclejdk8
-      script: mvn -DcompilerArgument=-Xlint:unchecked test -P all-unit-tests
-      after_success: mvn jacoco:report coveralls:report
-
     # Execute Cypress for UI testing
     # see https://docs.cypress.io/guides/guides/continuous-integration.html
     - stage: test

.github/workflows/maven_unit_test.yml

@@ -0,0 +1,42 @@
+name: Maven Unit Tests
+
+on:
+    push:
+        paths:
+            - "**.java"
+    pull_request:
+        paths:
+            - "**.java"
+
+jobs:
+    unittest:
+        name: (JDK ${{ matrix.jdk }} / ${{ matrix.os }}) Unit Tests
+        strategy:
+            fail-fast: false
+            matrix:
+                os: [ ubuntu-latest ]
+                jdk: [ '11' ]
+                #include:
+                #    -   os: ubuntu-latest
+                #        jdk: '16'
+        runs-on: ${{ matrix.os }}
+        steps:
+          - uses: actions/checkout@v2
+          - name: Set up JDK ${{ matrix.jdk }}
+            uses: actions/setup-java@v2
+            with:
+                java-version: ${{ matrix.jdk }}
+                distribution: 'adopt'
+          - name: Cache Maven packages
+            uses: actions/cache@v2
+            with:
+                path: ~/.m2
+                key: ${{ runner.os }}-m2-${{ hashFiles('**/pom.xml') }}
+                restore-keys: ${{ runner.os }}-m2
+          - name: Build with Maven
+            run: mvn -DcompilerArgument=-Xlint:unchecked -P all-unit-tests clean test
+          - name: Maven Code Coverage
+            env:
+                CI_NAME: github
+                COVERALLS_SECRET: ${{ secrets.GITHUB_TOKEN }}
+            run: mvn -V -B jacoco:report coveralls:report -DrepoToken=${COVERALLS_SECRET} -DpullRequest=${{ github.event.number }}

.github/workflows/reviewdog_checkstyle.yml

@@ -0,0 +1,21 @@
+name: Maven CheckStyle Task
+on:
+   pull_request:
+       paths:
+           - "**.java"
+
+jobs:
+    checkstyle_job:
+        runs-on: ubuntu-latest
+        name: Checkstyle job
+        steps:
+            - name: Checkout
+              uses: actions/checkout@v2
+            - name: Run check style
+              uses: nikitasavinov/checkstyle-action@master
+              with:
+                  fail_on_error: true
+                  reporter: github-pr-review
+                  checkstyle_config: checkstyle.xml
+                  github_token: ${{ secrets.GITHUB_TOKEN }}
+

CONTRIBUTING.md

@@ -6,12 +6,11 @@ We aren't just looking for developers. There are many ways to contribute to Data
 
 ## Ideas/Feature Requests
 
-Your idea or feature request might already be captured in the Dataverse [issue tracker] on GitHub but if not, the best way to bring it to the community's attention is by posting on the [dataverse-community Google Group][] or bringing it up on a [Community Call][]. You're also welcome make some noise in the [#dataverse IRC channel][] (which is [logged][]) or cram your idea into 280 characters and mention [@dataverseorg][] on Twitter. To discuss your idea privately, please email it to support@dataverse.org
+Your idea or feature request might already be captured in the Dataverse [issue tracker] on GitHub but if not, the best way to bring it to the community's attention is by posting on the [dataverse-community Google Group][] or bringing it up on a [Community Call][]. You're also welcome to make some noise in [chat.dataverse.org][] or cram your idea into 280 characters and mention [@dataverseorg][] on Twitter. To discuss your idea privately, please email it to support@dataverse.org
 
 There's a chance your idea is already on our roadmap, which is available at https://www.iq.harvard.edu/roadmap-dataverse-project
 
-[#dataverse IRC channel]: http://chat.dataverse.org
-[logged]: http://irclog.iq.harvard.edu/dataverse/today
+[chat.dataverse.org]: http://chat.dataverse.org
 [issue tracker]: https://github.com/IQSS/dataverse/issues
 [@dataverseorg]: https://twitter.com/dataverseorg
 
@@ -55,7 +54,7 @@ We love code contributions. Developers are not limited to the main Dataverse cod
 [API Guide]: http://guides.dataverse.org/en/latest/api
 [Installation Guide]: http://guides.dataverse.org/en/latest/installation
 
-If you are interested in working on the main Dataverse code, great! Before you start coding, please reach out to us either on the [dataverse-community Google Group][], the [dataverse-dev Google Group][], [IRC][] (#dataverse on freenode), or via support@dataverse.org to make sure the effort is well coordinated and we avoid merge conflicts. We maintain a list of [community contributors][] and [dev efforts][] the community is working on so please let us know if you'd like to be added or removed from either list.
+If you are interested in working on the main Dataverse code, great! Before you start coding, please reach out to us either on the [dataverse-community Google Group][], the [dataverse-dev Google Group][], [chat.dataverse.org][], or via support@dataverse.org to make sure the effort is well coordinated and we avoid merge conflicts. We maintain a list of [community contributors][] and [dev efforts][] the community is working on so please let us know if you'd like to be added or removed from either list.
 
 Please read http://guides.dataverse.org/en/latest/developers/version-control.html to understand how we use the "git flow" model of development and how we will encourage you to create a GitHub issue (if it doesn't exist already) to associate with your pull request. That page also includes tips on making a pull request.
 
@@ -66,6 +65,5 @@ Thanks for your contribution!
 [dataverse-community Google Group]: https://groups.google.com/group/dataverse-community
 [Community Call]: https://dataverse.org/community-calls
 [dataverse-dev Google Group]: https://groups.google.com/group/dataverse-dev
-[IRC]: http://chat.dataverse.org
 [community contributors]: https://docs.google.com/spreadsheets/d/1o9DD-MQ0WkrYaEFTD5rF_NtyL8aUISgURsAXSL7Budk/edit?usp=sharing
 [dev efforts]: https://github.com/orgs/IQSS/projects/2#column-5298405

README.md

@@ -19,7 +19,7 @@ Dataverse is a trademark of President and Fellows of Harvard College and is regi
 
 [![API Test Status](https://jenkins.dataverse.org/buildStatus/icon?job=IQSS-dataverse-develop&subject=API%20Test%20Status)](https://jenkins.dataverse.org/job/IQSS-dataverse-develop/)
 [![API Test Coverage](https://img.shields.io/jenkins/coverage/jacoco?jobUrl=https%3A%2F%2Fjenkins.dataverse.org%2Fjob%2FIQSS-dataverse-develop&label=API%20Test%20Coverage)](https://jenkins.dataverse.org/job/IQSS-dataverse-develop/ws/target/coverage-it/index.html)
-[![Unit Test Status](https://img.shields.io/travis/IQSS/dataverse?label=Unit%20Test%20Status)](https://travis-ci.org/IQSS/dataverse)
+[![Unit Test Status](https://github.com/IQSS/dataverse/actions/workflows/maven_unit_test.yml/badge.svg?branch=develop)](https://github.com/IQSS/dataverse/actions/workflows/maven_unit_test.yml)
 [![Unit Test Coverage](https://img.shields.io/coveralls/github/IQSS/dataverse?label=Unit%20Test%20Coverage)](https://coveralls.io/github/IQSS/dataverse?branch=develop)
 [![Guides Build Status](https://github.com/IQSS/dataverse/actions/workflows/guides_build_sphinx.yml/badge.svg)](https://github.com/IQSS/dataverse/actions/workflows/guides_build_sphinx.yml)
 

checkstyle.xml

@@ -97,7 +97,9 @@
             <property name="excludes" value="lombok,java.util,org.springframework.web.bind.annotation"/>
         </module>
         -->
-        <!-- <module name="IllegalImport"/> --> <!-- defaults to sun.* packages -->
+        <module name="IllegalImport">
+            <property name="illegalPkgs" value="org.apache.commons.lang"/>
+        </module>
         <!-- <module name="RedundantImport"/> -->
         <!-- <module name="UnusedImports">
             <property name="processJavadoc" value="false"/>

conf/docker-aio/0prep_deps.sh

@@ -4,10 +4,10 @@ if [ ! -d dv/deps ]; then
 fi
 wdir=`pwd`
 
-if [ ! -e dv/deps/payara-5.2020.6.zip ]; then
+if [ ! -e dv/deps/payara-5.2021.5.zip ]; then
 	echo "payara dependency prep"
 	# no more fiddly patching :)
-	wget https://s3-eu-west-1.amazonaws.com/payara.fish/Payara+Downloads/5.2020.6/payara-5.2020.6.zip  -O dv/deps/payara-5.2020.6.zip
+	wget https://s3-eu-west-1.amazonaws.com/payara.fish/Payara+Downloads/5.2021.5/payara-5.2021.5.zip  -O dv/deps/payara-5.2021.5.zip
 fi
 
 if [ ! -e dv/deps/solr-8.8.1dv.tgz ]; then

conf/docker-aio/c8.dockerfile

@@ -1,10 +1,11 @@
-FROM centos:8
+FROM rockylinux/rockylinux:latest
 # OS dependencies
-# PG 10 is the default in centos8; keep the repo comment for when we bump to 11+
-#RUN yum install -y https://download.postgresql.org/pub/repos/yum/reporpms/EL-8-x86_64/pgdg-redhat-repo-latest.noarch.rpm
+# IQSS now recommends Postgres 13.
+RUN dnf -qy module disable postgresql
+RUN yum install -y https://download.postgresql.org/pub/repos/yum/reporpms/EL-8-x86_64/pgdg-redhat-repo-latest.noarch.rpm
 
 RUN echo "fastestmirror=true" >> /etc/dnf/dnf.conf
-RUN yum install -y java-11-openjdk-devel postgresql-server sudo epel-release unzip curl httpd python2 diffutils
+RUN yum install -y java-11-openjdk-devel postgresql13-server sudo epel-release unzip curl httpd python2 diffutils
 RUN yum install -y jq lsof awscli
 
 # for older search scripts
@@ -23,16 +24,16 @@ COPY disableipv6.conf /etc/sysctl.d/
 RUN rm /etc/httpd/conf/*
 COPY httpd.conf /etc/httpd/conf 
 RUN cd /opt ; tar zxf /tmp/dv/deps/solr-8.8.1dv.tgz 
-RUN cd /opt ; unzip /tmp/dv/deps/payara-5.2020.6.zip ; ln -s /opt/payara5 /opt/glassfish4
+RUN cd /opt ; unzip /tmp/dv/deps/payara-5.2021.5.zip ; ln -s /opt/payara5 /opt/glassfish4
 
 # this copy of domain.xml is the result of running `asadmin set server.monitoring-service.module-monitoring-levels.jvm=LOW` on a default glassfish installation (aka - enable the glassfish REST monitir endpoint for the jvm`
 # this dies under Java 11, do we keep it?
 #COPY domain-restmonitor.xml /opt/payara5/glassfish/domains/domain1/config/domain.xml
 
-RUN sudo -u postgres /usr/bin/initdb /var/lib/pgsql/data
+RUN sudo -u postgres /usr/pgsql-13/bin/initdb -D /var/lib/pgsql/13/data
 
 # copy configuration related files
-RUN cp /tmp/dv/pg_hba.conf /var/lib/pgsql/data/
+RUN cp /tmp/dv/pg_hba.conf /var/lib/pgsql/13/data/
 RUN cp -r /opt/solr-8.8.1/server/solr/configsets/_default /opt/solr-8.8.1/server/solr/collection1
 RUN cp /tmp/dv/schema*.xml /opt/solr-8.8.1/server/solr/collection1/conf/
 RUN cp /tmp/dv/solrconfig.xml /opt/solr-8.8.1/server/solr/collection1/conf/solrconfig.xml

conf/docker-aio/entrypoint.bash

@@ -1,6 +1,6 @@
 #!/usr/bin/env bash
 export LANG=en_US.UTF-8
-sudo -u postgres /usr/bin/pg_ctl start -D /var/lib/pgsql/data &
+sudo -u postgres /usr/pgsql-13/bin/pg_ctl start -D /var/lib/pgsql/13/data &
 cd /opt/solr-8.8.1/
 # TODO: Run Solr as non-root and remove "-force".
 bin/solr start -force
@@ -9,8 +9,8 @@ bin/solr create_core -c collection1 -d server/solr/collection1/conf -force
 # start apache, in both foreground and background...
 apachectl -DFOREGROUND &
 
-# TODO: Run Glassfish as non-root.
-cd /opt/glassfish4
+# TODO: Run Payara as non-root.
+cd /opt/payara5
 bin/asadmin start-domain --debug
 sleep infinity
 

conf/docker-aio/run-test-suite.sh

@@ -6,6 +6,8 @@ if [ -z "$dvurl" ]; then
 	dvurl="http://localhost:8084"
 fi
 
+integrationtests=$(<tests/integration-tests.txt)
+
 # Please note the "dataverse.test.baseurl" is set to run for "all-in-one" Docker environment.
 # TODO: Rather than hard-coding the list of "IT" classes here, add a profile to pom.xml.
-source maven/maven.sh && mvn test -Dtest=DataversesIT,DatasetsIT,SwordIT,AdminIT,BuiltinUsersIT,UsersIT,UtilIT,ConfirmEmailIT,FileMetadataIT,FilesIT,SearchIT,InReviewWorkflowIT,HarvestingServerIT,MoveIT,MakeDataCountApiIT,FileTypeDetectionIT,EditDDIIT,ExternalToolsIT,AccessIT,DuplicateFilesIT,DownloadFilesIT,LinkIT,DeleteUsersIT,DeactivateUsersIT,AuxiliaryFilesIT -Ddataverse.test.baseurl=$dvurl
+mvn test -Dtest=$integrationtests -Ddataverse.test.baseurl=$dvurl

doc/release-notes/5.0-release-notes.md

@@ -313,7 +313,7 @@ The instructions below describe the upgrade procedure based on moving an existin
 
 1. Update Astrophysics Metadata Block (if used)
 
-   `wget https://github.com/IQSS/dataverse/releases/download/5.0/astrophysics.tsv`
+   `wget https://github.com/IQSS/dataverse/releases/download/v5.0/astrophysics.tsv`
    `curl http://localhost:8080/api/admin/datasetfield/load -X POST --data-binary @astrophysics.tsv -H "Content-type: text/tab-separated-values"` 
 
 2. (Recommended) Run ReExportall to update JSON Exports  

doc/release-notes/5.3-release-notes.md

@@ -124,7 +124,6 @@ If you are still using pre-5.0 version of Dataverse, and Glassfish version 4, pl
 3\. Update your database connection.
 
 Please configure your connection details, replacing all the `${DB_...}`.
-(If you are using a PostgreSQL server on `localhost:5432`, you can omit `dataverse.db.host` and `dataverse.db.port`.)
 
 - `<payara install path>/bin/asadmin create-system-properties "dataverse.db.user=${DB_USER}"`
 - `<payara install path>/bin/asadmin create-system-properties "dataverse.db.host=${DB_HOST}"`

doc/release-notes/5.4-release-notes.md

@@ -94,6 +94,11 @@ There are some things to note and keep in mind regarding the move to Java 11:
     `$ sudo yum install java-11-openjdk-devel`  
 
     The `remove` command may provide an error message if -headless isn't installed.
+
+ - Select the latest version of Java 
+
+    `$ sudo alternatives --config java`
+
 
 - We targeted and tested Java 11, but 11+ will likely work. Java 11 was targeted because of its long term support.
 - If you're moving from a Dataverse installation that was previously running Glassfish 4.x (typically this would be Dataverse Software 4.x), you will need to adjust some JVM options in domain.xml as part of the upgrade process. We've provided these optional steps below. These steps are not required if your first installed Dataverse Software version was running Payara 5.x (typically Dataverse Software 5.x).

doc/release-notes/5.5-release-notes.md

@@ -16,11 +16,18 @@ Users trying to download a zip file larger than the Dataverse installation's :Zi
 
 ### Guidelines on Depositing Code
 
-The Software Metadata Working Group has created guidelines on depositing research code in a Dataverse installation. Learn more in the [Dataset Management section](https://guides.dataverse.org/en/latest/dataset-management) of the Dataverse Guides.
+The Software Metadata Working Group has created guidelines on depositing research code in a Dataverse installation. Learn more in the [Dataset Management section](https://guides.dataverse.org/en/latest/user/dataset-management.html#research-code) of the Dataverse Guides.
 
 ### New Metrics API
 
-Users can retrieve new types of metrics and per-collection metrics. The new capabilities are described in the guides. A new version of the [Dataverse Metrics web app](https://github.com/IQSS/dataverse-metrics) adds interactive graphs to display these metrics. Anyone running the existing Dataverse Metrics app will need to upgrade or apply [a small patch](https://github.com/IQSS/dataverse-metrics/pull/60) to continue retrieving metrics from Dataverse instances upgrading to this release.
+Users can retrieve new types of metrics and per-collection metrics. The new capabilities are [described](https://guides.dataverse.org/en/5.5/api/metrics.html) in the guides. A new version of the [Dataverse Metrics web app](https://github.com/IQSS/dataverse-metrics) adds interactive graphs to display these metrics.
+
+There is a backward incompatible change to the Metrics API whereby you must now send the header `Accept: application/json` if you want to continue receiving JSON output. As of this release, absence of this header results in CSV rather than JSON for time series metrics. For details, see [Return Formats](https://guides.dataverse.org/en/5.5/api/metrics.html#return-formats) in the Metrics API section.
+
+Because of this change from JSON to CSV, adjustments have been made to the software that makes use of the Metrics API:
+
+- Anyone running the existing Dataverse Metrics app will need to upgrade to 0.2.8 (or higher) or apply [a small patch](https://github.com/IQSS/dataverse-metrics/pull/60) to continue retrieving metrics from Dataverse instances upgrading to this release.
+- Anyone who is using the Harvard Dataverse Repository custom homepage as a model might want to look at [changes](https://github.com/IQSS/dataverse.harvard.edu/pull/116) to get elements such as "browse by subject" working again.
 
 ## Major Use Cases
 
@@ -86,12 +93,20 @@ Most of your API users have likely figured it out already, since you enabled S3
 
 The "deactivated" field on the Authenticated User table has been updated to be a non-nullable field. When the field was added in version 5.3 it was set to 'false' in an update script. If for whatever reason that update failed in the 5.3 deploy you will need to re-run it before deploying 5.5. The update query you may need to run is: UPDATE authenticateduser SET deactivated = false WHERE deactivated IS NULL;
 
+### Metrics API Change
+
+See "New Metrics API" above for a backward incompatible change.
+
 ## Notes for Tool Developers and Integrators
 
 ### S3 Download Redirects
 
 See above note about download redirects. If your application integrates with the the Dataverse software using the APIs, you may need to change how redirects are handled in your tool or integration.
 
+### Metrics API Change
+
+See "New Metrics API" above for a backward incompatible change.
+
 ## Complete List of Changes
 
 For the complete list of code changes in this release, see the [5.5 Milestone](https://github.com/IQSS/dataverse/milestone/96?closed=1) in Github.