Merging with 3919-dataset-files #3921

doc/sphinx-guides/source/_static/installation/files/etc/init.d/glassfish.init.service

@@ -0,0 +1,34 @@
+#! /bin/sh
+# chkconfig: 2345 80 01
+# description: GlassFish App Server
+set -e
+
+ASADMIN=/usr/local/glassfish4/bin/asadmin
+GF_USER=glassfish
+
+case "$1" in
+  start)
+        echo -n "Starting GlassFish server: glassfish"
+        # Increase file descriptor limit:
+        ulimit -n 32768
+        # Allow "memory overcommit":
+        # (basically, this allows to run exec() calls from inside the
+        # app, without the Unix fork() call physically hogging 2X
+        # the amount of memory glassfish is already using)
+        echo 1 > /proc/sys/vm/overcommit_memory
+	LANG=en_US.UTF-8; export LANG
+
+        sudo -u $GF_USER $ASADMIN start-domain domain1
+        ;;
+  stop)
+        echo -n "Stopping GlassFish server: glassfish"
+
+        sudo -u $GF_USER $ASADMIN stop-domain domain1
+        echo "."
+         ;;
+  *)
+        echo "Usage: /etc/init.d/glassfish {start|stop}"
+
+        exit 1
+esac
+exit 0

doc/sphinx-guides/source/_static/installation/files/etc/systemd/glassfish.service

@@ -0,0 +1,13 @@
+[Unit]
+Description = GlassFish Server v4.1
+After = syslog.target network.target
+
+[Service]
+User=glassfish
+ExecStart = /usr/bin/java -jar /usr/local/glassfish4/glassfish/lib/client/appserver-cli.jar start-domain
+ExecStop = /usr/bin/java -jar /usr/local/glassfish4/glassfish/lib/client/appserver-cli.jar stop-domain
+ExecReload = /usr/bin/java -jar /usr/local/glassfish4/glassfish/lib/client/appserver-cli.jar restart-domain
+Type = forking
+
+[Install]
+WantedBy = multi-user.target

doc/sphinx-guides/source/installation/installation-main.rst

@@ -27,15 +27,24 @@ Execute the installer script like this::
         # cd dvinstall
         # ./install
 
-**NEW in Dataverse 4.3:** It is no longer necessary to run the installer as root!
-Just make sure the user that runs the installer has the write permission in the Glassfish directory. For example, if your Glassfish directory is owned by root, and you try to run the installer as a regular user, it's not going to work. 
-(Do note, that you want the Glassfish directory to be owned by the same user that will be running Glassfish. And you most likely won't need to run it as root. The only reason to run Glassfish as root would be to have a convenient way to run the application on the default HTTP(S) ports 80 and 443, instead of 8080 and 8181. However, an easier, and more secure way to achieve that would be to instead keep Glassfish running on a high port, and hide it behind an Apache Proxy, via AJP, running on port 80. This configuration is in fact required if you choose to have your Dataverse support Shibboleth authentication. See more discussion on this here: :doc:`shibboleth`.)
+It is no longer necessary to run the installer as root!
+=======================================================
+
+Just make sure the user running the installer has write permission to:
+
+- /usr/local/glassfish4/glassfish/lib
+- /usr/local/glassfish4/glassfish/domains/domain1
+- the current working directory of the installer (it currently writes its logfile there), and
+- your jvm-option specified files.dir
+
+The only reason to run Glassfish as root would be to allow Glassfish itself to listen on the default HTTP(S) ports 80 and 443, or any other port below 1024. However, it is simpler and more secure to run Glassfish run on its default port of 8080 and hide it behind an Apache Proxy, via AJP, running on port 80 or 443. This configuration is required if you're going to use Shibboleth authentication. See more discussion on this here: :doc:`shibboleth`.)
 
 
 The script will prompt you for some configuration values. If this is a test/evaluation installation, it may be possible to accept the default values provided for most of the settings:
 
 - Internet Address of your host: localhost
 - Glassfish Directory: /usr/local/glassfish4
+- Glassfish User: current user running the installer script
 - Administrator email address for this Dataverse: (none)
 - SMTP (mail) server to relay notification messages: localhost
 - Postgres Server Address: [127.0.0.1]

doc/sphinx-guides/source/installation/prerequisites.rst

@@ -2,7 +2,7 @@
 Prerequisites
 =============
 
-Before running the Dataverse installation script, you must install and configure the following software, preferably on a distribution of Linux such as RHEL or its derivatives such as CentOS. After following all the steps below (which have been written based on CentOS 6), you can proceed to the :doc:`installation-main` section.
+Before running the Dataverse installation script, you must install and configure the following software, preferably on a Linux distribution such as RHEL or CentOS. After following all the steps below (which are mostly based on CentOS 6), you can proceed to the :doc:`installation-main` section.
 
 You **may** find it helpful to look at how the configuration is done automatically by various tools such as Vagrant, Puppet, or Ansible. See the :doc:`prep` section for pointers on diving into these scripts.
 
@@ -43,40 +43,53 @@ Glassfish Version 4.1 is required. There are known issues with Glassfish 4.1.1 a
 Installing Glassfish
 ====================
 
-**Important**: once Glassfish is installed, a new version of the Weld library (v2.2.10.SP1) must be downloaded and installed. This fixes a serious issue in the library supplied with Glassfish 4.1 ( see https://github.com/IQSS/dataverse/issues/647 for details). Please note that if you plan to front Glassfish with Apache you must also patch Grizzly as explained in the :doc:`shibboleth` section.
+**Note:** The Dataverse installer need not be run as root, and it is recommended that Glassfish not run as root either. We suggest the creation of a glassfish service account for this purpose.
 
 - Download and install Glassfish (installed in ``/usr/local/glassfish4`` in the example commands below)::
 
 	# wget http://dlc-cdn.sun.com/glassfish/4.1/release/glassfish-4.1.zip
 	# unzip glassfish-4.1.zip
 	# mv glassfish4 /usr/local
 
+If you intend to install and run Glassfish under a service account (and we hope you do), chown -R the Glassfish hierarchy to root to protect it but give the service account access to the below directories:
+
+- Set service account permissions::
+
+	# chown -R root:root /usr/local/glassfish4
+	# chown glassfish /usr/local/glassfish4/glassfish/lib
+	# chown -R glassfish:glassfish /usr/local/glassfish4/glassfish/domains/domain1
+
+After installation, you may chown the lib/ directory back to root; the installer only needs write access to copy the JDBC driver into that directory.
+
+Once Glassfish is installed, you'll need a newer version of the Weld library (v2.2.10.SP1) to fix a serious issue in the library supplied with Glassfish 4.1 (see https://github.com/IQSS/dataverse/issues/647 for details). If you plan to front Glassfish with Apache you must also patch Grizzly as explained in the :doc:`shibboleth` section.
+
 - Remove the stock Weld jar; download Weld v2.2.10.SP1 and install it in the modules folder::
 
 	# cd /usr/local/glassfish4/glassfish/modules
 	# rm weld-osgi-bundle.jar
 	# wget http://central.maven.org/maven2/org/jboss/weld/weld-osgi-bundle/2.2.10.SP1/weld-osgi-bundle-2.2.10.SP1-glassfish4.jar
-	# /usr/local/glassfish4/bin/asadmin start-domain
 
-- Verify the Weld version::
+- Change from ``-client`` to ``-server`` under ``<jvm-options>-client</jvm-options>``::
 
-	# /usr/local/glassfish4/bin/asadmin osgi lb | grep 'Weld OSGi Bundle'
-
-- Stop Glassfish and change from ``-client`` to ``-server`` under ``<jvm-options>-client</jvm-options>``::
-
-	# /usr/local/glassfish4/bin/asadmin stop-domain
 	# vim /usr/local/glassfish4/glassfish/domains/domain1/config/domain.xml
 
 This recommendation comes from http://blog.c2b2.co.uk/2013/07/glassfish-4-performance-tuning.html among other places.
 
-Glassfish Init Script
-=====================
+- Start Glassfish and verify the Weld version::
+
+	# /usr/local/glassfish4/bin/asadmin start-domain
+	# /usr/local/glassfish4/bin/asadmin osgi lb | grep 'Weld OSGi Bundle'
+
+Launching Glassfish on system boot
+==================================
 
-The Dataverse installation script will start Glassfish if necessary, but while you're configuring Glassfish, you might find the following init script helpful to have Glassfish start on boot.
+The Dataverse installation script will start Glassfish if necessary, but you may find the following scripts helpful to launch Glassfish start automatically on boot.
 
-Adjust this :download:`Glassfish init script <../_static/installation/files/etc/init.d/glassfish>` for your needs or write your own.
+- This :download:`Systemd file<../_static/installation/files/etc/systemd/glassfish.service>` may be serve as a reference for systems using Systemd (such as RHEL/CentOS 7 or Ubuntu 16+)
+- This :download:`init script<../_static/installation/files/etc/init.d/glassfish.init.service>` may be useful for RHEL/CentOS6 or Ubuntu >= 14 if you're using a Glassfish service account, or
+- This :download:`Glassfish init script <../_static/installation/files/etc/init.d/glassfish.init.root>` may be helpful if you're just going to run Glassfish as root.
 
-It is not necessary to have Glassfish running before you execute the Dataverse installation script because it will start Glassfish for you.
+It is not necessary for Glassfish to be running before you execute the Dataverse installation script; it will start Glassfish for you.
 
 Please note that you must run Glassfish in an English locale. If you are using something like ``LANG=de_DE.UTF-8``, ingest of tabular data will fail with the message "RoundRoutines:decimal separator no in right place".
 

scripts/database/upgrades/upgrade_v4.7.1_to_v4.7.2.sql

@@ -4,10 +4,12 @@ ALTER TABLE dvobject ADD COLUMN storageidentifier character varying(255);
 UPDATE dvobject
 SET storageidentifier=(SELECT datafile.filesystemname
 FROM datafile
-WHERE datafile.id=dvobject.id AND dvobject.dtype='DataFile');
+WHERE datafile.id=dvobject.id AND dvobject.dtype='DataFile') where dvobject.dtype='DataFile';
 
 UPDATE dvobject 
 SET storageidentifier=(select concat('file://',authority::text,ds.doiseparator::text,ds.identifier::text) 
 FROM dataset ds 
 WHERE dvobject.id=ds.id) 
-WHERE storageidentifier IS NULL;
+WHERE storageidentifier IS NULL;
+
+ALTER TABLE datafile DROP COLUMN filesystemname;

scripts/installer/install

@@ -9,6 +9,7 @@ use File::Copy;
 my $verbose;
 my $pg_only;
 my $hostname;
+my $gfuser;
 my $gfdir;
 my $mailserver;
 my $yes;
@@ -21,6 +22,7 @@ my ($rez) = GetOptions(
     "verbose"      => \$verbose,
     "pg_only"      => \$pg_only,
     "hostname=s"   => \$hostname,
+    "gfuser=s"     => \$gfuser,
     "gfdir=s"      => \$gfdir,
     "mailserver=s" => \$mailserver,
     "y|yes"        => \$yes,
@@ -45,6 +47,7 @@ else
 
     @CONFIG_VARIABLES = (
 	'HOST_DNS_ADDRESS',
+	'GLASSFISH_USER',
 	'GLASSFISH_DIRECTORY',
 	'ADMIN_EMAIL',
 	'MAIL_SERVER',
@@ -70,7 +73,9 @@ else
 
 my %CONFIG_DEFAULTS = (
     'HOST_DNS_ADDRESS',    'localhost',
+    'GLASSFISH_USER',	   '',
     'GLASSFISH_DIRECTORY', '/usr/local/glassfish4',
+    'GLASSFISH_USER',	   '',
     'ADMIN_EMAIL',         '',
     'MAIL_SERVER',         'mail.hmdc.harvard.edu',
 
@@ -93,6 +98,7 @@ my %CONFIG_DEFAULTS = (
 );
 my %CONFIG_PROMPTS = (
     'HOST_DNS_ADDRESS',    'Fully Qualified Domain Name of your host',
+    'GLASSFISH_USER',	   'Glassfish service account username',
     'GLASSFISH_DIRECTORY', 'Glassfish Directory',
     'ADMIN_EMAIL',         'Administrator email address for this Dataverse',
     'MAIL_SERVER',         'SMTP (mail) server to relay notification messages',
@@ -118,6 +124,7 @@ my %CONFIG_PROMPTS = (
 
 my %CONFIG_COMMENTS = (
     'HOST_DNS_ADDRESS',    ":\n(enter numeric IP address, if FQDN is unavailable) ",
+    'GLASSFISH_USER',      ":\nThis user will be running Glassfish service on your system.\n - If this is a dev. environment, this should be your own username; \n - In production, we suggest \"glassfish\" or another unprivileged user\n: ",
     'GLASSFISH_DIRECTORY', '',
     'ADMIN_EMAIL',         ":\n(please enter a valid email address!) ",
     'MAIL_SERVER',         '',
@@ -202,6 +209,37 @@ else
 	print("using hard-coded default configuration values ($config_default_file not found)\n");
 }
 
+# get current user. first one wins.
+my $current_user = $ENV{LOGNAME} || $ENV{USER} || getpwuid($<);
+
+if (!$CONFIG_DEFAULTS{'GLASSFISH_USER'}) {
+   $CONFIG_DEFAULTS{'GLASSFISH_USER'} = $current_user;
+   print "No pre-configured user found; using $current_user.\n";
+}
+
+# command-line argument takes precendence
+if ($gfuser) {
+    print "Using CLI-specified user $gfuser.\n";
+    $CONFIG_DEFAULTS{'GLASSFISH_USER'} = $gfuser;
+}
+
+# prefer that we not install as root.
+unless ( $< != 0 ) {
+print "####################################################################\n";
+print "     It is recommended that this script not be run as root.\n";
+print " Consider creating a glassfish service account, giving it ownership\n";
+print "  on the glassfish/domains/domain1/ and glassfish/lib/ directories,\n";
+print "    along with the JVM-specified files.dir location, and running\n";
+print "       this installer as the user who will launch Glassfish.\n";
+print "####################################################################\n";
+}
+
+# ensure $gfuser exists or bail
+my $gfidcmd="id $CONFIG_DEFAULTS{'GLASSFISH_USER'}";
+my $gfreturncode=system($gfidcmd);
+if ($gfreturncode != 0) {
+   die "Couldn't find user $gfuser. Please ensure the account exists and is readable by the user running this installer.\n";
+}
 
 if ($mailserver) {
     $CONFIG_DEFAULTS{'MAIL_SERVER'} = $mailserver;
@@ -358,6 +396,15 @@ for my $ENTRY (@CONFIG_VARIABLES)
 		# TODO: 
 		# verify that we can write in the Glassfish directory 
 		# (now that we are no longer requiring to run the installer as root)
+
+		my $g_testdir = $g_dir . "/glassfish/domains/domain1";
+		my $g_libdir = $g_dir . "/glassfish/lib";
+		if (!(-w $g_testdir)) {
+		   die("$g_testdir not writable. Have you created a glassfish user, and given it write permission on $g_testdir?\n");
+		} elsif (!(-w $g_libdir)) {
+		   die("$g_libdir not writable. Have you created a glassfish user, and given it write permission on $g_libdir?\n");
+		}
+
 	    }
 
 	    print "OK!\n";
@@ -1011,7 +1058,7 @@ my $DOMAIN_DOWN =
 print STDERR $DOMAIN_DOWN . "\n";
 if ($DOMAIN_DOWN) {
     print "Trying to start domain up...\n";
-    system( $CONFIG_DEFAULTS{'GLASSFISH_DIRECTORY'} . "/bin/asadmin start-domain domain1" );
+    system( "sudo -u $CONFIG_DEFAULTS{'GLASSFISH_USER'} " . $CONFIG_DEFAULTS{'GLASSFISH_DIRECTORY'} . "/bin/asadmin start-domain domain1" );
 
     # TODO: (?) - retest that the domain is running now? 
 }

scripts/vagrant/install-dataverse.sh

@@ -1,4 +1,5 @@
 #!/usr/bin/env bash
+
 if [ ! -z "$1" ]; then
   MAILSERVER=$1
   MAILSERVER_ARG="--mailserver $MAILSERVER"
@@ -17,12 +18,16 @@ pid=$$
 if [ -e default.config ]; then
 	mv default.config tmp-${pid}-default.config
 fi
+
 echo "HOST_DNS_ADDRESS	localhost" > default.config
 echo "GLASSFISH_DIRECTORY	/home/glassfish/glassfish4" >> default.config
+
 if [ ! -z "$MAILSERVER" ]; then
 	echo "MAIL_SERVER	$MAILSERVER" >> default.config
 fi
+
 ./install -y -f
+
 if [ -e tmp-${pid}-default.config ]; then # if we moved it out, move it back
 	mv -f tmp-${pid}-default.config default.config
 fi

src/main/java/edu/harvard/iq/dataverse/DataFile.java

@@ -10,7 +10,7 @@
 import edu.harvard.iq.dataverse.api.WorldMapRelatedData;
 import edu.harvard.iq.dataverse.authorization.users.AuthenticatedUser;
 import edu.harvard.iq.dataverse.dataaccess.DataAccess;
-import edu.harvard.iq.dataverse.dataaccess.DataFileIO;
+import edu.harvard.iq.dataverse.dataaccess.StorageIO;
 import edu.harvard.iq.dataverse.dataset.DatasetThumbnail;
 import edu.harvard.iq.dataverse.ingest.IngestReport;
 import edu.harvard.iq.dataverse.ingest.IngestRequest;
@@ -82,10 +82,10 @@ public class DataFile extends DvObject implements Comparable {
     private String contentType;
 
 
-    @Expose    
-    @SerializedName("storageIdentifier")
-    @Column( nullable = false )
-    private String fileSystemName;
+//    @Expose    
+//    @SerializedName("storageIdentifier")
+//    @Column( nullable = false )
+//    private String fileSystemName;
 
     /**
      * End users will see "SHA-1" (with a hyphen) rather than "SHA1" in the GUI
@@ -438,13 +438,13 @@ public void setOwner(Dataset dataset) {
         super.setOwner(dataset);
     }
 
-    public String getStorageIdentifier() {
-        return this.fileSystemName;
-    }
-
-    public void setStorageIdentifier(String storageIdentifier) {
-        this.fileSystemName = storageIdentifier;
-    }
+//    public String getStorageIdentifier() {
+//        return this.fileSystemName;
+//    }
+//
+//    public void setStorageIdentifier(String storageIdentifier) {
+//        this.fileSystemName = storageIdentifier;
+//    }
 
     public String getDescription() {
         FileMetadata fmd = getLatestFileMetadata();
@@ -555,14 +555,14 @@ public String getOriginalChecksumType() {
         return BundleUtil.getStringFromBundle("file.originalChecksumType", Arrays.asList(this.checksumType.toString()) );
     }
 
-    public DataFileIO<DataFile> getDataFileIO() throws IOException {
-        DataFileIO<DataFile> dataFileIO = DataAccess.getDataFileIO(this);
+    public StorageIO<DataFile> getStorageIO() throws IOException {
+        StorageIO<DataFile> storageIO = DataAccess.getStorageIO(this);
 
-        if (dataFileIO == null) {
-            throw new IOException("Failed to create DataFileIO for datafile.");
+        if (storageIO == null) {
+            throw new IOException("Failed to create storageIO for datafile.");
         }
 
-        return dataFileIO; 
+        return storageIO; 
     }
 
     /*