add Private URL feature #1012

Requires scripts/database/upgrades/privateurl.sql

Design doc at doc/Architecture/privateurl.md

New API endpoints: doc/sphinx-guides/source/api/native-api.rst

doc/Architecture/privateurl.md

@@ -0,0 +1,34 @@
+Private URL Design Doc
+----------------------
+
+The Private URL feature has been implemented as a specialized role assignment with an associated token that permits read-only access to the metadata and all files (regardless of if the files are restricted or not) of a draft version of a dataset.
+
+The primary use case for a Private URL is for journal editors to send a link to reviewers of a dataset before publication. In most cases, these journal editors do not permit depositors to publish on their own, which is to say they only allow depositors to have the "Contributor" role on the datasets they create. With only the "Contributor" role, depositors are unable to grant even read-only access to any user within the Dataverse installation and must contact the journal editor to make any adjustments to permissions, which they can't even see. This is all by design because it is the journal editor, not the depositor, who is in charge of both the security of the dataset and the timing of when the dataset is published.
+
+A secondary use case for a Private URL is for depositors who have the ability to manage permissions on their dataset (depositors who have the "Curator" or "Admin" role, which grants much more power than the "Contributor" role) to send a link to coauthors or other trusted parties to preview the dataset before the depositors publish the dataset on their own. For better security, these depositors could ask their coauthors to create Dataverse accounts and assign roles to them directly, rather than using a Private URL which requires no username or password.
+
+The token associated with the Private URL role assignment that can be used either in the GUI or via the API to elevate privileges beyond what a "Guest" can see. The ability to use a Private URL token via API was added mostly to facilitate automated testing of the feature but the far more common case is expected to be use of the Private URL token in a link that is clicked to open a browser, similar to links shared via Dropbox, Google, etc.
+
+When reviewers click a Private URL their browser sessions are set to the "PrivateUrlUser" that has the "Member" role only on the dataset in question and redirected to that dataset, where they will see an indication in blue at the top of the page that they are viewing an unpublished dataset. If the reviewer happens to be logged into Dataverse already, clicking the link will log them out because the review is meant to be blind. Because the dataset is always in draft when a Private URL is in effect, no downloads or any other activity by the reviewer are logged to the guestbook. All reviewers click the same Private URL containing the same token, and with the exception of an IP address being logged, it should be impossible to trace which reviewers have clicked a Private URL. If the reviewer navigates to the home page, the session is set to the Guest user and they will see what a Guest would see.
+
+The "Member" role is used because it contains the necessary read-only permissions, which are ViewUnpublishedDataset and DownloadFile. (Technically, the "Member" role also has the ViewUnpublishedDataverse permission but because the role is assigned at the dataset level and dataverses cannot be children of datasets, this permission has no effect.) Reusing the "Member" role helps contain the list of roles available at the dataset level to a reasonable number (five).
+
+Because the PrivateUrlUser has the "Member" role, all the same permissions apply. This means that the PrivateUrlUser (the reviewer, typically) can download all files, even if they have been restricted, across any dataset version. A Member can also download restricted files that have been deleted from previously published versions.
+
+Likewise, when a Private URL token is used via API, commands are executed using the "PrivateUrlUser" that has the "Member" role only on the dataset in question. This means that read-only operations such as downloads of the dataset's files are permitted. The Search API does not respect the Private URL token but you can download unpublished metadata using the Native API and download files using the Access API.
+
+A Private URL cannot be created for a published version of a dataset. In the GUI, you will be reminded of this fact with a popup. The API will explain this as well.
+
+If a draft dataset containing a Private URL is published, the Private URL is deleted. This means that reviewers who click the link after publication will see a 404.
+
+If a post-publication draft containing a Private URL is deleted, the Private URL is deleted. This is to ensure that if a new draft is created in the future, a new token will be used.
+
+The creation and deletion of a Private URL are limited to the "Curator" and "Admin" roles because only those roles have the permission called "ManageDatasetPermissions", which is the permission used by the "AssignRoleCommand" and "RevokeRoleCommand" commands. If you have the permission to create or delete a Private URL, the fact that a Private URL is enabled for a dataset will be indicated in blue at the top of the page. Success messages are shown at the top of the page when you create or delete a Private URL. In the GUI, deleting a Private URL is called "disabling" and you will be prompted for a confirmation. No matter what you call it the role is revoked. You can also delete a Private URL by revoking the role.
+
+A "Contributor" does not have the "ManageDatasetPermissions" permission and cannot see "Permissions" nor "Private URL" under the "Edit" menu of their dataset. When a Curator or Admin has enabled a Private URL on a Contributor's dataset, the Contributor does not see a visual indication that a Private URL has been enabled for their dataset.
+
+There is no way for an "Admin" or "Curator" to see when a Private URL was created or deleted for a dataset but someone who has access to the database can see that the following commands are logged to the "actionlogrecord" database table: CreatePrivateUrlCommand, DeletePrivateUrlCommand, and GetPrivateUrlCommand.
+
+See also
+
+* Private URL To Unpublished Dataset BRD: https://docs.google.com/document/d/1FT47QkZKcmjSgRnePaJO2g1nzcotLyN3Yb2ORvBr6cs/edit?usp=sharing

doc/sphinx-guides/source/api/native-api.rst

@@ -162,6 +162,21 @@ Restores the default logic of the field type to be used as the citation date. Sa
     
     DELETE http://$SERVER/api/datasets/$id/citationdate?key=$apiKey
 
+List all the role assignments at the given dataset::
+
+    GET http://$SERVER/api/datasets/$id/assignments?key=$apiKey
+
+Create a Private URL (must be able to manage dataset permissions)::
+
+    POST http://$SERVER/api/datasets/$id/privateUrl?key=$apiKey
+
+Get a Private URL from a dataset (if available)::
+
+    GET http://$SERVER/api/datasets/$id/privateUrl?key=$apiKey
+
+Delete a Private URL from a dataset (if it exists)::
+
+    DELETE http://$SERVER/api/datasets/$id/privateUrl?key=$apiKey
 
 Builtin Users
 ~~~~~

scripts/database/upgrades/privateurl.sql

@@ -0,0 +1,2 @@
+-- A Private URL is a specialized role assignment with a token.
+ALTER TABLE roleassignment ADD COLUMN privateurltoken character varying(255);

src/main/java/Bundle.properties

@@ -750,6 +750,7 @@ dataset.editBtn.itemLabel.upload=Files (Upload)
 dataset.editBtn.itemLabel.metadata=Metadata
 dataset.editBtn.itemLabel.terms=Terms
 dataset.editBtn.itemLabel.permissions=Permissions
+dataset.editBtn.itemLabel.privateUrl=Private URL
 dataset.editBtn.itemLabel.deleteDataset=Delete Dataset
 dataset.editBtn.itemLabel.deleteDraft=Delete Draft Version
 dataset.editBtn.itemLabel.deaccession=Deaccession Dataset
@@ -862,6 +863,20 @@ dataset.mixedSelectedFilesForDownload=The restricted file(s) selected may not be
 dataset.downloadUnrestricted=Click Continue to download the files you have access to download.
 dataset.requestAccessToRestrictedFiles=You may request access to the restricted file(s) by clicking the Request Access button. 
 
+dataset.privateurl.infoMessageAuthor=Private URL is in use for this dataset: {0}
+dataset.privateurl.infoMessageReviewer=This is an unpublished draft of a dataset that has been shared privately.
+dataset.privateurl.header=Private URL
+dataset.privateurl.tip=Use a Private URL to allow those without Dataverse accounts to access your unpublished dataset.
+dataset.privateurl.absent=Private URL has not been created.
+dataset.privateurl.createPrivateUrl=Create Private URL
+dataset.privateurl.disablePrivateUrl=Disable Private URL
+dataset.privateurl.disableConfirmationTitle=Confirm Disable Private URL
+dataset.privateurl.disableConfirmationText=If you have sent the Private URL to others they will no longer be able to use it to access your unpublished dataset.
+dataset.privateurl.cannotCreate=Private URL can only be used with unpublished versions of datasets.
+dataset.privateurl.roleassigeeTitle=Private URL Enabled
+dataset.privateurl.createdSuccess=Private URL created: {0}
+dataset.privateurl.disabledSuccess=Private URL disabled.
+dataset.privateurl.noPermToCreate=To create a Private URL you must have the following permissions: {0}.
 file.count={0} {0, choice, 0#Files|1#File|2#Files}
 file.count.selected={0} {0, choice, 0#Files Selected|1#File Selected|2#Files Selected}
 file.selectToAddBtn=Select Files to Add

src/main/java/edu/harvard/iq/dataverse/DatasetPage.java

@@ -6,21 +6,29 @@
 import edu.harvard.iq.dataverse.authorization.users.ApiToken;
 import edu.harvard.iq.dataverse.authorization.users.User;
 import edu.harvard.iq.dataverse.authorization.users.AuthenticatedUser;
+import edu.harvard.iq.dataverse.authorization.users.PrivateUrlUser;
+import edu.harvard.iq.dataverse.authorization.users.GuestUser;
 import edu.harvard.iq.dataverse.datavariable.VariableServiceBean;
 import edu.harvard.iq.dataverse.engine.command.Command;
 import edu.harvard.iq.dataverse.engine.command.exception.CommandException;
 import edu.harvard.iq.dataverse.engine.command.impl.CreateDatasetCommand;
 import edu.harvard.iq.dataverse.engine.command.impl.CreateGuestbookResponseCommand;
+import edu.harvard.iq.dataverse.engine.command.impl.CreatePrivateUrlCommand;
 import edu.harvard.iq.dataverse.engine.command.impl.DeaccessionDatasetVersionCommand;
 import edu.harvard.iq.dataverse.engine.command.impl.DeleteDatasetVersionCommand;
+import edu.harvard.iq.dataverse.engine.command.impl.DeletePrivateUrlCommand;
 import edu.harvard.iq.dataverse.engine.command.impl.DestroyDatasetCommand;
+import edu.harvard.iq.dataverse.engine.command.impl.GetPrivateUrlCommand;
 import edu.harvard.iq.dataverse.engine.command.impl.LinkDatasetCommand;
 import edu.harvard.iq.dataverse.engine.command.impl.PublishDatasetCommand;
 import edu.harvard.iq.dataverse.engine.command.impl.PublishDataverseCommand;
 import edu.harvard.iq.dataverse.engine.command.impl.UpdateDatasetCommand;
 import edu.harvard.iq.dataverse.ingest.IngestRequest;
 import edu.harvard.iq.dataverse.ingest.IngestServiceBean;
 import edu.harvard.iq.dataverse.metadataimport.ForeignMetadataImportServiceBean;
+import edu.harvard.iq.dataverse.privateurl.PrivateUrl;
+import edu.harvard.iq.dataverse.privateurl.PrivateUrlServiceBean;
+import edu.harvard.iq.dataverse.privateurl.PrivateUrlUtil;
 import edu.harvard.iq.dataverse.search.SearchFilesServiceBean;
 import edu.harvard.iq.dataverse.search.SortBy;
 import edu.harvard.iq.dataverse.settings.SettingsServiceBean;
@@ -142,6 +150,10 @@ public enum DisplayMode {
     DatasetLinkingServiceBean dsLinkingService;
     @EJB
     SearchFilesServiceBean searchFilesService;
+    @EJB
+    DataverseRoleServiceBean dataverseRoleService;
+    @EJB
+    PrivateUrlServiceBean privateUrlService;
     @Inject
     DataverseRequestServiceBean dvRequestService;
     @Inject
@@ -486,10 +498,19 @@ public boolean canDownloadFile(FileMetadata fileMetadata){
         // --------------------------------------------------------------------
 
         // --------------------------------------------------------------------
-        // (2) Is user authenticated?
-        // No?  Then no button...
+        // (2) In Dataverse 4.3 and earlier we required that users be authenticated
+        // to download files, but in developing the Private URL feature, we have
+        // added a new subclass of "User" called "PrivateUrlUser" that returns false
+        // for isAuthenticated but that should be able to download restricted files
+        // when given the Member role (which includes the DownloadFile permission).
+        // This is consistent with how Builtin and Shib users (both are
+        // AuthenticatedUsers) can download restricted files when they are granted
+        // the Member role. For this reason condition 2 has been changed. Previously,
+        // we required isSessionUserAuthenticated to return true. Now we require
+        // that the User is not an instance of GuestUser, which is similar in
+        // spirit to the previous check.
         // --------------------------------------------------------------------
-        if (!(isSessionUserAuthenticated())){
+        if (session.getUser() instanceof GuestUser){
             this.fileDownloadPermissionMap.put(fid, false);
             return false;
         }
@@ -1555,6 +1576,20 @@ public String init() {
             return "/404.xhtml";
         }
 
+        try {
+            privateUrl = commandEngine.submit(new GetPrivateUrlCommand(dvRequestService.getDataverseRequest(), dataset));
+            if (privateUrl != null) {
+                JH.addMessage(FacesMessage.SEVERITY_INFO, BundleUtil.getStringFromBundle("dataset.privateurl.infoMessageAuthor", Arrays.asList(getPrivateUrlLink(privateUrl))));
+            }
+        } catch (CommandException ex) {
+            // No big deal. The user simply doesn't have access to create or delete a Private URL.
+        }
+        if (session.getUser() instanceof PrivateUrlUser) {
+            PrivateUrlUser privateUrlUser = (PrivateUrlUser) session.getUser();
+            if (dataset != null && dataset.getId().equals(privateUrlUser.getDatasetId())) {
+                JH.addMessage(FacesMessage.SEVERITY_INFO, BundleUtil.getStringFromBundle("dataset.privateurl.infoMessageReviewer"));
+            }
+        }
         return null;
     }
 
@@ -4282,4 +4317,44 @@ public String getSortByDescending() {
         return SortBy.DESCENDING;
     }
 
+    PrivateUrl privateUrl;
+
+    public PrivateUrl getPrivateUrl() {
+        return privateUrl;
+    }
+
+    public void setPrivateUrl(PrivateUrl privateUrl) {
+        this.privateUrl = privateUrl;
+    }
+
+    public void createPrivateUrl() {
+        try {
+            PrivateUrl createdPrivateUrl = commandEngine.submit(new CreatePrivateUrlCommand(dvRequestService.getDataverseRequest(), dataset));
+            privateUrl = createdPrivateUrl;
+            JH.addSuccessMessage(BundleUtil.getStringFromBundle("dataset.privateurl.createdSuccess", Arrays.asList(getPrivateUrlLink(privateUrl))));
+        } catch (CommandException ex) {
+            String msg = BundleUtil.getStringFromBundle("dataset.privateurl.noPermToCreate", PrivateUrlUtil.getRequiredPermissions(ex));
+            logger.info("Unable to create a Private URL for dataset id " + dataset.getId() + ". Message to user: " + msg + " Exception: " + ex);
+            JH.addErrorMessage(msg);
+        }
+    }
+
+    public void disablePrivateUrl() {
+        try {
+            commandEngine.submit(new DeletePrivateUrlCommand(dvRequestService.getDataverseRequest(), dataset));
+            privateUrl = null;
+            JH.addSuccessMessage(BundleUtil.getStringFromBundle("dataset.privateurl.disabledSuccess"));
+        } catch (CommandException ex) {
+            logger.info("CommandException caught calling DeletePrivateUrlCommand: " + ex);
+        }
+    }
+
+    public boolean isUserCanCreatePrivateURL() {
+        return dataset.getLatestVersion().isDraft();
+    }
+
+    public String getPrivateUrlLink(PrivateUrl privateUrl) {
+        return privateUrl.getLink();
+    }
+
 }

src/main/java/edu/harvard/iq/dataverse/DataverseSession.java

@@ -39,7 +39,7 @@ public User getUser() {
         return user;
     }
 
-    public void setUser(AuthenticatedUser aUser) {
+    public void setUser(User aUser) {
         logSvc.log( 
                 new ActionLogRecord(ActionLogRecord.ActionType.SessionManagement,(aUser==null) ? "logout" : "login")
                     .setUserIdentifier((aUser!=null) ? aUser.getIdentifier() : (user!=null ? user.getIdentifier() : "") ));

src/main/java/edu/harvard/iq/dataverse/EjbDataverseEngine.java

@@ -12,6 +12,7 @@
 import edu.harvard.iq.dataverse.engine.command.DataverseRequest;
 import edu.harvard.iq.dataverse.engine.command.exception.CommandException;
 import edu.harvard.iq.dataverse.engine.command.exception.PermissionException;
+import edu.harvard.iq.dataverse.privateurl.PrivateUrlServiceBean;
 import edu.harvard.iq.dataverse.search.IndexServiceBean;
 import edu.harvard.iq.dataverse.search.SearchServiceBean;
 import java.util.Map;
@@ -23,6 +24,7 @@
 import edu.harvard.iq.dataverse.search.SolrIndexServiceBean;
 import edu.harvard.iq.dataverse.search.savedsearch.SavedSearchServiceBean;
 import edu.harvard.iq.dataverse.settings.SettingsServiceBean;
+import edu.harvard.iq.dataverse.util.SystemConfig;
 import java.util.EnumSet;
 import java.util.logging.Level;
 import java.util.logging.Logger;
@@ -127,7 +129,13 @@ public class EjbDataverseEngine {
 
     @EJB
     AuthenticationServiceBean authentication; 
-
+
+    @EJB
+    SystemConfig systemConfig;
+
+    @EJB
+    PrivateUrlServiceBean privateUrlService;
+
     @PersistenceContext(unitName = "VDCNet-ejbPU")
     private EntityManager em;
 
@@ -371,7 +379,17 @@ public UserNotificationServiceBean notifications() {
                 public AuthenticationServiceBean authentication() {
                     return authentication;
                 } 
-
+
+                @Override
+                public SystemConfig systemConfig() {
+                    return systemConfig;
+                }
+
+                @Override
+                public PrivateUrlServiceBean privateUrl() {
+                    return privateUrlService;
+                }
+
             };
         }
 

src/main/java/edu/harvard/iq/dataverse/ManageFilePermissionsPage.java

@@ -399,7 +399,8 @@ private void rejectAccessToRequests(AuthenticatedUser au, List<DataFile> files)
 
     private boolean assignRole(RoleAssignee ra,  DataFile file, DataverseRole r) {
         try {
-            commandEngine.submit(new AssignRoleCommand(ra, r, file, dvRequestService.getDataverseRequest()));
+            String privateUrlToken = null;
+            commandEngine.submit(new AssignRoleCommand(ra, r, file, dvRequestService.getDataverseRequest(), privateUrlToken));
             JsfHelper.addSuccessMessage(r.getName() + " role assigned to " + ra.getDisplayInfo().getTitle() + " for " + file.getDisplayName() + ".");
         } catch (PermissionException ex) {
             JH.addMessage(FacesMessage.SEVERITY_ERROR, "The role was not able to be assigned.", "Permissions " + ex.getRequiredPermissions().toString() + " missing.");

src/main/java/edu/harvard/iq/dataverse/ManagePermissionsPage.java

@@ -395,7 +395,8 @@ private void notifyRoleChange(RoleAssignee ra, UserNotification.Type type) {
 
     private void assignRole(RoleAssignee ra, DataverseRole r) {
         try {
-            commandEngine.submit(new AssignRoleCommand(ra, r, dvObject, dvRequestService.getDataverseRequest()));
+            String privateUrlToken = null;
+            commandEngine.submit(new AssignRoleCommand(ra, r, dvObject, dvRequestService.getDataverseRequest(), privateUrlToken));
             JsfHelper.addSuccessMessage(r.getName() + " role assigned to " + ra.getDisplayInfo().getTitle() + " for " + dvObject.getDisplayName() + ".");
             // don't notify if role = file downloader and object is not released
             if (!(r.getAlias().equals(DataverseRole.FILE_DOWNLOADER) && !dvObject.isReleased()) ){