Merge branch 'develop' into 10611-harvested-datasets-display

Resolved merged conflicts in config.rst and FeatureFlags.java #10611

doc/release-notes/10503-cvoc-hidden-html-fields.md

@@ -0,0 +1,11 @@
+## Release Highlights
+
+### Updates on Support for External Vocabulary Services
+
+#### Hidden HTML Fields
+
+External Controlled Vocabulary scripts, configured via [:CVocConf](https://guides.dataverse.org/en/6.3/installation/config.html#cvocconf), can now access the values of managed fields as well as the term-uri-field for use in constructing the metadata view for a dataset.
+
+Those values are hidden and can be found with the html attribute `data-cvoc-metadata-name`.
+
+For more information, see [#10503](https://github.com/IQSS/dataverse/pull/10503).

doc/release-notes/10579-avoid-solr-deletes.md

@@ -0,0 +1,9 @@
+A features flag called "reduce-solr-deletes" has been added to improve how datafiles are indexed. When the flag is enabled, 
+Dataverse wil avoid pre-emptively deleting existing solr documents for the files prior to sending updated information. This
+should improve performance and will allow additional optimizations going forward.
+
+The /api/admin/index/status and /api/admin/index/clear-orphans calls
+(see https://guides.dataverse.org/en/latest/admin/solr-search-index.html#index-and-database-consistency)
+will now find and remove (respectively) additional permissions related solr documents that were not being detected before.
+Reducing the overall number of documents will improve solr performance and large sites may wish to periodically call the 
+clear-orphans API.

doc/sphinx-guides/source/developers/performance.rst

@@ -121,6 +121,7 @@ While in the past Solr performance hasn't been much of a concern, in recent year
 We are tracking performance problems in `#10469 <https://github.com/IQSS/dataverse/issues/10469>`_.
 
 In a meeting with a Solr expert on 2024-05-10 we were advised to avoid joins as much as possible. (It was acknowledged that many Solr users make use of joins because they have to, like we do, to keep some documents private.) Toward that end we have added two feature flags called ``avoid-expensive-solr-join`` and ``add-publicobject-solr-field`` as explained under :ref:`feature-flags`. It was confirmed experimentally that performing the join on all the public objects (published collections, datasets and files), i.e., the bulk of the content in the search index, was indeed very expensive, especially on a large instance the size of the IQSS prod. archive, especially under indexing load. We confirmed that it was in fact unnecessary and were able to replace it with a boolean field directly in the indexed documents, which is achieved by the two feature flags above. However, as of writing this, this mechanism should still be considered experimental.
+Another flag, ``reduce-solr-deletes``, avoids deleting solr documents for files in a dataset prior to sending updates. It also eliminates several causes of orphan permission documents. This is expected to improve indexing performance to some extent and is a step towards avoiding unnecessary updates (i.e. when a doc would not change).
 
 Datasets with Large Numbers of Files or Versions
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

doc/sphinx-guides/source/installation/config.rst

@@ -3274,6 +3274,9 @@ please find all known feature flags below. Any of these flags can be activated u
     * - add-publicobject-solr-field
       - Adds an extra boolean field `PublicObject_b:true` for public content (published Collections, Datasets and Files). Once reindexed with these fields, we can rely on it to remove a very expensive Solr join on all such documents in Solr queries, significantly improving overall performance (by enabling the feature flag above, `avoid-expensive-solr-join`). These two flags are separate so that an instance can reindex their holdings before enabling the optimization in searches, thus avoiding having their public objects temporarily disappear from search results while the reindexing is in progress. 
       - ``Off``
+    * - reduce-solr-deletes
+      - Avoids deleting and recreating solr documents for dataset files when reindexing. 
+      - ``Off``
     * - index-harvested-metadata-source
       - If enabled, this will index the name of the Harvesting Client as the "Metadata Source" of harvested datasets and files; so that the Metadata Source facet on the collection page will be showing separate entries for the content harvested from different sources/via different clients, instead of the current, default behavior where there is one "Harvested" facet for all such content. Requires a reindex.
       - ``Off``

src/main/java/edu/harvard/iq/dataverse/DataFileServiceBean.java

@@ -1,5 +1,6 @@
 package edu.harvard.iq.dataverse;
 
+import edu.harvard.iq.dataverse.DatasetVersion.VersionState;
 import edu.harvard.iq.dataverse.authorization.users.AuthenticatedUser;
 import edu.harvard.iq.dataverse.dataaccess.DataAccess;
 import edu.harvard.iq.dataverse.dataaccess.ImageThumbConverter;
@@ -383,7 +384,8 @@ public FileMetadata findMostRecentVersionFileIsIn(DataFile file) {
         if (fileMetadatas == null || fileMetadatas.isEmpty()) {
             return null;
         } else {
-            return fileMetadatas.get(0);
+            // This assumes the order of filemetadatas is from first to most recent, which is true as of v6.3 
+            return fileMetadatas.get(fileMetadatas.size() - 1);
         }
     }
 
@@ -759,6 +761,13 @@ public List<DataFile> findAll() {
         return em.createQuery("select object(o) from DataFile as o order by o.id", DataFile.class).getResultList();
     }
 
+    public List<VersionState> findVersionStates(Long fileId) {
+        Query query = em.createQuery(
+                "select distinct dv.versionState from DatasetVersion dv where dv.id in (select fm.datasetVersion.id from FileMetadata fm where fm.dataFile.id=:fileId)");
+        query.setParameter("fileId", fileId);
+        return query.getResultList();
+    }
+
     public DataFile save(DataFile dataFile) {
 
         if (dataFile.isMergeable()) {   

src/main/java/edu/harvard/iq/dataverse/GuestbookResponse.java

@@ -17,6 +17,8 @@
 import java.util.List;
 import jakarta.persistence.*;
 import jakarta.validation.constraints.Size;
+import java.util.Collections;
+import java.util.Comparator;
 
 /**
  *
@@ -178,7 +180,7 @@ public GuestbookResponse(GuestbookResponse source){
         this.setSessionId(source.getSessionId());
         List <CustomQuestionResponse> customQuestionResponses = new ArrayList<>();
         if (!source.getCustomQuestionResponses().isEmpty()){
-            for (CustomQuestionResponse customQuestionResponse : source.getCustomQuestionResponses() ){
+            for (CustomQuestionResponse customQuestionResponse : source.getCustomQuestionResponsesSorted() ){
                 CustomQuestionResponse customQuestionResponseAdd = new CustomQuestionResponse();
                 customQuestionResponseAdd.setResponse(customQuestionResponse.getResponse());  
                 customQuestionResponseAdd.setCustomQuestion(customQuestionResponse.getCustomQuestion());
@@ -254,6 +256,18 @@ public String getResponseDate() {
     public List<CustomQuestionResponse> getCustomQuestionResponses() {
         return customQuestionResponses;
     }
+
+    public List<CustomQuestionResponse> getCustomQuestionResponsesSorted(){
+
+        Collections.sort(customQuestionResponses, (CustomQuestionResponse cqr1, CustomQuestionResponse cqr2) -> {
+            int a = cqr1.getCustomQuestion().getDisplayOrder();
+            int b = cqr2.getCustomQuestion().getDisplayOrder();
+            return Integer.valueOf(a).compareTo(b);
+        });
+
+
+       return customQuestionResponses;
+    }
 
     public void setCustomQuestionResponses(List<CustomQuestionResponse> customQuestionResponses) {
         this.customQuestionResponses = customQuestionResponses;
@@ -317,7 +331,11 @@ public void setSessionId(String sessionId) {
         this.sessionId= sessionId;
     }
 
-    public String toHtmlFormattedResponse() {
+    public String toHtmlFormattedResponse(){
+        return toHtmlFormattedResponse(null);
+    }
+
+    public String toHtmlFormattedResponse(AuthenticatedUser requestor) {
 
         StringBuilder sb = new StringBuilder();
 
@@ -326,17 +344,25 @@ public String toHtmlFormattedResponse() {
         sb.append(BundleUtil.getStringFromBundle("dataset.guestbookResponse.respondent") + "<br><ul style=\"list-style-type:none;\">\n<li>"
                 + BundleUtil.getStringFromBundle("name") + ": " + getName() + "</li>\n<li>");
         sb.append("  " + BundleUtil.getStringFromBundle("email") + ": " + getEmail() + "</li>\n<li>");
-        sb.append(
-                "  " + BundleUtil.getStringFromBundle("institution") + ": " + wrapNullAnswer(getInstitution()) + "</li>\n<li>");
-        sb.append("  " + BundleUtil.getStringFromBundle("position") + ": " + wrapNullAnswer(getPosition()) + "</li></ul>\n");
+        sb.append("  " + BundleUtil.getStringFromBundle("institution") + ": " + wrapNullAnswer(getInstitution()) + "</li>\n<li>");
+        sb.append("  " + BundleUtil.getStringFromBundle("position") + ": " + wrapNullAnswer(getPosition()) + "</li>");
+
+        //Add requestor information to response to help dataset admin with request processing
+        if (requestor != null){
+            sb.append("\n<li>" + BundleUtil.getStringFromBundle("dataset.guestbookResponse.requestor.id") + ": " + requestor.getId()+ "</li>");
+            sb.append("\n<li>" + BundleUtil.getStringFromBundle("dataset.guestbookResponse.requestor.identifier") + ": " + requestor.getIdentifier()+ "</li></ul>\n");
+        } else {
+            sb.append("</ul>\n");
+        }
+
         sb.append(BundleUtil.getStringFromBundle("dataset.guestbookResponse.guestbook.additionalQuestions")
                 + ":<ul style=\"list-style-type:none;\">\n");
 
-        for (CustomQuestionResponse cqr : getCustomQuestionResponses()) {
+        for (CustomQuestionResponse cqr : getCustomQuestionResponsesSorted()) {
             sb.append("<li>" + BundleUtil.getStringFromBundle("dataset.guestbookResponse.question") + ": "
                     + cqr.getCustomQuestion().getQuestionString() + "<br>"
                     + BundleUtil.getStringFromBundle("dataset.guestbookResponse.answer") + ": "
-                    + wrapNullAnswer(cqr.getResponse()) + "</li>\n");
+                    + wrapNullAnswer(cqr.getResponse()) + "</li>\n<br>");
         }
         sb.append("</ul>");
         return sb.toString();

src/main/java/edu/harvard/iq/dataverse/MailServiceBean.java

@@ -456,7 +456,7 @@ public String getMessageTextBasedOnNotification(UserNotification userNotificatio
                 GuestbookResponse gbr = far.getGuestbookResponse();
                 if (gbr != null) {
                     messageText += MessageFormat.format(
-                            BundleUtil.getStringFromBundle("notification.email.requestFileAccess.guestbookResponse"), gbr.toHtmlFormattedResponse());
+                        BundleUtil.getStringFromBundle("notification.email.requestFileAccess.guestbookResponse"), gbr.toHtmlFormattedResponse(requestor));
                 }
                 return messageText;
             case GRANTFILEACCESS: