diff --git a/doc/sphinx-guides/source/installation/config.rst b/doc/sphinx-guides/source/installation/config.rst index 5e2978da108..36d3d288664 100644 --- a/doc/sphinx-guides/source/installation/config.rst +++ b/doc/sphinx-guides/source/installation/config.rst @@ -110,31 +110,58 @@ Publishing the Root Dataverse Non-superusers who are not "Admin" on the root dataverse will not be able to to do anything useful until the root dataverse has been published. +Customizing the Root Dataverse +++++++++++++++++++++++++++++++ + +As the person installing Dataverse you may or may not be a local metadata expert. You may want to have others sign up for accounts and grant them the "Admin" role at the root dataverse to configure metadata fields, templates, browse/search facets, guestbooks, etc. For more on these topics, consult the :doc:`/user/dataverse-management` section of the User Guide. + +Once this configuration is complete, your Dataverse installation should be ready for users to start playing with. That said, there are many more configuration options available, which will be explained below. + Persistent Identifiers and Publishing Datasets -++++++++++++++++++++++++++++++++++++++++++++++ +---------------------------------------------- -Persistent identifiers are a required and integral part of the Dataverse platform. They provide a URL that is guaranteed to resolve to the datasets they represent. Dataverse currently supports creating identifiers using DOI and HDL. +Persistent identifiers are a required and integral part of the Dataverse platform. They provide a URL that is guaranteed to resolve to the datasets they represent. Dataverse currently supports creating identifiers using DOI and Handle. By default and for testing convenience, the installer configures a temporary DOI test namespace through EZID. This is sufficient to create and publish datasets but they are not citable nor guaranteed to be preserved. Note that any datasets creating using the test configuration cannot be directly migrated and would need to be created again once a valid DOI namespace is configured. To properly configure persistent identifiers for a production installation, an account and associated namespace must be acquired for a fee from a DOI or HDL provider: **EZID** (http://ezid.cdlib.org), **DataCite** (https://www.datacite.org), **Handle.Net** (https://www.handle.net). -Once account credentials and namespace have been acquired, please complete the identifier configuration parameters that are relevant to your installation. In the following list, parameters that apply only to DOI are preceded by "doi", those that apply only to handles include "handlenet", and those that apply to both kinds of installation contain neither. +Once you have your DOI or Handle account credentials and a namespace, configure Dataverse to use them using the JVM options and database settings below. -**JVM Options:** :ref:`doi.baseurlstring`, :ref:`doi.username`, :ref:`doi.password`, :ref:`dataverse.handlenet.admcredfile`, :ref:`dataverse.handlenet.admprivphrase` +Configuring Dataverse for DOIs +++++++++++++++++++++++++++++++ -**Database Settings:** :ref:`:DoiProvider <:DoiProvider>`, :ref:`:Protocol <:Protocol>`, :ref:`:Authority <:Authority>`, :ref:`:DoiSeparator <:DoiSeparator>` +Out of the box, Dataverse is configured for DOIs. Here are the configuration options for DOIs: -Note: If you are **minting your own handles** and plan to set up your own handle service, please refer to `Handle.Net documentation `_. +**JVM Options:** + +- :ref:`doi.baseurlstring` +- :ref:`doi.username` +- :ref:`doi.password` +**Database Settings:** +- :ref:`:DoiProvider <:DoiProvider>` +- :ref:`:Protocol <:Protocol>` +- :ref:`:Authority <:Authority>` +- :ref:`:DoiSeparator <:DoiSeparator>` -Customizing the Root Dataverse -++++++++++++++++++++++++++++++ +Configuring Dataverse for Handles ++++++++++++++++++++++++++++++++++ -As the person installing Dataverse you may or may not be local metadata expert. You may want to have others sign up for accounts and grant them the "Admin" role at the root dataverse to configure metadata fields, browse/search facets, templates, guestbooks, etc. For more on these topics, consult the :doc:`/user/dataverse-management` section of the User Guide. +Here are the configuration options for handles: -Once this configuration is complete, your Dataverse installation should be ready for users to start playing with it. That said, there are many more configuration options available, which will be explained below. +**JVM Options:** + +- :ref:`dataverse.handlenet.admcredfile` +- :ref:`dataverse.handlenet.admprivphrase` + +**Database Settings:** + +- :ref:`:Protocol <:Protocol>` +- :ref:`:Authority <:Authority>` + +Note: If you are **minting your own handles** and plan to set up your own handle service, please refer to `Handle.Net documentation `_. Auth Modes: Local vs. Remote vs. Both ------------------------------------- diff --git a/doc/sphinx-guides/source/installation/intro.rst b/doc/sphinx-guides/source/installation/intro.rst index f3a2d226fb0..9a6b40cda89 100644 --- a/doc/sphinx-guides/source/installation/intro.rst +++ b/doc/sphinx-guides/source/installation/intro.rst @@ -39,4 +39,6 @@ To get help installing or configuring Dataverse, please try one or more of: Improving this Guide -------------------- -If you spot a typo in this guide or would like to suggest an improvement, please find the appropriate file in https://github.com/IQSS/dataverse/tree/develop/doc/sphinx-guides/source/installation and send a pull request. You are also welcome to simply open an issue at https://github.com/IQSS/dataverse/issues to describe the problem with this guide. +If you spot a typo in this guide or would like to suggest an improvement, please find the appropriate file in https://github.com/IQSS/dataverse/tree/develop/doc/sphinx-guides/source/installation and send a pull request as explained in the :doc:`/developers/documentation` section of the Developer Guide. You are also welcome to simply open an issue at https://github.com/IQSS/dataverse/issues to describe the problem with this guide. + +Next is the :doc:`prep` section. diff --git a/doc/sphinx-guides/source/installation/prep.rst b/doc/sphinx-guides/source/installation/prep.rst index d6d531cdfaf..9452fdf5d3b 100644 --- a/doc/sphinx-guides/source/installation/prep.rst +++ b/doc/sphinx-guides/source/installation/prep.rst @@ -37,10 +37,11 @@ There are some community-lead projects to use configuration management tools suc The Dataverse development team is happy to "bless" additional community efforts along these lines (i.e. Docker, Chef, Salt, etc.) by creating a repo under https://github.com/IQSS and managing team access. -Dataverse permits a fair amount of flexibility in where you choose to install the various components. The diagram below shows a load balancer, multiple proxies and web servers, redundant database servers, and offloading of potentially resource intensive work to a separate server. A setup such as this is advanced enough to be considered out of scope for this guide but you are welcome to ask questions about similar configurations via the support channels listed in the :doc:`intro`. +Dataverse permits a fair amount of flexibility in where you choose to install the various components. The diagram below shows a load balancer, multiple proxies and web servers, redundant database servers, and offloading of potentially resource intensive work to a separate server. |3webservers| +A setup such as this is advanced enough to be considered out of scope for this guide but you are welcome to ask questions about similar configurations via the support channels listed in the :doc:`intro`. .. _architecture: @@ -56,7 +57,7 @@ When planning your installation you should be aware of the following components - PostgreSQL: a relational database. - Solr: a search engine. A Dataverse-specific schema is provided. - SMTP server: for sending mail for password resets and other notifications. -- Persistent identifier service: DOI support is provided. An EZID subscription or DataCite account is required for production use. +- Persistent identifier service: DOI and Handle support are provided. Production use requires a registered DOI or Handle.net authority. There are a number of optional components you may choose to install or configure, including: @@ -98,7 +99,7 @@ Here are some questions to keep in the back of your mind as you test and move in - How do I want my users to log in to Dataverse? With local accounts? With Shibboleth/SAML? With OAuth providers such as ORCID, GitHub, or Google? - Do I want to to run Glassfish on the standard web ports (80 and 443) or do I want to "front" Glassfish with a proxy such as Apache or nginx? See "Network Ports" in the :doc:`config` section. - How many points of failure am I willing to tolerate? How much complexity do I want? -- How much does it cost to subscribe to a service to create persistent identifiers such as DOIs? +- How much does it cost to subscribe to a service to create persistent identifiers such as DOIs or handles? Next Steps ---------- diff --git a/src/main/java/edu/harvard/iq/dataverse/DOIEZIdServiceBean.java b/src/main/java/edu/harvard/iq/dataverse/DOIEZIdServiceBean.java index fe31618962c..f1a53e8ff8f 100644 --- a/src/main/java/edu/harvard/iq/dataverse/DOIEZIdServiceBean.java +++ b/src/main/java/edu/harvard/iq/dataverse/DOIEZIdServiceBean.java @@ -61,10 +61,15 @@ public boolean registerWhenPublished() { public boolean alreadyExists(Dataset dataset) throws Exception { logger.log(Level.FINE,"alreadyExists"); try { - HashMap result = ezidService.getMetadata(getIdentifierFromDataset(dataset)); + HashMap result = ezidService.getMetadata(getIdentifierFromDataset(dataset)); return result != null && !result.isEmpty(); // TODO just check for HTTP status code 200/404, sadly the status code is swept under the carpet } catch (EZIDException e ){ + //No such identifier is treated as an exception + //but if that is the case then we want to just return false + if (e.getLocalizedMessage().contains("no such identifier")){ + return false; + } logger.log(Level.WARNING, "alreadyExists failed"); logger.log(Level.WARNING, "String {0}", e.toString()); logger.log(Level.WARNING, "localized message {0}", e.getLocalizedMessage()); diff --git a/src/main/java/edu/harvard/iq/dataverse/DataFileServiceBean.java b/src/main/java/edu/harvard/iq/dataverse/DataFileServiceBean.java index 5413aa035aa..0599f508ddd 100644 --- a/src/main/java/edu/harvard/iq/dataverse/DataFileServiceBean.java +++ b/src/main/java/edu/harvard/iq/dataverse/DataFileServiceBean.java @@ -446,7 +446,8 @@ public DataFile findCheapAndEasy(Long id) { dataFile.setOwner(owner); - // look up data table; but only if content type indicates it's tabular data: + // If content type indicates it's tabular data, spend 2 extra queries + // looking up the data table and tabular tags objects: if (MIME_TYPE_TAB.equalsIgnoreCase(contentType)) { Object[] dtResult = null; @@ -471,6 +472,28 @@ public DataFile findCheapAndEasy(Long id) { dataTable.setDataFile(dataFile); dataFile.setDataTable(dataTable); + + // tabular tags: + + List tagResults = null; + try { + tagResults = em.createNativeQuery("SELECT t.TYPE, t.DATAFILE_ID FROM DATAFILETAG t WHERE t.DATAFILE_ID = " + id).getResultList(); + } catch (Exception ex) { + logger.info("EXCEPTION looking up tags."); + tagResults = null; + } + + if (tagResults != null) { + List fileTagLabels = DataFileTag.listTags(); + + for (Object[] tagResult : tagResults) { + Integer tagId = (Integer)tagResult[0]; + DataFileTag tag = new DataFileTag(); + tag.setTypeByLabel(fileTagLabels.get(tagId)); + tag.setDataFile(dataFile); + dataFile.addTag(tag); + } + } } } @@ -1011,8 +1034,6 @@ public boolean isThumbnailAvailable (DataFile file) { queries; checking if the thumbnail is available may cost cpu time, if it has to be generated on the fly - so you have to figure out which is more important... - TODO: adding a boolean flag isImageAlreadyGenerated to the DataFile - db table should help with this. -- L.A. 4.2.1 DONE: 4.2.2 */ diff --git a/src/main/java/edu/harvard/iq/dataverse/Dataset.java b/src/main/java/edu/harvard/iq/dataverse/Dataset.java index 79be18642f9..9b376b5f5dc 100644 --- a/src/main/java/edu/harvard/iq/dataverse/Dataset.java +++ b/src/main/java/edu/harvard/iq/dataverse/Dataset.java @@ -723,4 +723,16 @@ public DatasetThumbnail getDatasetThumbnail() { return DatasetUtil.getThumbnail(this); } + /** + * Handle the case where we also have the datasetVersionId. + * This saves trying to find the latestDatasetVersion, and + * other costly queries, etc. + * + * @param datasetVersionId + * @return + */ + public DatasetThumbnail getDatasetThumbnail(DatasetVersion datasetVersion) { + return DatasetUtil.getThumbnail(this, datasetVersion); + } + } diff --git a/src/main/java/edu/harvard/iq/dataverse/DatasetPage.java b/src/main/java/edu/harvard/iq/dataverse/DatasetPage.java index 7407bff4c97..dd1daefa2c6 100644 --- a/src/main/java/edu/harvard/iq/dataverse/DatasetPage.java +++ b/src/main/java/edu/harvard/iq/dataverse/DatasetPage.java @@ -160,10 +160,16 @@ public enum DisplayMode { DataverseRequestServiceBean dvRequestService; @Inject DatasetVersionUI datasetVersionUI; - @Inject PermissionsWrapper permissionsWrapper; - @Inject FileDownloadHelper fileDownloadHelper; - @Inject TwoRavensHelper twoRavensHelper; - @Inject WorldMapPermissionHelper worldMapPermissionHelper; + @Inject + PermissionsWrapper permissionsWrapper; + @Inject + FileDownloadHelper fileDownloadHelper; + @Inject + TwoRavensHelper twoRavensHelper; + @Inject + WorldMapPermissionHelper worldMapPermissionHelper; + @Inject + ThumbnailServiceWrapper thumbnailServiceWrapper; @Inject SettingsWrapper settingsWrapper; @@ -242,13 +248,30 @@ public String getThumbnailString() { return thumbnailString; } + if (!readOnly) { DatasetThumbnail datasetThumbnail = dataset.getDatasetThumbnail(); if (datasetThumbnail == null) { thumbnailString = ""; return null; } + + if (datasetThumbnail.isFromDataFile()) { + if (!datasetThumbnail.getDataFile().equals(dataset.getThumbnailFile())) { + datasetService.assignDatasetThumbnailByNativeQuery(dataset, datasetThumbnail.getDataFile()); + dataset = datasetService.find(dataset.getId()); + } + } thumbnailString = datasetThumbnail.getBase64image(); + } else { + thumbnailString = thumbnailServiceWrapper.getDatasetCardImageAsBase64Url(dataset, workingVersion.getId()); + if (thumbnailString == null) { + thumbnailString = ""; + return null; + } + + + } return thumbnailString; } @@ -2294,9 +2317,6 @@ public String save() { //FacesContext.getCurrentInstance().addMessage(null, new FacesMessage(FacesMessage.SEVERITY_ERROR, "Validation Error", "See below for details.")); return ""; } - - // Finally, save the files permanently: - ingestService.addFiles(workingVersion, newFiles); // Use the API to save the dataset: Command cmd; diff --git a/src/main/java/edu/harvard/iq/dataverse/DatasetServiceBean.java b/src/main/java/edu/harvard/iq/dataverse/DatasetServiceBean.java index f8855984065..1aea56d279e 100644 --- a/src/main/java/edu/harvard/iq/dataverse/DatasetServiceBean.java +++ b/src/main/java/edu/harvard/iq/dataverse/DatasetServiceBean.java @@ -848,5 +848,23 @@ public Dataset removeDatasetThumbnail(Dataset dataset) { dataset.setUseGenericThumbnail(true); return merge(dataset); } + + // persist assigned thumbnail in a single one-field-update query: + // (the point is to avoid doing an em.merge() on an entire dataset object...) + public void assignDatasetThumbnailByNativeQuery(Long datasetId, Long dataFileId) { + try { + em.createNativeQuery("UPDATE dataset SET thumbnailfile_id=" + dataFileId + " WHERE id=" + datasetId).executeUpdate(); + } catch (Exception ex) { + // it's ok to just ignore... + } + } + + public void assignDatasetThumbnailByNativeQuery(Dataset dataset, DataFile dataFile) { + try { + em.createNativeQuery("UPDATE dataset SET thumbnailfile_id=" + dataFile.getId() + " WHERE id=" + dataset.getId()).executeUpdate(); + } catch (Exception ex) { + // it's ok to just ignore... + } + } } diff --git a/src/main/java/edu/harvard/iq/dataverse/DatasetVersionServiceBean.java b/src/main/java/edu/harvard/iq/dataverse/DatasetVersionServiceBean.java index 84517d045d6..69c1be49b3b 100644 --- a/src/main/java/edu/harvard/iq/dataverse/DatasetVersionServiceBean.java +++ b/src/main/java/edu/harvard/iq/dataverse/DatasetVersionServiceBean.java @@ -687,6 +687,107 @@ public RetrieveDatasetVersionResponse retrieveDatasetVersionByVersionId(Long ver return null; } // end: retrieveDatasetVersionByVersionId + public Long getThumbnailByVersionId(Long versionId) { + if (versionId == null) { + return null; + } + + Long thumbnailFileId; + + // First, let's see if there are thumbnails that have already been + // generated: + try { + thumbnailFileId = (Long) em.createNativeQuery("SELECT df.id " + + "FROM datafile df, filemetadata fm, datasetversion dv, dvobject o " + + "WHERE dv.id = " + versionId + " " + + "AND df.id = o.id " + + "AND fm.datasetversion_id = dv.id " + + "AND fm.datafile_id = df.id " + + "AND o.previewImageAvailable = true " + + "ORDER BY df.id LIMIT 1;").getSingleResult(); + } catch (Exception ex) { + thumbnailFileId = null; + } + + if (thumbnailFileId != null) { + logger.fine("DatasetVersionService,getThumbnailByVersionid(): found already generated thumbnail for version " + versionId + ": " + thumbnailFileId); + assignDatasetThumbnailByNativeQuery(versionId, thumbnailFileId); + return thumbnailFileId; + } + + if (!systemConfig.isThumbnailGenerationDisabledForImages()) { + // OK, let's try and generate an image thumbnail! + long imageThumbnailSizeLimit = systemConfig.getThumbnailSizeLimitImage(); + + try { + thumbnailFileId = (Long) em.createNativeQuery("SELECT df.id " + + "FROM datafile df, filemetadata fm, datasetversion dv, dvobject o " + + "WHERE dv.id = " + versionId + " " + + "AND df.id = o.id " + + "AND fm.datasetversion_id = dv.id " + + "AND fm.datafile_id = df.id " + // + "AND o.previewImageAvailable = false " + + "AND df.contenttype LIKE 'image/%' " + + "AND NOT df.contenttype = 'image/fits' " + + "AND df.filesize < " + imageThumbnailSizeLimit + " " + + "ORDER BY df.filesize ASC LIMIT 1;").getSingleResult(); + } catch (Exception ex) { + thumbnailFileId = null; + } + + if (thumbnailFileId != null) { + logger.fine("obtained file id: " + thumbnailFileId); + DataFile thumbnailFile = datafileService.find(thumbnailFileId); + if (thumbnailFile != null) { + if (datafileService.isThumbnailAvailable(thumbnailFile)) { + assignDatasetThumbnailByNativeQuery(versionId, thumbnailFileId); + return thumbnailFileId; + } + } + } + } + + // And if that didn't work, try the same thing for PDFs: + if (!systemConfig.isThumbnailGenerationDisabledForPDF()) { + // OK, let's try and generate an image thumbnail! + long imageThumbnailSizeLimit = systemConfig.getThumbnailSizeLimitPDF(); + try { + thumbnailFileId = (Long) em.createNativeQuery("SELECT df.id " + + "FROM datafile df, filemetadata fm, datasetversion dv, dvobject o " + + "WHERE dv.id = " + versionId + " " + + "AND df.id = o.id " + + "AND fm.datasetversion_id = dv.id " + + "AND fm.datafile_id = df.id " + // + "AND o.previewImageAvailable = false " + + "AND df.contenttype = 'application/pdf' " + + "AND df.filesize < " + imageThumbnailSizeLimit + " " + + "ORDER BY df.filesize ASC LIMIT 1;").getSingleResult(); + } catch (Exception ex) { + thumbnailFileId = null; + } + + if (thumbnailFileId != null) { + DataFile thumbnailFile = datafileService.find(thumbnailFileId); + if (thumbnailFile != null) { + if (datafileService.isThumbnailAvailable(thumbnailFile)) { + assignDatasetThumbnailByNativeQuery(versionId, thumbnailFileId); + return thumbnailFileId; + } + } + } + } + + return null; + } + + private void assignDatasetThumbnailByNativeQuery(Long versionId, Long dataFileId) { + try { + em.createNativeQuery("UPDATE dataset SET thumbnailfile_id=" + dataFileId + " WHERE id in (SELECT dataset_id FROM datasetversion WHERE id=" + versionId + ")").executeUpdate(); + } catch (Exception ex) { + // it's ok to just ignore... + } + } + public void populateDatasetSearchCard(SolrSearchResult solrSearchResult) { Long dataverseId = Long.parseLong(solrSearchResult.getParent().get("id")); Long datasetVersionId = solrSearchResult.getDatasetVersionId(); @@ -700,7 +801,7 @@ public void populateDatasetSearchCard(SolrSearchResult solrSearchResult) { try { if (datasetId != null) { - searchResult = (Object[]) em.createNativeQuery("SELECT t0.VERSIONSTATE, t1.ALIAS, t2.THUMBNAILFILE_ID FROM DATASETVERSION t0, DATAVERSE t1, DATASET t2 WHERE t0.ID = " + searchResult = (Object[]) em.createNativeQuery("SELECT t0.VERSIONSTATE, t1.ALIAS, t2.THUMBNAILFILE_ID, t2.USEGENERICTHUMBNAIL FROM DATASETVERSION t0, DATAVERSE t1, DATASET t2 WHERE t0.ID = " + datasetVersionId + " AND t1.ID = " + dataverseId @@ -709,6 +810,7 @@ public void populateDatasetSearchCard(SolrSearchResult solrSearchResult) { ; } else { + // Why is this method ever called with dataset_id = null? -- L.A. searchResult = (Object[]) em.createNativeQuery("SELECT t0.VERSIONSTATE, t1.ALIAS FROM DATASETVERSION t0, DATAVERSE t1 WHERE t0.ID = " + datasetVersionId + " AND t1.ID = " + dataverseId).getSingleResult(); } } catch (Exception ex) { @@ -734,23 +836,38 @@ public void populateDatasetSearchCard(SolrSearchResult solrSearchResult) { solrSearchResult.setDataverseAlias((String) searchResult[1]); } - if (searchResult.length == 3 && searchResult[2] != null) { - // This is the image file specifically assigned as the "icon" for - // the dataset: - Long thumbnailFile_id = (Long)searchResult[2]; - if (thumbnailFile_id != null) { - DataFile thumbnailFile = null; - try { - thumbnailFile = datafileService.findCheapAndEasy(thumbnailFile_id); - } catch (Exception ex) { - thumbnailFile = null; - } - - if (thumbnailFile != null) { - solrSearchResult.setEntity(new Dataset()); - ((Dataset)solrSearchResult.getEntity()).setThumbnailFile(thumbnailFile); + if (searchResult.length == 4) { + Dataset datasetEntity = new Dataset(); + String globalIdentifier = solrSearchResult.getIdentifier(); + GlobalId globalId = new GlobalId(globalIdentifier); + + datasetEntity.setProtocol(globalId.getProtocol()); + datasetEntity.setAuthority(globalId.getAuthority()); + datasetEntity.setIdentifier(globalId.getIdentifier()); + + solrSearchResult.setEntity(datasetEntity); + if (searchResult[2] != null) { + // This is the image file specifically assigned as the "icon" for + // the dataset: + Long thumbnailFile_id = (Long) searchResult[2]; + if (thumbnailFile_id != null) { + DataFile thumbnailFile = null; + try { + thumbnailFile = datafileService.findCheapAndEasy(thumbnailFile_id); + } catch (Exception ex) { + thumbnailFile = null; + } + + if (thumbnailFile != null) { + ((Dataset) solrSearchResult.getEntity()).setThumbnailFile(thumbnailFile); + } } } + if (searchResult[3] != null) { + ((Dataset)solrSearchResult.getEntity()).setUseGenericThumbnail((Boolean) searchResult[3]); + } else { + ((Dataset)solrSearchResult.getEntity()).setUseGenericThumbnail(false); + } } } diff --git a/src/main/java/edu/harvard/iq/dataverse/GlobalId.java b/src/main/java/edu/harvard/iq/dataverse/GlobalId.java index 04a195d7aff..3761245f583 100644 --- a/src/main/java/edu/harvard/iq/dataverse/GlobalId.java +++ b/src/main/java/edu/harvard/iq/dataverse/GlobalId.java @@ -28,10 +28,6 @@ public class GlobalId implements java.io.Serializable { @EJB SettingsServiceBean settingsService; - // I'm marking this constructor as "deprecated" because it's not reliable; - // it uses the parser (below) that makes incorrect assumptions about - // handles and dois. (see comments there) - @Deprecated public GlobalId(String identifier) { // set the protocol, authority, and identifier via parsePersistentId @@ -112,71 +108,44 @@ public URL toURL() { * authority: 1902.1 * identifier: 111012 * - * @param persistentId + * @param identifierString * */ - // This parser makes an incorrect assumption, that a handle has to be made of 2 - // "/"-separated parts, and a doi - of 3. ICPSR's DOIs are an example of a DOI - // that has 2 parts only: doi:10.3886/ICPSR24006.v2 - // We already have working global id parsers elsewhere in the app: - // for ex., parseStudyIdDOI() and parseStudyIdHandle() in importDDIServiceBean; - // We should probably copy that code here, than change the app so that all - // the pieces are using this class, instead of replicating the functionality - // in multiple places. - - @Deprecated - private boolean parsePersistentId(String persistentId){ - - if (persistentId==null){ + + private boolean parsePersistentId(String identifierString){ + + if (identifierString == null){ return false; } - String doiSeparator = "/";//settingsService.getValueForKey(SettingsServiceBean.Key.DoiSeparator, "/"); + int index1 = identifierString.indexOf(':'); + int index2 = identifierString.lastIndexOf('/'); + if (index1==-1) { + return false; + } + + String protocol = identifierString.substring(0, index1); - // Looking for this split - // doi:10.5072/FK2/BYM3IW => (doi) (10.5072/FK2/BYM3IW) - // - // or this one: (hdl) (1902.1/xxxxx) - // - String[] items = persistentId.split(":"); - if (items.length != 2){ + if (!"doi".equals(protocol) && !"hdl".equals(protocol)) { return false; } - String protocolPiece = items[0].toLowerCase(); - String[] pieces = items[1].split(doiSeparator); - - // ----------------------------- - // Is this a handle? - // ----------------------------- - if ( pieces.length == 2 && protocolPiece.equals(HDL_PROTOCOL)){ - // example: hdl:1902.1/111012 - - this.protocol = protocolPiece; // hdl - this.authority = formatIdentifierString(pieces[0]); // 1902.1 - this.identifier = formatIdentifierString(pieces[1]); // 111012 - return true; - - }else if (pieces.length == 3 && protocolPiece.equals(DOI_PROTOCOL)){ - // ----------------------------- - // Is this a DOI? - // ----------------------------- - // example: doi:10.5072/FK2/BYM3IW - - this.protocol = protocolPiece; // doi - - String doiAuthority = formatIdentifierString(pieces[0]) + doiSeparator + formatIdentifierString(pieces[1]); // "10.5072/FK2" - if (this.checkDOIAuthority(doiAuthority)){ - this.authority = doiAuthority; - } else { + + if (index2 == -1) { + return false; + } + + this.protocol = protocol; + this.authority = formatIdentifierString(identifierString.substring(index1+1, index2)); + this.identifier = formatIdentifierString(identifierString.substring(index2+1)); + + if (this.protocol.equals(DOI_PROTOCOL)) { + if (!this.checkDOIAuthority(this.authority)) { return false; } - - this.identifier = formatIdentifierString(pieces[2]); // "BYM3IW" - return true; } - - return false; + return true; + } diff --git a/src/main/java/edu/harvard/iq/dataverse/ThumbnailServiceWrapper.java b/src/main/java/edu/harvard/iq/dataverse/ThumbnailServiceWrapper.java new file mode 100644 index 00000000000..a1db6d88303 --- /dev/null +++ b/src/main/java/edu/harvard/iq/dataverse/ThumbnailServiceWrapper.java @@ -0,0 +1,283 @@ +/* + * To change this license header, choose License Headers in Project Properties. + * To change this template file, choose Tools | Templates + * and open the template in the editor. + */ +package edu.harvard.iq.dataverse; + +import edu.harvard.iq.dataverse.dataaccess.ImageThumbConverter; +import edu.harvard.iq.dataverse.dataset.DatasetUtil; +import edu.harvard.iq.dataverse.search.SolrSearchResult; +import edu.harvard.iq.dataverse.util.FileUtil; +import java.io.File; +import java.io.IOException; +import java.nio.file.Files; +import java.nio.file.Path; +import java.nio.file.Paths; +import java.util.Base64; +import java.util.HashMap; +import java.util.Map; +import javax.ejb.EJB; +import javax.faces.view.ViewScoped; +import javax.inject.Inject; +import javax.inject.Named; + +/** + * + * @author Leonid Andreev + */ +@ViewScoped +@Named +public class ThumbnailServiceWrapper implements java.io.Serializable { + @Inject + PermissionsWrapper permissionsWrapper; + @EJB + DataverseServiceBean dataverseService; + @EJB + DatasetServiceBean datasetService; + @EJB + DatasetVersionServiceBean datasetVersionService; + @EJB + DataFileServiceBean dataFileService; + + private Map dvobjectThumbnailsMap = new HashMap<>(); + private Map dvobjectViewMap = new HashMap<>(); + + private String getAssignedDatasetImage(Dataset dataset) { + if (dataset == null) { + return null; + } + + DataFile assignedThumbnailFile = dataset.getThumbnailFile(); + + if (assignedThumbnailFile != null) { + Long assignedThumbnailFileId = assignedThumbnailFile.getId(); + + if (this.dvobjectThumbnailsMap.containsKey(assignedThumbnailFileId)) { + // Yes, return previous answer + //logger.info("using cached result for ... "+assignedThumbnailFileId); + if (!"".equals(this.dvobjectThumbnailsMap.get(assignedThumbnailFileId))) { + return this.dvobjectThumbnailsMap.get(assignedThumbnailFileId); + } + return null; + } + + String imageSourceBase64 = ImageThumbConverter.getImageThumbnailAsBase64( + assignedThumbnailFile, + ImageThumbConverter.DEFAULT_CARDIMAGE_SIZE); + + if (imageSourceBase64 != null) { + this.dvobjectThumbnailsMap.put(assignedThumbnailFileId, imageSourceBase64); + return imageSourceBase64; + } + + // OK - we can't use this "assigned" image, because of permissions, or because + // the thumbnail failed to generate, etc... in this case we'll + // mark this dataset in the lookup map - so that we don't have to + // do all these lookups again... + this.dvobjectThumbnailsMap.put(assignedThumbnailFileId, ""); + + // TODO: (?) + // do we need to cache this datafile object in the view map? + // -- L.A., 4.2.2 + } + + return null; + + } + + // it's the responsibility of the user - to make sure the search result + // passed to this method is of the Datafile type! + public String getFileCardImageAsBase64Url(SolrSearchResult result) { + // Before we do anything else, check if it's a harvested dataset; + // no need to check anything else if so (harvested objects never have + // thumbnails) + + if (result.isHarvested()) { + return null; + } + + Long imageFileId = result.getEntity().getId(); + + if (imageFileId != null) { + if (this.dvobjectThumbnailsMap.containsKey(imageFileId)) { + // Yes, return previous answer + //logger.info("using cached result for ... "+datasetId); + if (!"".equals(this.dvobjectThumbnailsMap.get(imageFileId))) { + return this.dvobjectThumbnailsMap.get(imageFileId); + } + return null; + } + + String cardImageUrl = null; + + if (result.getTabularDataTags() != null) { + for (String tabularTagLabel : result.getTabularDataTags()) { + DataFileTag tag = new DataFileTag(); + try { + tag.setTypeByLabel(tabularTagLabel); + tag.setDataFile((DataFile) result.getEntity()); + ((DataFile) result.getEntity()).addTag(tag); + } catch (IllegalArgumentException iax) { + // ignore + } + } + } + + if ((!((DataFile)result.getEntity()).isRestricted() + || permissionsWrapper.hasDownloadFilePermission(result.getEntity())) + && dataFileService.isThumbnailAvailable((DataFile) result.getEntity())) { + + cardImageUrl = ImageThumbConverter.getImageThumbnailAsBase64( + (DataFile) result.getEntity(), + ImageThumbConverter.DEFAULT_CARDIMAGE_SIZE); + } + + if (cardImageUrl != null) { + this.dvobjectThumbnailsMap.put(imageFileId, cardImageUrl); + //logger.info("datafile id " + imageFileId + ", returning " + cardImageUrl); + + if (!(dvobjectViewMap.containsKey(imageFileId) + && dvobjectViewMap.get(imageFileId).isInstanceofDataFile())) { + + dvobjectViewMap.put(imageFileId, result.getEntity()); + + } + + return cardImageUrl; + } else { + this.dvobjectThumbnailsMap.put(imageFileId, ""); + } + } + return null; + } + + // it's the responsibility of the user - to make sure the search result + // passed to this method is of the Dataset type! + public String getDatasetCardImageAsBase64Url(SolrSearchResult result) { + // Before we do anything else, check if it's a harvested dataset; + // no need to check anything else if so (harvested datasets never have + // thumbnails) + + if (result.isHarvested()) { + return null; + } + + + Dataset dataset = (Dataset)result.getEntity(); + Long versionId = result.getDatasetVersionId(); + + return getDatasetCardImageAsBase64Url(dataset, versionId); + } + public String getDatasetCardImageAsBase64Url(Dataset dataset, Long versionId) { + Long datasetId = dataset.getId(); + if (datasetId != null) { + if (this.dvobjectThumbnailsMap.containsKey(datasetId)) { + // Yes, return previous answer + // (at max, there could only be 2 cards for the same dataset + // on the page - the draft, and the published version; but it's + // still nice to try and cache the result - especially if it's an + // uploaded logo - we don't want to read it off disk twice). + + if (!"".equals(this.dvobjectThumbnailsMap.get(datasetId))) { + return this.dvobjectThumbnailsMap.get(datasetId); + } + return null; + } + } + + if (dataset.isUseGenericThumbnail()) { + this.dvobjectThumbnailsMap.put(datasetId, ""); + return null; + } + + String cardImageUrl = null; + + Path path = Paths.get(dataset.getFileSystemDirectory() + File.separator + DatasetUtil.datasetLogoThumbnail + DatasetUtil.thumb48addedByImageThumbConverter); + if (Files.exists(path)) { + try { + byte[] bytes = Files.readAllBytes(path); + String base64image = Base64.getEncoder().encodeToString(bytes); + cardImageUrl = FileUtil.DATA_URI_SCHEME + base64image; + this.dvobjectThumbnailsMap.put(datasetId, cardImageUrl); + return cardImageUrl; + } catch (IOException ex) { + this.dvobjectThumbnailsMap.put(datasetId, ""); + return null; + // (alternatively, we could ignore the exception, and proceed with the + // regular process of selecting the thumbnail from the available + // image files - ?) + } + } + + if (dataset != null) { + cardImageUrl = this.getAssignedDatasetImage(dataset); + + if (cardImageUrl != null) { + //logger.info("dataset id " + result.getEntity().getId() + " has a dedicated image assigned; returning " + cardImageUrl); + return cardImageUrl; + } + } + + Long thumbnailImageFileId = datasetVersionService.getThumbnailByVersionId(versionId); + + if (thumbnailImageFileId != null) { + //cardImageUrl = FILE_CARD_IMAGE_URL + thumbnailImageFileId; + if (this.dvobjectThumbnailsMap.containsKey(thumbnailImageFileId)) { + // Yes, return previous answer + //logger.info("using cached result for ... "+datasetId); + if (!"".equals(this.dvobjectThumbnailsMap.get(thumbnailImageFileId))) { + return this.dvobjectThumbnailsMap.get(thumbnailImageFileId); + } + return null; + } + + DataFile thumbnailImageFile = null; + + if (dvobjectViewMap.containsKey(thumbnailImageFileId) + && dvobjectViewMap.get(thumbnailImageFileId).isInstanceofDataFile()) { + thumbnailImageFile = (DataFile) dvobjectViewMap.get(thumbnailImageFileId); + } else { + thumbnailImageFile = dataFileService.findCheapAndEasy(thumbnailImageFileId); + if (thumbnailImageFile != null) { + // TODO: + // do we need this file on the map? - it may not even produce + // a thumbnail! + dvobjectViewMap.put(thumbnailImageFileId, thumbnailImageFile); + } else { + this.dvobjectThumbnailsMap.put(thumbnailImageFileId, ""); + return null; + } + } + + if (dataFileService.isThumbnailAvailable(thumbnailImageFile)) { + cardImageUrl = ImageThumbConverter.getImageThumbnailAsBase64( + thumbnailImageFile, + ImageThumbConverter.DEFAULT_CARDIMAGE_SIZE); + } + + if (cardImageUrl != null) { + this.dvobjectThumbnailsMap.put(thumbnailImageFileId, cardImageUrl); + } else { + this.dvobjectThumbnailsMap.put(thumbnailImageFileId, ""); + } + } + + //logger.info("dataset id " + result.getEntityId() + ", returning " + cardImageUrl); + + return cardImageUrl; + } + + // it's the responsibility of the user - to make sure the search result + // passed to this method is of the Dataverse type! + public String getDataverseCardImageAsBase64Url(SolrSearchResult result) { + return dataverseService.getDataverseLogoThumbnailAsBase64ById(result.getEntityId()); + } + + public void resetObjectMaps() { + dvobjectThumbnailsMap = new HashMap<>(); + dvobjectViewMap = new HashMap<>(); + } + + +} diff --git a/src/main/java/edu/harvard/iq/dataverse/dataset/DatasetUtil.java b/src/main/java/edu/harvard/iq/dataverse/dataset/DatasetUtil.java index 8e643184b76..8f86fba373e 100644 --- a/src/main/java/edu/harvard/iq/dataverse/dataset/DatasetUtil.java +++ b/src/main/java/edu/harvard/iq/dataverse/dataset/DatasetUtil.java @@ -2,6 +2,7 @@ import edu.harvard.iq.dataverse.DataFile; import edu.harvard.iq.dataverse.Dataset; +import edu.harvard.iq.dataverse.DatasetVersion; import edu.harvard.iq.dataverse.FileMetadata; import edu.harvard.iq.dataverse.dataaccess.ImageThumbConverter; import edu.harvard.iq.dataverse.util.FileUtil; @@ -56,11 +57,11 @@ public static List getThumbnailCandidates(Dataset dataset, boo } } for (FileMetadata fileMetadata : dataset.getLatestVersion().getFileMetadatas()) { - DataFile dataFile = fileMetadata.getDataFile(); - + DataFile dataFile = fileMetadata.getDataFile(); + if (dataFile != null && FileUtil.isThumbnailSupported(dataFile) && ImageThumbConverter.isThumbnailAvailable(dataFile) - && !dataFile.isRestricted()) { + && !dataFile.isRestricted()) { String imageSourceBase64 = null; imageSourceBase64 = ImageThumbConverter.getImageThumbnailAsBase64(dataFile, ImageThumbConverter.DEFAULT_CARDIMAGE_SIZE); @@ -68,16 +69,23 @@ public static List getThumbnailCandidates(Dataset dataset, boo DatasetThumbnail datasetThumbnail = new DatasetThumbnail(imageSourceBase64, dataFile); thumbnails.add(datasetThumbnail); } - } + } } return thumbnails; } - public static DatasetThumbnail getThumbnail(Dataset dataset) { + /** + * Note "datasetVersionId" can be null. If needed, it helps the "efficiency" + * of "attemptToAutomaticallySelectThumbnailFromDataFiles" + * + * @param dataset + * @param datasetVersion + * @return + */ + public static DatasetThumbnail getThumbnail(Dataset dataset, DatasetVersion datasetVersion) { if (dataset == null) { return null; } - String title = dataset.getLatestVersion().getTitle(); Path path = Paths.get(dataset.getFileSystemDirectory() + File.separator + datasetLogoThumbnail + thumb48addedByImageThumbConverter); if (Files.exists(path)) { @@ -85,7 +93,7 @@ public static DatasetThumbnail getThumbnail(Dataset dataset) { byte[] bytes = Files.readAllBytes(path); String base64image = Base64.getEncoder().encodeToString(bytes); DatasetThumbnail datasetThumbnail = new DatasetThumbnail(FileUtil.DATA_URI_SCHEME + base64image, null); - logger.fine(title + " will get thumbnail from dataset logo."); + logger.fine("will get thumbnail from dataset logo"); return datasetThumbnail; } catch (IOException ex) { logger.fine("Unable to read thumbnail image from file: " + ex); @@ -96,33 +104,40 @@ public static DatasetThumbnail getThumbnail(Dataset dataset) { if (thumbnailFile == null) { if (dataset.isUseGenericThumbnail()) { - logger.fine(title + " does not have a thumbnail and is 'Use Generic'."); + logger.fine("Dataset (id :" + dataset.getId() + ") does not have a thumbnail and is 'Use Generic'."); return null; } else { - thumbnailFile = attemptToAutomaticallySelectThumbnailFromDataFiles(dataset); + thumbnailFile = attemptToAutomaticallySelectThumbnailFromDataFiles(dataset, datasetVersion); if (thumbnailFile == null) { - logger.fine(title + " does not have a thumbnail available that could be selected automatically."); + logger.fine("Dataset (id :" + dataset.getId() + ") does not have a thumbnail available that could be selected automatically."); return null; } else { String imageSourceBase64 = ImageThumbConverter.getImageThumbnailAsBase64(thumbnailFile, ImageThumbConverter.DEFAULT_CARDIMAGE_SIZE); DatasetThumbnail defaultDatasetThumbnail = new DatasetThumbnail(imageSourceBase64, thumbnailFile); - logger.fine(title + " will get thumbnail through automatic selection from DataFile id " + thumbnailFile.getId()); + logger.fine("thumbnailFile (id :" + thumbnailFile.getId() + ") will get thumbnail through automatic selection from DataFile id " + thumbnailFile.getId()); return defaultDatasetThumbnail; } } } else if (thumbnailFile.isRestricted()) { - logger.fine(title + " has a thumbnail the user selected but the file must have later been restricted. Returning null."); + logger.fine("Dataset (id :" + dataset.getId() + ") has a thumbnail the user selected but the file must have later been restricted. Returning null."); return null; } else { String imageSourceBase64 = ImageThumbConverter.getImageThumbnailAsBase64(thumbnailFile, ImageThumbConverter.DEFAULT_CARDIMAGE_SIZE); DatasetThumbnail userSpecifiedDatasetThumbnail = new DatasetThumbnail(imageSourceBase64, thumbnailFile); - logger.fine(title + " will get thumbnail the user specified from DataFile id " + thumbnailFile.getId()); + logger.fine("Dataset (id :" + dataset.getId() + ") will get thumbnail the user specified from DataFile id " + thumbnailFile.getId()); return userSpecifiedDatasetThumbnail; } } } + public static DatasetThumbnail getThumbnail(Dataset dataset) { + if (dataset == null) { + return null; + } + return getThumbnail(dataset, null); + } + public static boolean deleteDatasetLogo(Dataset dataset) { if (dataset == null) { return false; @@ -139,15 +154,29 @@ public static boolean deleteDatasetLogo(Dataset dataset) { } } - public static DataFile attemptToAutomaticallySelectThumbnailFromDataFiles(Dataset dataset) { + /** + * Pass an optional datasetVersion in case the file system is checked + * + * @param dataset + * @param datasetVersion + * @return + */ + public static DataFile attemptToAutomaticallySelectThumbnailFromDataFiles(Dataset dataset, DatasetVersion datasetVersion) { if (dataset == null) { return null; } + if (dataset.isUseGenericThumbnail()) { logger.fine("Bypassing logic to find a thumbnail because a generic icon for the dataset is desired."); return null; } - for (FileMetadata fmd : dataset.getLatestVersion().getFileMetadatas()) { + + if (datasetVersion == null) { + logger.fine("getting latest version of dataset"); + datasetVersion = dataset.getLatestVersion(); + } + + for (FileMetadata fmd : datasetVersion.getFileMetadatas()) { DataFile testFile = fmd.getDataFile(); if (FileUtil.isThumbnailSupported(testFile) && ImageThumbConverter.isThumbnailAvailable(testFile, ImageThumbConverter.DEFAULT_CARDIMAGE_SIZE)) { return testFile; diff --git a/src/main/java/edu/harvard/iq/dataverse/engine/command/impl/CreateDatasetCommand.java b/src/main/java/edu/harvard/iq/dataverse/engine/command/impl/CreateDatasetCommand.java index 643b3cd879a..ae19fa7d50c 100644 --- a/src/main/java/edu/harvard/iq/dataverse/engine/command/impl/CreateDatasetCommand.java +++ b/src/main/java/edu/harvard/iq/dataverse/engine/command/impl/CreateDatasetCommand.java @@ -152,6 +152,8 @@ by the Dataset page (in CREATE mode), it already has the persistent theDataset.setIdentifier(ctxt.datasets().generateDatasetIdentifier(theDataset, idServiceBean)); } + logger.fine("Saving the files permanently."); + ctxt.ingest().addFiles(dsv, theDataset.getFiles()); logger.log(Level.FINE,"doiProvider={0} protocol={1} importType={2} GlobalIdCreateTime=={3}", new Object[]{doiProvider, protocol, importType, theDataset.getGlobalIdCreateTime()}); // Attempt the registration if importing dataset through the API, or the app (but not harvest or migrate) if ((importType == null || importType.equals(ImportType.NEW)) diff --git a/src/main/java/edu/harvard/iq/dataverse/engine/command/impl/DestroyDatasetCommand.java b/src/main/java/edu/harvard/iq/dataverse/engine/command/impl/DestroyDatasetCommand.java index b91ae0c96b8..fdf49929ac0 100644 --- a/src/main/java/edu/harvard/iq/dataverse/engine/command/impl/DestroyDatasetCommand.java +++ b/src/main/java/edu/harvard/iq/dataverse/engine/command/impl/DestroyDatasetCommand.java @@ -53,12 +53,17 @@ protected void executeImpl(CommandContext ctxt) throws CommandException { this, Collections.singleton(Permission.DeleteDatasetDraft), doomed); } + // If there is a dedicated thumbnail DataFile, it needs to be reset + // explicitly, or we'll get a constraint violation when deleting: + doomed.setThumbnailFile(null); final Dataset managedDoomed = ctxt.em().merge(doomed); List datasetAndFileSolrIdsToDelete = new ArrayList<>(); // files need to iterate through and remove 'by hand' to avoid - // optimistic lock issues.... + // optimistic lock issues... (plus the physical files need to be + // deleted too!) + Iterator dfIt = doomed.getFiles().iterator(); while (dfIt.hasNext()){ DataFile df = dfIt.next(); diff --git a/src/main/java/edu/harvard/iq/dataverse/search/SearchIncludeFragment.java b/src/main/java/edu/harvard/iq/dataverse/search/SearchIncludeFragment.java index c72cf2d2cb1..34042e78d1b 100644 --- a/src/main/java/edu/harvard/iq/dataverse/search/SearchIncludeFragment.java +++ b/src/main/java/edu/harvard/iq/dataverse/search/SearchIncludeFragment.java @@ -15,14 +15,22 @@ import edu.harvard.iq.dataverse.PermissionServiceBean; import edu.harvard.iq.dataverse.PermissionsWrapper; import edu.harvard.iq.dataverse.SettingsWrapper; +import edu.harvard.iq.dataverse.ThumbnailServiceWrapper; import edu.harvard.iq.dataverse.WidgetWrapper; import edu.harvard.iq.dataverse.dataaccess.ImageThumbConverter; import edu.harvard.iq.dataverse.dataset.DatasetThumbnail; import edu.harvard.iq.dataverse.dataset.DatasetUtil; import edu.harvard.iq.dataverse.engine.command.DataverseRequest; +import edu.harvard.iq.dataverse.util.FileUtil; import edu.harvard.iq.dataverse.util.SystemConfig; +import java.io.File; +import java.io.IOException; +import java.nio.file.Files; +import java.nio.file.Path; +import java.nio.file.Paths; import java.util.ArrayList; import java.util.Arrays; +import java.util.Base64; import java.util.Date; import java.util.HashMap; import java.util.HashSet; @@ -66,6 +74,8 @@ public class SearchIncludeFragment implements java.io.Serializable { @Inject PermissionsWrapper permissionsWrapper; @Inject + ThumbnailServiceWrapper thumbnailServiceWrapper; + @Inject WidgetWrapper widgetWrapper; @EJB SystemConfig systemConfig; @@ -390,7 +400,7 @@ public void search(boolean onlyDataRelatedToMe) { dataverseService.populateDvSearchCard(solrSearchResult); /* - Datasets cannot be harvested yet. + Dataverses cannot be harvested yet. if (isHarvestedDataverse(solrSearchResult.getEntityId())) { solrSearchResult.setHarvested(true); }*/ @@ -1152,13 +1162,11 @@ public String dataFileChecksumDisplay(DataFile datafile) { } public void setDisplayCardValues() { - int i = 0; - dvobjectThumbnailsMap = new HashMap<>(); - dvobjectViewMap = new HashMap<>(); + Set harvestedDatasetIds = null; for (SolrSearchResult result : searchResultsList) { //logger.info("checking DisplayImage for the search result " + i++); - if (result.getType().equals("dataverses") /*&& result.getEntity() instanceof Dataverse*/) { + if (result.getType().equals("dataverses")) { /** * @todo Someday we should probably revert this setImageUrl to * the original meaning "image_url" to address this issue: @@ -1166,17 +1174,19 @@ public void setDisplayCardValues() { * downloadable image - * https://github.com/IQSS/dataverse/issues/3616 */ - result.setImageUrl(getDataverseCardImageUrl(result)); - } else if (result.getType().equals("datasets") /*&& result.getEntity() instanceof Dataset*/) { - DatasetThumbnail datasetThumbnail = result.getDatasetThumbnail(); - if (datasetThumbnail != null) { - result.setImageUrl(datasetThumbnail.getBase64image()); + result.setImageUrl(thumbnailServiceWrapper.getDataverseCardImageAsBase64Url(result)); + } else if (result.getType().equals("datasets")) { + + result.setImageUrl(thumbnailServiceWrapper.getDatasetCardImageAsBase64Url(result)); + + if (result.isHarvested()) { + if (harvestedDatasetIds == null) { + harvestedDatasetIds = new HashSet<>(); + } + harvestedDatasetIds.add(result.getEntityId()); } - } else if (result.getType().equals("files") /*&& result.getEntity() instanceof DataFile*/) { - // TODO: - // use permissionsWrapper? -- L.A. 4.2.1 - // OK, done! (4.2.2; in the getFileCardImageUrl() method, below) - result.setImageUrl(getFileCardImageUrl(result)); + } else if (result.getType().equals("files")) { + result.setImageUrl(thumbnailServiceWrapper.getFileCardImageAsBase64Url(result)); if (result.isHarvested()) { if (harvestedDatasetIds == null) { harvestedDatasetIds = new HashSet<>(); @@ -1185,8 +1195,8 @@ public void setDisplayCardValues() { } } } - dvobjectThumbnailsMap = null; - dvobjectViewMap = null; + + thumbnailServiceWrapper.resetObjectMaps(); // Now, make another pass, and add the remote archive descriptions to the // harvested dataset and datafile cards (at the expense of one extra @@ -1251,141 +1261,7 @@ public void setDisplayCardValues() { } } - - private Map dvobjectThumbnailsMap = null; - private Map dvobjectViewMap = null; - - private String getAssignedDatasetImage(Dataset dataset) { - if (dataset == null) { - return null; - } - - DataFile assignedThumbnailFile = dataset.getThumbnailFile(); - - if (assignedThumbnailFile != null) { - Long assignedThumbnailFileId = null; - - if (this.dvobjectThumbnailsMap.containsKey(assignedThumbnailFileId)) { - // Yes, return previous answer - //logger.info("using cached result for ... "+assignedThumbnailFileId); - if (!"".equals(this.dvobjectThumbnailsMap.get(assignedThumbnailFileId))) { - return this.dvobjectThumbnailsMap.get(assignedThumbnailFileId); - } - return null; - } - - String imageSourceBase64 = ImageThumbConverter.getImageThumbnailAsBase64( - assignedThumbnailFile, - ImageThumbConverter.DEFAULT_CARDIMAGE_SIZE); - - if (imageSourceBase64 != null) { - this.dvobjectThumbnailsMap.put(assignedThumbnailFileId, imageSourceBase64); - return imageSourceBase64; - } - - // OK - we can't use this "assigned" image, because of permissions, or because - // the thumbnail failed to generate, etc... in this case we'll - // mark this dataset in the lookup map - so that we don't have to - // do all these lookups again... - this.dvobjectThumbnailsMap.put(assignedThumbnailFileId, ""); - - // TODO: (?) - // do we need to cache this datafile object in the view map? - // -- L.A., 4.2.2 - } - - return null; - - } - - // it's the responsibility of the user - to make sure the search result - // passed to this method is of the Datafile type! - private String getFileCardImageUrl(SolrSearchResult result) { - Long imageFileId = result.getEntity().getId(); - - if (imageFileId != null) { - if (this.dvobjectThumbnailsMap.containsKey(imageFileId)) { - // Yes, return previous answer - //logger.info("using cached result for ... "+datasetId); - if (!"".equals(this.dvobjectThumbnailsMap.get(imageFileId))) { - return this.dvobjectThumbnailsMap.get(imageFileId); - } - return null; - } - - String cardImageUrl = null; - - if (result.getTabularDataTags() != null) { - for (String tabularTagLabel : result.getTabularDataTags()) { - DataFileTag tag = new DataFileTag(); - try { - tag.setTypeByLabel(tabularTagLabel); - tag.setDataFile((DataFile) result.getEntity()); - ((DataFile) result.getEntity()).addTag(tag); - } catch (IllegalArgumentException iax) { - // ignore - } - } - } - - if ((!((DataFile)result.getEntity()).isRestricted() - || permissionsWrapper.hasDownloadFilePermission(result.getEntity())) - && dataFileService.isThumbnailAvailable((DataFile) result.getEntity())) { - - cardImageUrl = ImageThumbConverter.getImageThumbnailAsBase64( - (DataFile) result.getEntity(), - ImageThumbConverter.DEFAULT_CARDIMAGE_SIZE); - } - - if (cardImageUrl != null) { - this.dvobjectThumbnailsMap.put(imageFileId, cardImageUrl); - //logger.info("datafile id " + imageFileId + ", returning " + cardImageUrl); - - if (!(dvobjectViewMap.containsKey(imageFileId) - && dvobjectViewMap.get(imageFileId).isInstanceofDataFile())) { - - dvobjectViewMap.put(imageFileId, result.getEntity()); - - } - - return cardImageUrl; - } else { - this.dvobjectThumbnailsMap.put(imageFileId, ""); - } - } - return null; - } - - // it's the responsibility of the user - to make sure the search result - // passed to this method is of the Dataverse type! - private String getDataverseCardImageUrl(SolrSearchResult result) { - return dataverseService.getDataverseLogoThumbnailAsBase64ById(result.getEntityId()); - } - - /* - These commented out methods below are old optimizations that are no longer - necessary, since there is now a more direct connection between a harvested - dataset and its HarvestingClient configuration. -- L.A. 4.5 - */ - /* - private Map getHarvestedDatasetDescriptions() { - if (harvestedDatasetDescriptions != null) { - return harvestedDatasetDescriptions; - } - harvestedDatasetDescriptions = dataverseService.getAllHarvestedDataverseDescriptions(); - return harvestedDataverseDescriptions; - }*/ - /*private boolean isHarvestedDataverse(Long id) { - return this.getHarvestedDataverseDescriptions().containsKey(id); - } - - private String getHarvestingDataverseDescription(Long id) { - if (this.isHarvestedDataverse(id)) { - return this.getHarvestedDataverseDescriptions().get(id); - } - return null; - }*/ public enum SortOrder { asc, desc diff --git a/src/main/java/edu/harvard/iq/dataverse/search/SearchServiceBean.java b/src/main/java/edu/harvard/iq/dataverse/search/SearchServiceBean.java index c868ec4d24a..f0abf08a261 100644 --- a/src/main/java/edu/harvard/iq/dataverse/search/SearchServiceBean.java +++ b/src/main/java/edu/harvard/iq/dataverse/search/SearchServiceBean.java @@ -5,6 +5,8 @@ import edu.harvard.iq.dataverse.DatasetFieldConstant; import edu.harvard.iq.dataverse.DatasetFieldServiceBean; import edu.harvard.iq.dataverse.DatasetFieldType; +import edu.harvard.iq.dataverse.DatasetVersion; +import edu.harvard.iq.dataverse.DatasetVersionServiceBean; import edu.harvard.iq.dataverse.Dataverse; import edu.harvard.iq.dataverse.DataverseFacet; import edu.harvard.iq.dataverse.DvObject; @@ -69,6 +71,8 @@ public class SearchServiceBean { @EJB DvObjectServiceBean dvObjectService; @EJB + DatasetVersionServiceBean datasetVersionService; + @EJB DatasetFieldServiceBean datasetFieldService; @EJB GroupServiceBean groupService; @@ -451,7 +455,10 @@ public SolrQueryResponse search(DataverseRequest dataverseRequest, Dataverse dat if (type.equals("dataverses")) { solrSearchResult.setName(name); solrSearchResult.setHtmlUrl(baseUrl + "/dataverse/" + identifier); - solrSearchResult.setImageUrl(baseUrl + "/api/access/dvCardImage/" + entityid); + // Do not set the ImageUrl, let the search include fragment fill in + // the thumbnail, similarly to how the dataset and datafile cards + // are handled. + //solrSearchResult.setImageUrl(baseUrl + "/api/access/dvCardImage/" + entityid); /** * @todo Expose this API URL after "dvs" is changed to * "dataverses". Also, is an API token required for published @@ -463,11 +470,13 @@ public SolrQueryResponse search(DataverseRequest dataverseRequest, Dataverse dat solrSearchResult.setApiUrl(baseUrl + "/api/datasets/" + entityid); //Image url now set via thumbnail api //solrSearchResult.setImageUrl(baseUrl + "/api/access/dsCardImage/" + datasetVersionId); - DvObject dvObject = dvObjectService.findDvObject(entityid); - if (dvObject != null) { - Dataset dataset = (Dataset) dvObject; - solrSearchResult.setDatasetThumbnail(dataset.getDatasetThumbnail()); - } + // No, we don't want to set the base64 thumbnails here. + // We want to do it inside SearchIncludeFragment, AND ONLY once the rest of the + // page has already loaded. + //DatasetVersion datasetVersion = datasetVersionService.find(datasetVersionId); + //if (datasetVersion != null){ + // solrSearchResult.setDatasetThumbnail(datasetVersion.getDataset().getDatasetThumbnail(datasetVersion)); + //} /** * @todo Could use getFieldValues (plural) here. */ @@ -510,7 +519,7 @@ public SolrQueryResponse search(DataverseRequest dataverseRequest, Dataverse dat * JSON. */ // solrSearchResult.setApiUrl(baseUrl + "/api/meta/datafile/" + entityid); - solrSearchResult.setImageUrl(baseUrl + "/api/access/fileCardImage/" + entityid); + //solrSearchResult.setImageUrl(baseUrl + "/api/access/fileCardImage/" + entityid); solrSearchResult.setName(name); solrSearchResult.setFiletype(filetype); solrSearchResult.setFileContentType(fileContentType); diff --git a/src/main/java/edu/harvard/iq/dataverse/search/SolrSearchResult.java b/src/main/java/edu/harvard/iq/dataverse/search/SolrSearchResult.java index db839309205..a482ca73bc9 100644 --- a/src/main/java/edu/harvard/iq/dataverse/search/SolrSearchResult.java +++ b/src/main/java/edu/harvard/iq/dataverse/search/SolrSearchResult.java @@ -915,7 +915,7 @@ public String getDatasetUrl() { String badString = "null"; if (!identifier.contains(badString)) { if (entity != null && entity instanceof Dataset) { - if (this.isHarvested()) { + if (this.isHarvested() && ((Dataset)entity).getHarvestedFrom() != null) { String remoteArchiveUrl = ((Dataset) entity).getRemoteArchiveURL(); if (remoteArchiveUrl != null) { return remoteArchiveUrl; diff --git a/src/test/java/edu/harvard/iq/dataverse/dataset/DatasetUtilTest.java b/src/test/java/edu/harvard/iq/dataverse/dataset/DatasetUtilTest.java index 0bf70c97ccb..29c10f82c66 100644 --- a/src/test/java/edu/harvard/iq/dataverse/dataset/DatasetUtilTest.java +++ b/src/test/java/edu/harvard/iq/dataverse/dataset/DatasetUtilTest.java @@ -2,6 +2,7 @@ import edu.harvard.iq.dataverse.DataFile; import edu.harvard.iq.dataverse.Dataset; +import edu.harvard.iq.dataverse.DatasetVersion; import java.io.InputStream; import java.util.ArrayList; import java.util.List; @@ -46,18 +47,45 @@ public void testGetThumbnailCandidates() { assertEquals(expResult, result); } - /** - * Test of getThumbnail method, of class DatasetUtil. - */ @Test - public void testGetThumbnail() { - System.out.println("getThumbnail"); + public void testGetThumbnailNullDataset() { + System.out.println("testGetThumbnailNullDataset"); Dataset dataset = null; DatasetThumbnail expResult = null; DatasetThumbnail result = DatasetUtil.getThumbnail(dataset); assertEquals(expResult, result); } + @Test + public void testGetThumbnailUseGeneric() { + System.out.println("testGetThumbnailUseGeneric"); + Dataset dataset = new Dataset(); + dataset.setUseGenericThumbnail(true); + DatasetThumbnail result = DatasetUtil.getThumbnail(dataset); + assertNull(result); + } + + @Test + public void testGetThumbnailRestricted() { + System.out.println("testGetThumbnailRestricted"); + Dataset dataset = new Dataset(); + DataFile thumbnailFile = new DataFile(); + thumbnailFile.setId(42l); + thumbnailFile.setRestricted(true); + dataset.setThumbnailFile(thumbnailFile); + DatasetThumbnail result = DatasetUtil.getThumbnail(dataset); + assertNull(result); + } + + @Test + public void testGetThumbnailNullDatasetNullDatasetVersion() { + System.out.println("testGetThumbnailNullDatasetNullDatasetVersion"); + Dataset dataset = null; + DatasetVersion datasetVersion = null; + DatasetThumbnail result = DatasetUtil.getThumbnail(dataset, datasetVersion); + assertEquals(null, result); + } + /** * Test of deleteDatasetLogo method, of class DatasetUtil. */ @@ -78,7 +106,7 @@ public void testGetDefaultThumbnailFile() { System.out.println("getDefaultThumbnailFile"); Dataset dataset = null; DataFile expResult = null; - DataFile result = DatasetUtil.attemptToAutomaticallySelectThumbnailFromDataFiles(dataset); + DataFile result = DatasetUtil.attemptToAutomaticallySelectThumbnailFromDataFiles(dataset, null); assertEquals(expResult, result); }