Merge pull request #31 from IQSS/develop

Update

doc/release-notes/4.16-release-notes.md

@@ -0,0 +1,100 @@
+# Dataverse 4.16
+
+This release brings new features, enhancements, and bug fixes to Dataverse. Thank you to all of the community members who contributed code, suggestions, bug reports, and other assistance across the project.
+
+## Release Highlights
+
+### Metrics Redesign
+
+The metrics view at both the Dataset and File level has been redesigned. The main driver of this redesign has been the expanded metrics (citations and views) provided through an integration with Make Data Count, but installations that do not adopt Make Data Count will also be able to take advantage of the new metrics panel.
+
+### HTML Codebook Export
+
+Users will now be able to download HTML Codebooks as an additional Dataset Export type. This codebook is a more human-readable version of the DDI Codebook 2.5 metadata export and provides valuable information about the contents and structure of a dataset and will increase reusability of the datasets in Dataverse.
+
+### Harvesting Improvements
+
+The Harvesting code will now better handle problematic records during incremental harvests. Fixing this will mean not only fewer manual interventions by installation administrators to keep harvesting running, but it will also mean users can more easily find and access data that is important to their research.
+
+## Major Use Cases
+
+Newly-supported use cases in this release include:
+
+- As a user, I can view the works that have cited a dataset.
+- As a user, I can view the downloads and views for a dataset, based on the Make Data Count standard.
+- As a user, I can export an HTML codebook for a dataset.
+- As a user, I can expect harvested datasets to be made available more regularly.
+- As a user, I'll encounter fewer locks as I go through the publishing process.
+- As an installation administrator, I no longer need to destroy a PID in another system after destroying a dataset in Dataverse.
+
+## Notes for Dataverse Installation Administrators
+
+### Run ReExportall
+
+We made changes to the citation block in this release that will require installations to run ReExportall as part of the upgrade process. We've included this in the detailed instructions below.
+
+### Custom Analytics Code Changes
+
+You should update your custom analytics code to include CDATA sections, inside the `script` tags, around the javascript code. We have updated the documentation and sample analytics code snippet provided in [Installation Guide > Configuration > Web Analytics Code](http://guides.dataverse.org/en/latest/installation/config.html#web-analytics-code) to fix a bug that broke the rendering of the 403 and 500 custom error pgs (#5967).
+
+### Destroy Updates
+
+Destroying Datasets in Dataverse will now unregister/delete the PID with the PID provider. This eliminates the need for an extra step to "clean up" a PID registration after destroying a Dataset.
+
+### Deleting Notifications
+
+In making the fix for #5687 we discovered that notifications created prior to 2018 may have been invalidated. With this release we advise that these older notifications are deleted from the database. The following query can be used for this purpose:
+
+`delete from usernotification where date_part('year', senddate) < 2018;`
+
+### Lock Improvements
+
+In 4.15 a new lock was added to prevent parallel edits. After seeing that the lock was not being released as expected, which required administrator intervention, we've adjusted this code to release the lock as expected.
+
+### New Database Settings
+
+:AllowCors - Allows Cross-Origin Resource sharing(CORS). By default this setting is absent and Dataverse assumes it to be true.
+
+## Notes for Tool Developers and Integrators
+
+### OpenAIRE Export Changes
+
+The OpenAIRE metadata export now correctly expresses information about a dataset's Production Place and GeoSpatial Bounding Box. When users add metadata to Dataverse's Production Place and GeoSpatial Bounding Box fields, those fields are now mapped to separate DataCite geoLocation properties.
+
+Metadata about the software name and version used to create a dataset, Software Name and Software Version, are re-mapped from DataCite's more general descriptionType="Methods" property to descriptionType="TechnicalInfo", which was added in a recent version of the DataCite schema in order to improve discoverability of metadata about the software used to create datasets.
+
+## Complete List of Changes
+
+For the complete list of code changes in this release, see the <a href="https://github.com/IQSS/dataverse/milestone/83?closed=1">4.16 milestone</a> in Github.
+
+For help with upgrading, installing, or general questions please post to the <a href="https://groups.google.com/forum/#!forum/dataverse-community">Dataverse Google Group</a> or email support@dataverse.org.
+
+## Installation
+
+If this is a new installation, please see our <a href=http://guides.dataverse.org/en/4.16/installation/>Installation Guide</a>.
+
+## Upgrade
+
+1. Undeploy the previous version.
+
+- &lt;glassfish install path&gt;/glassfish4/bin/asadmin list-applications
+- &lt;glassfish install path&gt;/glassfish4/bin/asadmin undeploy dataverse
+
+2. Stop glassfish and remove the generated directory, start
+
+- service glassfish stop
+- remove the generated directory: rm -rf &lt;glassfish install path&gt;glassfish4/glassfish/domains/domain1/generated
+- service glassfish start
+
+3. Deploy this version.
+
+- &lt;glassfish install path&gt;/glassfish4/bin/asadmin deploy &lt;path&gt;dataverse-4.16.war
+
+4. Restart glassfish  
+5. Update Citation Metadata Block    
+
+   `curl http://localhost:8080/api/admin/datasetfield/load -X POST --data-binary @citation.tsv -H "Content-type: text/tab-separated-values"`  
+
+6. Run ReExportall to update the citations  
+
+   <http://guides.dataverse.org/en/4.16/admin/metadataexport.html?highlight=export#batch-exports-through-the-api>

doc/release-notes/6035-configurable-user-sessions-timeout.md

@@ -0,0 +1,8 @@
+Idle session timeout for logged-in users has been made configurable in this release. 
+The default is now set to 8 hours (this is a change from the previous default value of 24 hours). 
+If you want to change it, set the setting :LoginSessionTimeout to the new value *in minutes*. 
+For example, to reduce the timeout to 4 hours:                                         
+
+ curl -X PUT -d 240 http://localhost:8080/api/admin/settings/:LoginSessionTimeout
+
+Once again, this is the session timeout for *logged-in* users only. For the anonymous sessions the sessions are set to time out after the default ``session-timeout`` value (also in minutes) in the web.xml of the Dataverse application, which is set to 10 minutes. You will most likely not ever need to change this, but if you do, configure it by editing the web.xml file. 

doc/sphinx-guides/source/_static/api/define-metadatablocks.json

@@ -0,0 +1,8 @@
+[
+  "citation",
+  "geospatial",
+  "socialscience",
+  "astrophysics",
+  "biomedical",
+  "journal"
+]

doc/sphinx-guides/source/_static/docsdataverse_org.css

@@ -114,4 +114,8 @@ div.form-group .glyphicon.glyphicon-asterisk {font-size: .5em; vertical-align: t
 
 /* #sidebar.bs-sidenav.affix {
 	position: static;
-} REMOVED STATIC ToC */
+} REMOVED STATIC ToC */
+
+pre {
+    white-space: pre-wrap;
+}

doc/sphinx-guides/source/_static/installation/files/etc/maintenance/maintenance.xhtml

@@ -66,7 +66,7 @@
             <nav id="navbarFixed" class="navbar navbar-default navbar-fixed-top" role="navigation">
                 <div class="container">
                     <div class="navbar-header">
-                        <span class="navbar-brand"><i id="icon-dataverse" class="icon-dataverse"></i> Dataverse</span>
+                        <span class="navbar-brand"><span id="icon-dataverse" class="icon-dataverse"></span> Dataverse</span>
                     </div>
                     <div class="collapse navbar-collapse" id="topNavBar">
                         <ul class="nav navbar-nav navbar-right">

doc/sphinx-guides/source/admin/troubleshooting.rst

@@ -3,11 +3,35 @@
 Troubleshooting
 ===============
 
-This new (as of v.4.6) section of the Admin guide is for tips on how to diagnose and fix system problems. 
+Sometimes Dataverse users get into trouble. Sometimes Dataverse itself gets into trouble. If something has gone wrong, this section is for you.
 
 .. contents:: Contents:
 	:local:
 
+Using Dataverse APIs to Troubleshoot and Fix Problems
+-----------------------------------------------------
+
+See the :doc:`/api/intro` section of the API Guide for a high level overview of Dataverse APIs. Below are listed problems that support teams might encounter that can be handled via API (sometimes only via API).
+
+A Dataset Is Locked And Cannot Be Edited or Published
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+It's normal for the ingest process described in the :doc:`/user/tabulardataingest/ingestprocess` section of the User Guide to take some time but if hours or days have passed and the dataset is still locked, you might want to inspect the locks and consider deleting some or all of them.
+
+See :doc:`dataverses-datasets`.
+
+Someone Created Spam Datasets and I Need to Delete Them
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Depending on how open your installation of Dataverse is to the general public creating datasets, you may sometimes need to deal with spam datasets.
+
+Look for "destroy" in the :doc:`/api/native-api` section of the API Guide.
+
+A User Needs Their Account to Be Converted From Institutional (Shibboleth), ORCID, Google, or GitHub to Something Else
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+See :ref:`converting-shibboleth-users-to-local` and :ref:`converting-oauth-users-to-local`.
+
 Glassfish
 ---------
 

doc/sphinx-guides/source/api/apps.rst

@@ -1,40 +1,69 @@
 Apps
 ====
 
-The introduction of Dataverse APIs has fostered the development of apps that are listed at http://dataverse.org/integrations and the :doc:`/admin/integrations` section of the Admin Guide.
+The introduction of Dataverse APIs has fostered the development of a variety of software applications that are listed in the :doc:`/admin/integrations` and :doc:`/admin/reporting-tools` sections of the Admin Guide and the :doc:`/installation/external-tools` section of the Installation Guide.
 
-The apps below are open source, demonstrating how to use Dataverse APIs. Some of these apps (and others) are built on :doc:`/api/client-libraries` that are available for Dataverse APIs.
+The apps below are open source and demonstrate how to use Dataverse APIs. Some of these apps are built on :doc:`/api/client-libraries` that are available for Dataverse APIs in Python, R, and Java.
 
 .. contents:: |toctitle|
 	:local:
 
 Javascript
 ----------
 
+Data Explorer
+~~~~~~~~~~~~~
+
+Data Explorer is a GUI which lists the variables in a tabular data file allowing searching, charting and cross tabulation analysis.
+
+https://github.com/scholarsportal/Dataverse-Data-Explorer
+
+Data Curation Tool
+~~~~~~~~~~~~~~~~~~
+
+Data Curation Tool is  a GUI for curating data by adding labels, groups, weights and other details to assist with informed reuse.
+
+https://github.com/scholarsportal/Dataverse-Data-Curation-Tool
+
+File Previewers
+~~~~~~~~~~~~~~~
+
+File Previewers are tools that display the content of files - including audio, html, Hypothes.is annotations, images, PDF, text, video - allowing them to be viewed without downloading.
+
+https://github.com/QualitativeDataRepository/dataverse-previewers
+
 TwoRavens
 ~~~~~~~~~
 
 TwoRavens is a system of interlocking statistical tools for data exploration, analysis, and meta-analysis.
 
 https://github.com/IQSS/TwoRavens
 
-PHP
----
+Python
+------
 
-OJS
-~~~
+Please note that there are multiple Python modules for Dataverse APIs listed in the :doc:`client-libraries` section.
 
-The Open Journal Systems (OJS) Dataverse Plugin adds data sharing and preservation to the OJS publication process.
+dataverse-sample-data
+~~~~~~~~~~~~~~~~~~~~~
 
-https://github.com/pkp/ojs/tree/ojs-stable-2_4_8/plugins/generic/dataverse
+dataverse-sample-data allows you to populate your Dataverse installation with sample data. It makes uses of pyDataverse, which is listed in the :doc:`client-libraries` section.
 
-Python
-------
+https://github.com/IQSS/dataverse-sample-data
+
+Texas Digital Library dataverse-reports
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Dataverse Reports for Texas Digital Library generates and emails statistical reports for an installation of Dataverse using the native API and database queries.
+
+https://github.com/TexasDigitalLibrary/dataverse-reports
 
 OSF
 ~~~
 
-Allows you to view, download, and upload files to and from a Dataverse dataset from an Open Science Framework (OSF) project: https://github.com/CenterForOpenScience/osf.io/tree/develop/addons/dataverse
+OSF allows you to view, download, and upload files to and from a Dataverse dataset from an Open Science Framework (OSF) project.
+
+https://github.com/CenterForOpenScience/osf.io/tree/develop/addons/dataverse
 
 GeoConnect
 ~~~~~~~~~~
@@ -46,22 +75,49 @@ https://github.com/IQSS/geoconnect
 dataverse-metrics
 ~~~~~~~~~~~~~~~~~
 
-dataverse-metrics aggregates and visualizes metrics across multiple Dataverse installations but can also be used with a single installation: https://github.com/IQSS/dataverse-metrics
+dataverse-metrics aggregates and visualizes metrics across multiple Dataverse installations but can also be used with a single installation
+
+https://github.com/IQSS/dataverse-metrics
+
+Whole Tale
+~~~~~~~~~~
+
+Whole Tale enables researchers to analyze data using popular tools including Jupyter and RStudio with the ultimate goal of supporting publishing of reproducible research packages.
+
+https://github.com/whole-tale/girder_wholetale/tree/v0.7/server/lib/dataverse
+
+Archivematica
+~~~~~~~~~~~~~
+
+Archivematica is an integrated suite of open-source tools for processing digital objects for long-term preservation.
+
+https://github.com/artefactual/archivematica/tree/v1.9.2/src/MCPClient/lib/clientScripts
 
 Java
 ----
 
+Please note that there is a Java library for Dataverse APIs listed in the :doc:`client-libraries` section.
+
 DVUploader
-~~~~~~~~~~~~~~~~~~~~~
+~~~~~~~~~~
 
 The open-source DVUploader tool is a stand-alone command-line Java application that uses the Dataverse API to upload files to a specified Dataset. Files can be specified by name, or the DVUploader can upload all files in a directory or recursively from a directory tree. The DVUploader can also verify that uploaded files match their local sources by comparing the local and remote fixity checksums. Source code, release 1.0.0- jar file, and documentation are available on GitHub. DVUploader's creation was supported by the Texas Digital Library.
 
 https://github.com/IQSS/dataverse-uploader
 
-
 Dataverse for Android
 ~~~~~~~~~~~~~~~~~~~~~
 
-For now this is only a proof of concept.
+Dataverse for Android makes use of Dataverse's Search API.
 
 https://github.com/IQSS/dataverse-android
+
+PHP
+---
+
+OJS
+~~~
+
+The Open Journal Systems (OJS) Dataverse Plugin adds data sharing and preservation to the OJS publication process.
+
+https://github.com/pkp/ojs/tree/ojs-stable-2_4_8/plugins/generic/dataverse