OpenAPI definition endpoint (#10328)

* Plugin initial config * Initial changes to provide OpenAPI definition * Added integration test * Imports fix * Add patchnotes * Update the changelog * Update src/main/java/edu/harvard/iq/dataverse/api/Info.java Co-authored-by: Philip Durbin <philip_durbin@harvard.edu> * Update doc/release-notes/10236-openapi-definition-endpoint.md Co-authored-by: Philip Durbin <philip_durbin@harvard.edu> * Update doc/release-notes/10236-openapi-definition-endpoint.md Co-authored-by: Philip Durbin <philip_durbin@harvard.edu> * Add native API docs * Remove generated definitions * Add to gitignore generated openapi files * Updates to docs * Ignore files correction * Remove files created by the plugin * Changes to move the definition files to META-INF * Changes to move the definitions to WEB-INF * Changes to get the files from META-INF * Changed the phase of execution of the smallrye plugin * Changes of names to improve the generation of the spec * Add support for OpenAPI annotations and documents the version endpoint * Multipart Annotations * Typos correction * Changes for tags * Renaming of methods * Changes to the endpoint * Added test * Add test * Deleted extra import * Docs updated * openapi doc tweaks #9981 #10236 * improve release note #9981 #10236 * Remove old test and changes response to JSON * stub out guidance on openapi validation #9981 #10236 * add InfoIT to list of tests * use description of Dataverse from website * mention status codes in openapi doc * update api faq about changelog, link to breaking changes doc * typo * Change to OpenApi * Changes to docs * Name fix * Removing the multipart from unirest --------- Co-authored-by: Philip Durbin <philip_durbin@harvard.edu>
IQSS · Jun 6, 2024 · c052773 · c052773
1 parent 3c55c3f
commit c052773
Show file tree

Hide file tree

Showing 23 changed files with 372 additions and 21 deletions.
diff --git a/.gitignore b/.gitignore
@@ -34,6 +34,7 @@ oauth-credentials.md
 /src/main/webapp/oauth2/newAccount.html
 scripts/api/setup-all.sh*
 scripts/api/setup-all.*.log
+src/main/resources/edu/harvard/iq/dataverse/openapi/
 
 # ctags generated tag file
 tags

diff --git a/doc/release-notes/10236-openapi-definition-endpoint.md b/doc/release-notes/10236-openapi-definition-endpoint.md
@@ -0,0 +1,8 @@
+In Dataverse 6.0 Payara was updated, which caused the url `/openapi` to stop working:
+
+- https://github.com/IQSS/dataverse/issues/9981
+- https://github.com/payara/Payara/issues/6369
+
+When it worked in Dataverse 5.x, the `/openapi` output was generated automatically by Payara, but in this release we have switched to OpenAPI output produced by the [SmallRye OpenAPI plugin](https://github.com/smallrye/smallrye-open-api/tree/main/tools/maven-plugin). This gives us finer control over the output.
+
+For more information, see the section on  [OpenAPI](https://dataverse-guide--10328.org.readthedocs.build/en/10328/api/getting-started.html#openapi) in the API Guide.
diff --git a/doc/sphinx-guides/source/api/changelog.rst b/doc/sphinx-guides/source/api/changelog.rst
@@ -30,6 +30,7 @@ v6.0
 ----
 
 - **/api/access/datafile**: When a null or invalid API token is provided to download a public (non-restricted) file with this API call, it will result on a ``401`` error response. Previously, the download was allowed (``200`` response). Please note that we noticed this change sometime between 5.9 and 6.0. If you can help us pinpoint the exact version (or commit!), please get in touch. See :doc:`dataaccess`.
+- **/openapi**: This endpoint is currently broken. See https://github.com/IQSS/dataverse/issues/9981
 
 v5.6
 ----

diff --git a/doc/sphinx-guides/source/api/faq.rst b/doc/sphinx-guides/source/api/faq.rst
@@ -56,12 +56,12 @@ Where is the Comprehensive List of All API Functionality?
 
 There are so many Dataverse Software APIs that a single page in this guide would probably be overwhelming. See :ref:`list-of-dataverse-apis` for links to various pages.
 
-It is possible to get a complete list of API functionality in Swagger/OpenAPI format if you deploy Dataverse Software 5.x. For details, see https://github.com/IQSS/dataverse/issues/5794
+It is possible to get a complete list of API functionality in Swagger/OpenAPI format. See :ref:`openapi`.
 
 Is There a Changelog of API Functionality That Has Been Added Over Time?
 ------------------------------------------------------------------------
 
-No, but there probably should be. If you have suggestions for how it should look, please create an issue at https://github.com/IQSS/dataverse/issues
+Changes to the API that don't break anything can be found in the `release notes <https://github.com/IQSS/dataverse/releases>`_ of each release. Breaking changes are documented in :doc:`changelog`.
 
 .. _no-api:
 
@@ -89,6 +89,8 @@ Why Are the Return Values (HTTP Status Codes) Not Documented?
 
 They should be. Please consider making a pull request to help. The :doc:`/developers/documentation` section of the Developer Guide should help you get started. :ref:`create-dataverse-api` has an example you can follow or you can come up with a better way.
 
+Also, please note that we are starting to experiment with putting response codes in our OpenAPI document. See :ref:`openapi`.
+
 What If My Question Is Not Answered Here?
 -----------------------------------------
 

diff --git a/doc/sphinx-guides/source/api/getting-started.rst b/doc/sphinx-guides/source/api/getting-started.rst
@@ -154,6 +154,32 @@ Listing Permissions (Role Assignments)
 
 See :ref:`list-role-assignments-on-a-dataverse-api`.
 
+.. _openapi:
+
+Getting the OpenAPI Document
+----------------------------
+
+You can access our `OpenAPI document`_ using the ``/openapi`` endpoint. The default format is YAML if no parameter is provided, but you can also obtain the JSON version by either passing ``format=json`` as a query parameter or by sending ``Accept:application/json`` (case-sensitive) as a header.
+
+.. _OpenAPI document: https://spec.openapis.org/oas/latest.html#openapi-document
+
+.. note:: See :ref:`curl-examples-and-environment-variables` if you are unfamiliar with the use of export below.
+
+.. code-block:: bash
+
+  export SERVER_URL=https://demo.dataverse.org
+  export FORMAT=json
+
+  curl "$SERVER_URL/openapi?format=$FORMAT"
+
+The fully expanded example above (without environment variables) looks like this:
+
+.. code-block:: bash
+
+  curl "https://demo.dataverse.org/openapi?format=json"
+
+We are aware that our OpenAPI document is not perfect. You can find more information about validating the document under :ref:`openapi-dev` in the Developer Guide.
+
 Beyond "Getting Started" Tasks
 ------------------------------
 

diff --git a/doc/sphinx-guides/source/developers/api-design.rst b/doc/sphinx-guides/source/developers/api-design.rst
@@ -7,6 +7,21 @@ API design is a large topic. We expect this page to grow over time.
 .. contents:: |toctitle|
 	:local:
 
+.. _openapi-dev:
+
+OpenAPI
+-------
+
+As you add API endpoints, please be conscious that we are exposing these endpoints as an OpenAPI document at ``/openapi`` (e.g. http://localhost:8080/openapi ). See :ref:`openapi` in the API Guide for the user-facing documentation.
+
+We've played around with validation tools such as https://quobix.com/vacuum/ and https://pb33f.io/doctor/ only to discover that our OpenAPI output is less than ideal, generating various warnings and errors.
+
+You can prevent additional problems in our OpenAPI document by observing the following practices:
+
+- When creating a method name within an API class, make it unique.
+
+If you are looking for a reference about the annotations used to generate the OpenAPI document, you can find it in the `MicroProfile OpenAPI Specification <https://download.eclipse.org/microprofile/microprofile-open-api-3.1/microprofile-openapi-spec-3.1.html#_detailed_usage_of_key_annotations>`_.
+
 Paths
 -----
 

diff --git a/pom.xml b/pom.xml
@@ -32,6 +32,12 @@
         <poi.version>5.2.1</poi.version>
         <tika.version>2.4.1</tika.version>
         <netcdf.version>5.5.3</netcdf.version>
+
+        <openapi.infoTitle>Dataverse API</openapi.infoTitle>
+        <openapi.infoVersion>${project.version}</openapi.infoVersion>
+        <openapi.infoDescription>Open source research data repository software.</openapi.infoDescription>
+        <!-- https://download.eclipse.org/microprofile/microprofile-open-api-3.1.1/microprofile-openapi-spec-3.1.1.html#_location_and_formats -->
+        <openapi.outputDirectory>${project.build.outputDirectory}/META-INF</openapi.outputDirectory>
     </properties>
 
     <!-- Versions of dependencies used both directly and transitive are managed here.
@@ -195,6 +201,11 @@
             <artifactId>microprofile-metrics-api</artifactId>
             <scope>provided</scope>
         </dependency>
+        <dependency>
+            <groupId>org.eclipse.microprofile.openapi</groupId>
+            <artifactId>microprofile-openapi-api</artifactId>
+            <scope>provided</scope>
+        </dependency>
         <dependency>
             <groupId>jakarta.platform</groupId>
             <artifactId>jakarta.jakartaee-api</artifactId>
@@ -930,6 +941,30 @@
                     <consoleOutput>true</consoleOutput>
                 </configuration>
             </plugin>
+            <plugin>
+                <groupId>io.smallrye</groupId>
+                <artifactId>smallrye-open-api-maven-plugin</artifactId>
+                <version>3.10.0</version>
+                <executions>
+                    <execution>
+                        <goals>
+                            <goal>generate-schema</goal>
+                        </goals>
+                        <!-- Plugin scans class files, not sources. Execute after compile phase but before package -->
+                        <phase>process-classes</phase>
+                        <configuration>
+                            <outputDirectory>${openapi.outputDirectory}</outputDirectory>
+                            <schemaFilename>openapi</schemaFilename>
+                            <infoTitle>${openapi.infoTitle}</infoTitle>
+                            <infoVersion>${openapi.infoVersion}</infoVersion>
+                            <infoDescription>${openapi.infoDescription}</infoDescription>
+                            <operationIdStrategy>CLASS_METHOD</operationIdStrategy>
+                            <scanPackages>edu.harvard.iq.dataverse</scanPackages>
+                            <scanDependenciesDisable>true</scanDependenciesDisable>
+                        </configuration>
+                    </execution>
+                </executions>
+            </plugin>
         </plugins>
     </build>
     <profiles>
@@ -1087,4 +1122,4 @@
             </build>
         </profile>
     </profiles>
-</project>
+</project>
diff --git a/src/main/java/edu/harvard/iq/dataverse/api/Access.java b/src/main/java/edu/harvard/iq/dataverse/api/Access.java
@@ -130,6 +130,14 @@
 import jakarta.ws.rs.core.MediaType;
 import static jakarta.ws.rs.core.Response.Status.FORBIDDEN;
 import static jakarta.ws.rs.core.Response.Status.UNAUTHORIZED;
+
+import org.eclipse.microprofile.openapi.annotations.Operation;
+import org.eclipse.microprofile.openapi.annotations.media.Content;
+import org.eclipse.microprofile.openapi.annotations.media.Schema;
+import org.eclipse.microprofile.openapi.annotations.parameters.RequestBody;
+import org.eclipse.microprofile.openapi.annotations.responses.APIResponse;
+import org.eclipse.microprofile.openapi.annotations.responses.APIResponses;
+import org.eclipse.microprofile.openapi.annotations.tags.Tag;
 import org.glassfish.jersey.media.multipart.FormDataBodyPart;
 import org.glassfish.jersey.media.multipart.FormDataParam;
 
@@ -1248,6 +1256,20 @@ private String getWebappImageResource(String imageName) {
     @AuthRequired
     @Path("datafile/{fileId}/auxiliary/{formatTag}/{formatVersion}")
     @Consumes(MediaType.MULTIPART_FORM_DATA)
+    @Produces("application/json")
+    @Operation(summary = "Save auxiliary file with version", 
+               description = "Saves an auxiliary file")
+    @APIResponses(value = {
+        @APIResponse(responseCode = "200",
+                    description = "File saved response"),
+        @APIResponse(responseCode = "403",
+                    description = "User not authorized to edit the dataset."),
+        @APIResponse(responseCode = "400",
+                    description = "File not found based on id.")
+    })
+    @Tag(name = "saveAuxiliaryFileWithVersion", 
+         description = "Save Auxiliary File With Version")
+    @RequestBody(content = @Content(mediaType = MediaType.MULTIPART_FORM_DATA))
     public Response saveAuxiliaryFileWithVersion(@Context ContainerRequestContext crc,
                                                  @PathParam("fileId") Long fileId,
                                                  @PathParam("formatTag") String formatTag,

diff --git a/src/main/java/edu/harvard/iq/dataverse/api/Admin.java b/src/main/java/edu/harvard/iq/dataverse/api/Admin.java
@@ -201,7 +201,7 @@ public Response putSetting(@PathParam("name") String name, String content) {
 
 	@Path("settings/{name}/lang/{lang}")
 	@PUT
-	public Response putSetting(@PathParam("name") String name, @PathParam("lang") String lang, String content) {
+	public Response putSettingLang(@PathParam("name") String name, @PathParam("lang") String lang, String content) {
 		Setting s = settingsSvc.set(name, lang, content);
 		return ok("Setting " + name + " - " + lang + " - added.");
 	}
@@ -224,7 +224,7 @@ public Response deleteSetting(@PathParam("name") String name) {
 
 	@Path("settings/{name}/lang/{lang}")
 	@DELETE
-	public Response deleteSetting(@PathParam("name") String name, @PathParam("lang") String lang) {
+	public Response deleteSettingLang(@PathParam("name") String name, @PathParam("lang") String lang) {
 		settingsSvc.delete(name, lang);
 		return ok("Setting " + name + " - " + lang + " deleted.");
 	}

diff --git a/src/main/java/edu/harvard/iq/dataverse/api/BuiltinUsers.java b/src/main/java/edu/harvard/iq/dataverse/api/BuiltinUsers.java
@@ -119,7 +119,7 @@ public Response create(BuiltinUser user, @PathParam("password") String password,
      */
     @POST
     @Path("{password}/{key}/{sendEmailNotification}")
-    public Response create(BuiltinUser user, @PathParam("password") String password, @PathParam("key") String key, @PathParam("sendEmailNotification") Boolean sendEmailNotification) {
+    public Response createWithNotification(BuiltinUser user, @PathParam("password") String password, @PathParam("key") String key, @PathParam("sendEmailNotification") Boolean sendEmailNotification) {
         return internalSave(user, password, key, sendEmailNotification);
     }
 

diff --git a/src/main/java/edu/harvard/iq/dataverse/api/Datasets.java b/src/main/java/edu/harvard/iq/dataverse/api/Datasets.java
@@ -1,6 +1,7 @@
 package edu.harvard.iq.dataverse.api;
 
 import com.amazonaws.services.s3.model.PartETag;
+
 import edu.harvard.iq.dataverse.*;
 import edu.harvard.iq.dataverse.DatasetLock.Reason;
 import edu.harvard.iq.dataverse.actionlogging.ActionLogRecord;
@@ -66,6 +67,12 @@
 import jakarta.ws.rs.core.*;
 import jakarta.ws.rs.core.Response.Status;
 import org.apache.commons.lang3.StringUtils;
+import org.eclipse.microprofile.openapi.annotations.Operation;
+import org.eclipse.microprofile.openapi.annotations.media.Content;
+import org.eclipse.microprofile.openapi.annotations.media.Schema;
+import org.eclipse.microprofile.openapi.annotations.parameters.RequestBody;
+import org.eclipse.microprofile.openapi.annotations.responses.APIResponse;
+import org.eclipse.microprofile.openapi.annotations.tags.Tag;
 import org.glassfish.jersey.media.multipart.FormDataBodyPart;
 import org.glassfish.jersey.media.multipart.FormDataContentDisposition;
 import org.glassfish.jersey.media.multipart.FormDataParam;
@@ -796,7 +803,7 @@ public Response getVersionJsonLDMetadata(@Context ContainerRequestContext crc, @
     @AuthRequired
     @Path("{id}/metadata")
     @Produces("application/ld+json, application/json-ld")
-    public Response getVersionJsonLDMetadata(@Context ContainerRequestContext crc, @PathParam("id") String id, @Context UriInfo uriInfo, @Context HttpHeaders headers) {
+    public Response getJsonLDMetadata(@Context ContainerRequestContext crc, @PathParam("id") String id, @Context UriInfo uriInfo, @Context HttpHeaders headers) {
         return getVersionJsonLDMetadata(crc, id, DS_VERSION_LATEST, uriInfo, headers);
     }
 
@@ -2261,6 +2268,14 @@ public Response setDataFileAsThumbnail(@Context ContainerRequestContext crc, @Pa
     @AuthRequired
     @Path("{id}/thumbnail")
     @Consumes(MediaType.MULTIPART_FORM_DATA)
+    @Produces("application/json")
+    @Operation(summary = "Uploads a logo for a dataset", 
+               description = "Uploads a logo for a dataset")
+    @APIResponse(responseCode = "200",
+               description = "Dataset logo uploaded successfully")
+    @Tag(name = "uploadDatasetLogo", 
+         description = "Uploads a logo for a dataset")
+    @RequestBody(content = @Content(mediaType = MediaType.MULTIPART_FORM_DATA))          
     public Response uploadDatasetLogo(@Context ContainerRequestContext crc, @PathParam("id") String idSupplied, @FormDataParam("file") InputStream inputStream) {
         try {
             DatasetThumbnail datasetThumbnail = execCommand(new UpdateDatasetThumbnailCommand(createDataverseRequest(getRequestUser(crc)), findDatasetOrDie(idSupplied), UpdateDatasetThumbnailCommand.UserIntent.setNonDatasetFileAsThumbnail, null, inputStream));
@@ -2733,6 +2748,14 @@ public Response completeMPUpload(@Context ContainerRequestContext crc, String pa
     @AuthRequired
     @Path("{id}/add")
     @Consumes(MediaType.MULTIPART_FORM_DATA)
+    @Produces("application/json")
+    @Operation(summary = "Uploads a file for a dataset", 
+               description = "Uploads a file for a dataset")
+    @APIResponse(responseCode = "200",
+               description = "File uploaded successfully to dataset")
+    @Tag(name = "addFileToDataset", 
+         description = "Uploads a file for a dataset")
+    @RequestBody(content = @Content(mediaType = MediaType.MULTIPART_FORM_DATA))  
     public Response addFileToDataset(@Context ContainerRequestContext crc,
                     @PathParam("id") String idSupplied,
                     @FormDataParam("jsonData") String jsonData,
@@ -3958,6 +3981,14 @@ public Response requestGlobusUpload(@Context ContainerRequestContext crc, @PathP
     @AuthRequired
     @Path("{id}/addGlobusFiles")
     @Consumes(MediaType.MULTIPART_FORM_DATA)
+    @Produces("application/json")
+    @Operation(summary = "Uploads a Globus file for a dataset", 
+               description = "Uploads a Globus file for a dataset")
+    @APIResponse(responseCode = "200",
+               description = "Globus file uploaded successfully to dataset")
+    @Tag(name = "addGlobusFilesToDataset", 
+         description = "Uploads a Globus file for a dataset")
+    @RequestBody(content = @Content(mediaType = MediaType.MULTIPART_FORM_DATA))  
     public Response addGlobusFilesToDataset(@Context ContainerRequestContext crc,
                                             @PathParam("id") String datasetId,
                                             @FormDataParam("jsonData") String jsonData,
@@ -4340,6 +4371,14 @@ public Response monitorGlobusDownload(@Context ContainerRequestContext crc, @Pat
     @AuthRequired
     @Path("{id}/addFiles")
     @Consumes(MediaType.MULTIPART_FORM_DATA)
+    @Produces("application/json")
+    @Operation(summary = "Uploads a set of files to a dataset", 
+               description = "Uploads a set of files to a dataset")
+    @APIResponse(responseCode = "200",
+               description = "Files uploaded successfully to dataset")
+    @Tag(name = "addFilesToDataset", 
+         description = "Uploads a set of files to a dataset")
+    @RequestBody(content = @Content(mediaType = MediaType.MULTIPART_FORM_DATA))  
     public Response addFilesToDataset(@Context ContainerRequestContext crc, @PathParam("id") String idSupplied,
             @FormDataParam("jsonData") String jsonData) {
 
@@ -4407,6 +4446,14 @@ public Response addFilesToDataset(@Context ContainerRequestContext crc, @PathPar
     @AuthRequired
     @Path("{id}/replaceFiles")
     @Consumes(MediaType.MULTIPART_FORM_DATA)
+    @Produces("application/json")
+    @Operation(summary = "Replace a set of files to a dataset", 
+               description = "Replace a set of files to a dataset")
+    @APIResponse(responseCode = "200",
+               description = "Files replaced successfully to dataset")
+    @Tag(name = "replaceFilesInDataset", 
+         description = "Replace a set of files to a dataset")
+    @RequestBody(content = @Content(mediaType = MediaType.MULTIPART_FORM_DATA)) 
     public Response replaceFilesInDataset(@Context ContainerRequestContext crc,
                                           @PathParam("id") String idSupplied,
                                           @FormDataParam("jsonData") String jsonData) {

diff --git a/src/main/java/edu/harvard/iq/dataverse/api/Files.java b/src/main/java/edu/harvard/iq/dataverse/api/Files.java
@@ -64,6 +64,13 @@
 import static jakarta.ws.rs.core.Response.Status.FORBIDDEN;
 
 import jakarta.ws.rs.core.UriInfo;
+
+import org.eclipse.microprofile.openapi.annotations.Operation;
+import org.eclipse.microprofile.openapi.annotations.media.Content;
+import org.eclipse.microprofile.openapi.annotations.media.Schema;
+import org.eclipse.microprofile.openapi.annotations.parameters.RequestBody;
+import org.eclipse.microprofile.openapi.annotations.responses.APIResponse;
+import org.eclipse.microprofile.openapi.annotations.tags.Tag;
 import org.glassfish.jersey.media.multipart.FormDataBodyPart;
 import org.glassfish.jersey.media.multipart.FormDataContentDisposition;
 import org.glassfish.jersey.media.multipart.FormDataParam;
@@ -176,6 +183,14 @@ public Response restrictFileInDataset(@Context ContainerRequestContext crc, @Pat
     @AuthRequired
     @Path("{id}/replace")
     @Consumes(MediaType.MULTIPART_FORM_DATA)
+    @Produces("application/json")
+    @Operation(summary = "Replace a file on a dataset", 
+               description = "Replace a file to a dataset")
+    @APIResponse(responseCode = "200",
+               description = "File replaced successfully on the dataset")
+    @Tag(name = "replaceFilesInDataset", 
+         description = "Replace a file to a dataset")
+    @RequestBody(content = @Content(mediaType = MediaType.MULTIPART_FORM_DATA)) 
     public Response replaceFileInDataset(
                     @Context ContainerRequestContext crc,
                     @PathParam("id") String fileIdOrPersistentId,
@@ -497,7 +512,7 @@ public Response getFileData(@Context ContainerRequestContext crc,
     @GET
     @AuthRequired
     @Path("{id}/versions/{datasetVersionId}")
-    public Response getFileData(@Context ContainerRequestContext crc,
+    public Response getFileDataForVersion(@Context ContainerRequestContext crc,
                                 @PathParam("id") String fileIdOrPersistentId,
                                 @PathParam("datasetVersionId") String datasetVersionId,
                                 @QueryParam("includeDeaccessioned") boolean includeDeaccessioned,