auxiliary file download: JSON file extension/type is lost

[raprasad]

What steps does it take to reproduce the issue?

1. Use the Dataverse API to upload an Auxiliary file that is a JSON file (e.g. dp_release.json):
   • api: https://guides.dataverse.org/en/latest/developers/aux-file-support.html
2. Download the file via the UI
3. The downloaded file has ".txt" appended to the end. The user may rename the file to end with ".json" but there is no direction that it is supposed to be a JSON file.
   (Note: this doesn't happen when adding PDFs as auxiliary files)

• When does this issue occur?
  When downloading an auxiliary JSON file.

• Which page(s) does it occurs on?
  Dataset page, file download dropdown.

• What happens?
  See steps to reproduce above

• To whom does it occur (all users, curators, superusers)?
  Tested this as the dataset owner.

• What did you expect to happen?
  The file would be downloaded with a .json extension

Which version of Dataverse are you using?
v. 5.5 build develop-6cb87ee5d

Any related open or closed issues to this bug report?

[djbrooke]

Thanks @raprasad. We can take a look. If you wouldn't mind, could you please see if this happens in 5.8 (on demo.dataverse.org)?

[pdurbin]

This is definitely a known issue and has been true since 5.5. 5c67f95 is a related commit we can look at. When we fix this we should also adjust the tests that have comments like FIXME: This should be ".json" instead of ".txt".

In short, the context type (MIME type) for these JSON files is being stored in Dataverse as "text/plain" rather than "application/json".

[qqmyers]

Looks like the core problem is that the mimetype detection used (

dataverse/src/main/java/edu/harvard/iq/dataverse/AuxiliaryFileServiceBean.java

Line 84 in ec4908b

auxFile.setContentType(tika.detect(storageIO.getAuxFileAsInputStream(auxExtension)));

) doesn't recognize application/json and just assigns it text/plain. The best suggestion for ~fixes I've seen are to use the tike.detect(Stream, Metadata) method and pass a Metadata with the RESOURCE_NAME_KEY set to a filename/extension associated with application/json or Metadata.CONTENT_TYPE set to application/json (somewhat circular since we're trying to detect the type, but this evidently makes tika assign application/json as the right subtype of text/plain), or to just try parsing with a json parser (perhaps only if tika says text/plain to start).

[pdurbin]

Since this is at the top of Up Next I thought I'd mention that pull request #8237 is related so watch our for potential merge conflicts.

[djbrooke]

• There are two approaches for this - we could detect and provide an extension based on the result of that detection OR we could just give back the file extension that was put in in the first place. To be discussed if we want to do what we do it other parts of the application (that may be driven by ingest) or if we want to treat this separately due to the use cases around auxiliary files - that the client knows what it's uploading and therefore what it wants to get back.

[raprasad]

@pdurbin: Has a format been decided upon for sending over the content type? If so, will it be backward compatible with the current API? Thanks.

[pdurbin]

@raprasad yes, a format has been decided. Pull request #8282 has been merged and it's probably easiest to show the "diff" of the curl command:

That is to say, from curl you'll pass something like this:

-F file=@data.json;type=application/json

And yes, the change is backward compatible. If you don't specify a content type we'll run the file through Tika.detect in the hopes of figuring out what it is.

[raprasad]

Thanks @pdurbin