Iqss/6497 semantic api #7414

qqmyers · 2020-11-16T20:54:52Z

What this PR does / why we need it: This adds an API call that allows metadata to be added using json-ld - in the same format as the OAI-ORE export, and without having to retrieve the existing metadata and having to understand the way Dataverse is storing things internally. It's one step in supporting dataset migration via RDA Bags (see #6497) that might be useful independent of additional api calls to migrate a dataset. For example, it allows submission of new terms and conditions.

Which issue(s) this PR closes:

Closes #

Special notes for your reviewer:

Suggestions on how to test this: see examples in comments below

Does this PR introduce a user interface change? If mockups are available, please link/include them here:

Is there a release notes update needed for this change?:

Additional documentation:

Conflicts: src/main/java/edu/harvard/iq/dataverse/api/Datasets.java

This reverts commit 8596ac8.

pdurbin

Overall, this is looking really good and I'm excited to have a smaller, more compact JSON format for creating a dataset. That was the dream of #3068 (which focused on output, initially).

I left comments and questions throughout. I'd be happy to get on a call to discuss them. I haven't run the code locally yet but I might like to do that too.

src/test/java/edu/harvard/iq/dataverse/api/DatasetsIT.java

conf/solr/8.8.1/schema_dv_mdb_copies.xml

doc/release-notes/6497-semantic-api.md

doc/sphinx-guides/source/developers/dataset-semantic-metadata-api.rst

src/main/java/edu/harvard/iq/dataverse/api/DatasetFieldServiceApi.java

src/main/java/edu/harvard/iq/dataverse/api/Datasets.java

src/main/java/edu/harvard/iq/dataverse/util/bagit/BagGenerator.java

src/main/java/edu/harvard/iq/dataverse/util/json/JsonLDTerm.java

Co-authored-by: Philip Durbin <philipdurbin@gmail.com>

…api.rst Co-authored-by: Philip Durbin <philipdurbin@gmail.com>

…averseCommunityConsortium/dataverse.git into IQSS/6497-semantic_api

qqmyers · 2021-07-02T18:14:06Z

@pdurbin - thanks for reviewing! I think I've addressed the straight-forward issues. The one thing that is tbd would be to decide on whether to leave the metadataOnOrig functionality in or not. It could probably be pulled out for now and treated as another PR (that could sit for a while) if there's concern about having it in the code now. (FWIW: I think DANS has created a custom metadata block for the metadata they had in their old system that doesn't match and therefore isn't relying on the metadataOnOrig functionality.)

If I missed something, let me know. I'm also happy to just talk through things as well.

pdurbin

Overall, the changes are looking good. A few things on my mind:

As suggested by @qqmyers, I vote we remove the metadataOnOrig code and tsv if we really don't need it. It's the kind of thing we can add in later with more discussion, if necessary. If we decided to keep it, it definitely needs to be documented.
In the guides, I'd be happy to have a little more of an intro to JSON-LD and examples of how to create a dataset with one author vs. multiple, for example. We can live without this though, so please consider it a "nice to have".
Under "Suggestions on how to test this" it says "see examples in comments below" but I'm confused about what this means. Perhaps it means the examples in the guides. I'd suggest clarifying this.
I'm still slightly weirded out that we have URLs like https://dataverse.org/schema/citation/Description in examples but they're 404s.
I left a comment about the case being used in the JSON-LD examples. At first glance it doesn't seem consistent.

doc/sphinx-guides/source/developers/dataset-semantic-metadata-api.rst

pdurbin · 2021-07-06T18:50:52Z

scripts/api/data/metadatablocks/migration.tsv

@@ -0,0 +1,5 @@
+#metadataBlock	name	dataverseAlias	displayName	blockURI											


This was all non-obvious to me from a quick look at the code. If it's on the table to remove the metadataOnOrig functionality, I think we should. If you get other opinions that we should keep it, please document how it works.

pdurbin · 2021-07-06T18:54:03Z

scripts/search/tests/data/dataset-finch1.jsonld

+  "http://purl.org/dc/terms/creator": {
+      "https://dataverse.org/schema/citation/author#Name": "Finch, Fiona",
+      "https://dataverse.org/schema/citation/author#Affiliation": "Birds Inc."
+  },


That's what I figured. When I was working on the Schema.org JSON-LD output I was shocked by how loosey goosey JSON-LD is.

At some point it might behoove us to add a little crash course on JSON-LD to the guides, or at least plenty of examples so that users get the hang of it. Otherwise, I think we can anticipate questions like "How do I specify multiple authors?"

src/main/java/edu/harvard/iq/dataverse/api/DatasetFieldServiceApi.java

src/main/java/edu/harvard/iq/dataverse/util/bagit/BagGenerator.java

pdurbin · 2021-07-06T19:14:09Z

doc/sphinx-guides/source/_static/api/dataset-create.jsonld

+    "https://dataverse.org/schema/citation/datasetContact#E-mail": "finch@mailinator.com",
+    "https://dataverse.org/schema/citation/datasetContact#Name": "Finch, Fiona"
+  },
+  "https://dataverse.org/schema/citation/Description": {


The capitalization seems inconsistent.

Contact and Description (title case) but also author, datasetContact and dsDescription (camel case)?

What are the rules?

These come from the citation.tsv block with the pattern <block name>/<field title> for single fields and <block name>/<field title>/<child field name> being used. That choice was made back when OAI_ORE/BagIt/archiving was introduced. The use if title was an attempt to use URIs that mirrored what users see in the UI. I'm less sure why I used name for child fields - not sure if there was a conflict or if it was an issue with originally just trying a flatter <blockname>/<childfield title> and realizing that there are multiple child fields with the title 'Name' for example.

Ok. Sounds like a preexisting condition to me. 😄 It would be nice to have more consistency but oh well. I assume we don't want to revisit decisions made during BagIt export.

qqmyers · 2021-07-08T17:08:03Z

I've removed the metadataOnOrig logic (and created https://github.com/GlobalDataverseCommunityConsortium/dataverse/tree/RDA-with_metadataOnOrig to keep that code around as a proof-of-concept for RDA/if/when needed).

pdurbin

I still haven't run the code myself but this seems ready for QA. Thanks for the pull request and for hanging there with changes I asked for, @qqmyers!

pdurbin · 2021-07-08T18:59:58Z

scripts/search/tests/data/dataset-finch1.jsonld

+  "http://purl.org/dc/terms/creator": {
+      "https://dataverse.org/schema/citation/author#Name": "Finch, Fiona",
+      "https://dataverse.org/schema/citation/author#Affiliation": "Birds Inc."
+  },


👍 for more docs over time. I think we can live with what we have now, especially since it's experimental.

qqmyers added 30 commits March 23, 2020 13:37

initial semantic API endpoint

4d70971

Merge remote-tracking branch 'IQSS/develop' into IQSS/6497

b2befca

merge new fields with existing ones

fb6421b

Merge remote-tracking branch 'IQSS/develop' into IQSS/6497

8bc3df6

Merge remote-tracking branch 'IQSS/develop' into IQSS/6497

5b828aa

differences from IQSS/develop that break compilation

f472b6c

Merge remote-tracking branch 'IQSS/develop' into IQSS/6497-semantic_api

bab11f0

Conflicts: src/main/java/edu/harvard/iq/dataverse/api/Datasets.java

Add jsonld lib to compact to local context

d305867

use expand/compact, refactor, add :startmigration endpoint

98d978e

try fix for parse error

2443702

log value

d442643

return dataset

fdeac97

manage versionState, add debug output

554e620

move debug ore generation after configuring dataset

bee7731

set versionstate, simplify, move terms init outside loop

f4cecd3

parse version number

a4189de

fix toStrings

e0de1db

debug null pointer in DataverseFieldTypeInputLevel

928a88e

add support for fields with their own formal URI

b78aed1

allow non-published to support debugging and future use

3a47630

refactor, use expanded version directly

3f8534b

add modification time

64af0e8

expanded has array with 1 val - handle it

04ee08a

log compound values to start

c7c2573

compact with no context for decontextualize

fc77f92

handle appending and compound fields

e287644

sort compound field children by display order

8596ac8

parse date/time correctly

ffbc05a

Revert "sort compound field children by display order"

0226b0d

This reverts commit 8596ac8.

typo

fad62f4

pdurbin reviewed Jul 1, 2021

View reviewed changes

pdurbin assigned qqmyers and unassigned pdurbin Jul 1, 2021

qqmyers and others added 6 commits July 2, 2021 12:20

fix test

e159003

Update doc/release-notes/6497-semantic-api.md

cf8b2b5

Co-authored-by: Philip Durbin <philipdurbin@gmail.com>

Update doc/sphinx-guides/source/developers/dataset-semantic-metadata-…

d5ff955

…api.rst Co-authored-by: Philip Durbin <philipdurbin@gmail.com>

add create example, remove solr schema copies file

61627d1

removed debug logging

1d54c68

Merge branch 'IQSS/6497-semantic_api' of https://github.com/GlobalDat…

bc82180

…averseCommunityConsortium/dataverse.git into IQSS/6497-semantic_api

missing header

4c1d31a

pdurbin mentioned this pull request Jul 6, 2021

Allow terms metadata to be changed with Native API #5899

Closed

pdurbin requested changes Jul 6, 2021

View reviewed changes

qqmyers added 2 commits July 8, 2021 12:32

remove metadataOnOrig per review

a5a745d

Merge remote-tracking branch 'IQSS/develop' into IQSS/6497-semantic_api

bd37e30

pdurbin approved these changes Jul 8, 2021

View reviewed changes

djbrooke unassigned qqmyers Jul 8, 2021

kcondon self-assigned this Jul 12, 2021

qqmyers added 5 commits July 13, 2021 12:54

add missing create method (in migrate PR)

0138ebb

No "@id" npe fix

13a7841

avoid npe in logging

86a08e3

only require "@id" when migrating

0c64c68

fix logging in create case

8e9f2f7

kcondon merged commit 15e7b7c into IQSS:develop Jul 13, 2021

djbrooke added this to the 5.6 milestone Jul 15, 2021

This was referenced Sep 30, 2022

Cannot update dataset with new value of field termsOfUse #3813

Closed

Templates: Able to update all (or multiple) Terms of Use + Access #2585

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Iqss/6497 semantic api #7414

Iqss/6497 semantic api #7414

qqmyers commented Nov 16, 2020 •

edited

Loading

pdurbin left a comment

qqmyers commented Jul 2, 2021

pdurbin left a comment

pdurbin Jul 6, 2021

pdurbin Jul 6, 2021

pdurbin Jul 6, 2021

qqmyers Jul 8, 2021 •

edited

Loading

pdurbin Jul 8, 2021

qqmyers commented Jul 8, 2021

pdurbin left a comment

pdurbin Jul 8, 2021

		@@ -0,0 +1,5 @@
		#metadataBlock name dataverseAlias displayName blockURI

Iqss/6497 semantic api #7414

Iqss/6497 semantic api #7414

Conversation

qqmyers commented Nov 16, 2020 • edited Loading

pdurbin left a comment

Choose a reason for hiding this comment

qqmyers commented Jul 2, 2021

pdurbin left a comment

Choose a reason for hiding this comment

pdurbin Jul 6, 2021

Choose a reason for hiding this comment

pdurbin Jul 6, 2021

Choose a reason for hiding this comment

pdurbin Jul 6, 2021

Choose a reason for hiding this comment

qqmyers Jul 8, 2021 • edited Loading

Choose a reason for hiding this comment

pdurbin Jul 8, 2021

Choose a reason for hiding this comment

qqmyers commented Jul 8, 2021

pdurbin left a comment

Choose a reason for hiding this comment

pdurbin Jul 8, 2021

Choose a reason for hiding this comment

qqmyers commented Nov 16, 2020 •

edited

Loading

qqmyers Jul 8, 2021 •

edited

Loading