MyData metadata validity facet violates Don't Make Me Think

[pdurbin]

I was just playing around with the "Creating datasets with incomplete metadata through API" feature developed in this PR:

• 8822 incomplete datasets via api #8940

I'm on Dataverse 5.14.

It's a cool feature, but I think MyData suffers a bit from it.

Imagine the following scenario:

• a user creates a collection (dataverse), a dataset, and uploads a file
• time passes
• the user is looking for their dataset and files

Below is what might happen.

User clicks MyData

User sees "sorry no, results are found"

Now what? 🤔

Realistically, at this point the user probably contacts support. Where's my data? Help!

A user with time on their hands and a penchant for clicking things might explore further.

What if I uncheck "incomplete metadata"?

Nope, nothing.

What if I uncheck "incomplete metadata" AND "valid"?

Finally, we are getting somewhere! There's my dataset and my collection (dataverse)! But where's my file?

Check files to see files

As usual, files are not selected by default but if you check "files" you can see them. This is out of scope for this issue.

In conclusion

When a user visits MyData they should see their data, not "Sorry, no results were found." They shouldn't have to uncheck boxes to see their data.

[ErykKul]

@pdurbin, It looks like you are missing the solr field for metadata validity. Updating schema and reindexing should fix it. I hope that the documentation mentions this, at least the release notes should. I have verified this on our pilot, which is on 5.14:

[ErykKul]

@pdurbin
I was revisiting the code for that feature, and I might have been too defensive with the programming:
IndexServiceBean.java#L814-L816

        if (JvmSettings.API_ALLOW_INCOMPLETE_METADATA.lookupOptional(Boolean.class).orElse(false)) {
            solrInputDocument.addField(SearchFields.DATASET_VALID, valid);
        }

This has a small advantage in performance when you are not using that feature. It also makes the schema update and reindexing unnecessary in that case. However, you need to reindex all dataset after enabling the feature. If you disable it, add some dataset, and enable it again, then you also need to reindex the datasets as the newly added datasets do not have the necessary solr field. The field is only added to datasets while the feature is enabled. It might make more sense to remove the check and always add the completeness field to the solr document. This would throw exceptions if your schema is not up-to-date, so you know you need to update the schema (and reindex), and remove the complexity of always reindexing all datasets after enabling this feature. What do you think? I can also simply add this information in the documentation for that feature.

[ErykKul]

@pdurbin The PR should have simplified it. Can you check the description of the PR to see if it is clear what it does?

[pdurbin]

@ErykKul I just took a quick look at the PR you opened:

• Always add validity field when indexing a dataset #9964

This helps explain the behavior I was seeing. Thanks!