Bulk upload support

[philippconzett]

I just noticed the following desired characteristics in the COAR Community Framework for Good Practices in Repositories (https://doi.org/10.5281/zenodo.4110829); cf.:

9.6 The submission / deposit system supports both individual creator uploads and bulk uploads of records and resources.

Bulk upload of files seems to be pretty straightforward in Dataverse, but what about bulk creation of datasets including file upload? I guess you can use DVUploader and/or write a script to get this done through API? Would it be useful to make such bulk deposit more convenient?

[pdurbin]

@philippconzett you've been playing with https://github.com/gdcc/dvwebloader

Is it helping? What else do we need?

[pdurbin]

Oh and we're shipping Globus support with 5.12:

• GDCC/Globus and Big Data Support #8891

Both of these are thanks to @qqmyers! 🎉

[pdurbin]

Another thought is that migrations are a form of bulk uploads. @qqmyers recently added a migration API: https://guides.dataverse.org/en/5.12/developers/dataset-migration-api.html

@philippconzett at some point we should figure out the "definition of done" for this issue. Perhaps we document what we've got in the guides with a reference to that framework you mentioned. Then close this issue and keep adding more bulk upload features (new issues for each feature, of course) and keep adding to whatever documentation we write for this issue. Just an idea. I'm curious what you think the end game should be! 😄

[philippconzett]

Thanks for this reminder, Phil, and sorry for my late reply. You're right, I think we can close this issue. In addition to the features already mentioned above, I guess we also may add pyDataverse and other client libraries; cf. https://guides.dataverse.org/en/latest/api/client-libraries.html?highlight=pydataverse#client-libraries. I'm going to close the issue.