739 html codebook #6081

lubitchv · 2019-08-09T15:35:32Z

It creates an html codebook during publishing. Html codebook can be accessed from ExportMetadata->DDI html codebook

Related Issues

closes Make a HTML version of the DDI Codebook XML #739

Pull Request Checklist

Unit tests completed
Integration tests: None
Deployment requirements, SQL updates, Solr updates, etc.: None
Documentation completed
[x ] Merged latest from "develop" branch and resolved conflicts

update

djbrooke · 2019-08-09T15:56:02Z

Thanks @lubitchv! I've added this to Code Review and we'll take a look soon.

Can you add a few notes here about how we'd test this? I think much of this is self explanatory, but I want to make sure we're not missing any details.

Tagging @jggautier for his thoughts as well.

coveralls · 2019-08-09T15:57:04Z

Coverage decreased (-0.009%) to 19.488% when pulling 2163dc6 on lubitchv:739-html-codebook into 271dbd0 on IQSS:develop.

djbrooke · 2019-08-09T16:02:57Z

@lubitchv, I added some docs about the new export format, feel free to check if these are in the correct spot and feel free to expand on them if necessary. Thanks again for this contribution!!

lubitchv · 2019-08-09T16:04:22Z

@djbrooke This can be tested by comparing corresponding XML file from ExportMetadata->DDI with html file from ExportMetadata->DDI html codebook
It should have the same sections, variables etc, such as Document Description (docDscr), Study Description (stdyDscr), File Description (fileDscr), Variable Description (dataDscr)

pdurbin

This is a very cool feature and might make it easier to apply stylesheets other XML output. I emailed @jggautier about #5960 (comment) but I'll probably add some screenshots and thoughts on that issue, even though it's a little off topic. In short https://easy.dans.knaw.nl/oai/?verb=ListMetadataFormats is much prettier than https://demo.dataverse.org/oai?verb=ListMetadataFormats

The code seems fine (some tests would be nice) so I guess I'll go ahead and click "Approve". If we care to, one of us can change "html" to "HTML" in the bundle.

pdurbin · 2019-08-09T16:33:50Z

src/main/resources/edu/harvard/iq/dataverse/codebook2-0.xsl

@@ -0,0 +1,2538 @@
+<?xml version="1.0" encoding="UTF-8"?>


I just wanted to remind everyone that there's a related codebook file at src/test/java/edu/harvard/iq/dataverse/export/ddi/codebook.xsd

pdurbin · 2019-08-09T16:34:17Z

src/main/java/propertyFiles/Bundle.properties

@@ -1209,6 +1209,7 @@ dataset.exportBtn.itemLabel.datacite=DataCite
 dataset.exportBtn.itemLabel.json=JSON
 dataset.exportBtn.itemLabel.oai_ore=OAI_ORE
 dataset.exportBtn.itemLabel.dataciteOpenAIRE=OpenAIRE
+dataset.exportBtn.itemLabel.html=DDI html codebook


I would suggest capitalizing this to "HTML".

pdurbin · 2019-08-09T16:35:04Z

src/main/java/edu/harvard/iq/dataverse/export/ddi/DdiExportUtil.java

@@ -1610,4 +1632,47 @@ private static boolean checkParentElement(XMLStreamWriter xmlw, String elementNa
        return true;
    }

+    public static void datasetHtmlDDI(JsonObject datasetDtoAsJson, DatasetVersion version, OutputStream outputStream) throws XMLStreamException {


How much effort would it be to write a test for this "datasetHtmlDDI" method?

To write a test to see that conversion from xml to html is successful would not be difficult. It can be added to DdiExportUtilTest. But to parse the resulting html file to see that the fields of xml converted properly to html would be quite cumbersome and time consuming.

@lubitchv ok, if you'd like to write the easier test, please let us know if you'd like us to pull this out of QA until you're done. Or we can leave it in QA and tests can be added in a future pull request. It's up to you. 😄

Ok, I will try to write the easier test on Monday. I think this PR can wait till I finish that test.

@lubitchv sounds good. Please ping us when you're ready for us to move it back to code review. Have a nice weekend!

Hey @lubitchv - thanks for the commit adding the test. Is this ready for another round of code review and then QA?

@djbrooke Not yet, I need to check some things out. I will let you know when its ready.

@djbrooke It is now ready for code review and QA.

Thanks @lubitchv! We'll take another look.

…erse into 739-html-codebook

pdurbin · 2019-08-14T11:21:15Z

I was curious how "DDI HTML codebook" looks and works. Here are some screenshots as of 2163dc6 showing how a new tab is opened:

pdurbin

This looks great but does it work on S3 or Swift? Do you know @lubitchv or can you tell @landreev ?

pdurbin · 2019-08-14T11:27:29Z

src/main/java/propertyFiles/Bundle.properties

@@ -1209,6 +1209,7 @@ dataset.exportBtn.itemLabel.datacite=DataCite
 dataset.exportBtn.itemLabel.json=JSON
 dataset.exportBtn.itemLabel.oai_ore=OAI_ORE
 dataset.exportBtn.itemLabel.dataciteOpenAIRE=OpenAIRE
+dataset.exportBtn.itemLabel.html=DDI HTML codebook


Title case is used for buttons, like this:

Suggested change

dataset.exportBtn.itemLabel.html=DDI HTML codebook

dataset.exportBtn.itemLabel.html=DDI HTML Codebook

pdurbin · 2019-08-14T11:30:01Z

src/main/java/edu/harvard/iq/dataverse/export/HtmlCodeBookExporter.java

+    @Override
+    public void exportDataset(DatasetVersion version, JsonObject json, OutputStream outputStream) throws ExportException {
+        try {
+            Path cachedMetadataFilePath = Paths.get(version.getDataset().getFileSystemDirectory().toString(), "export_ddi" + ".cached");


getFileSystemDirectory makes me suspect that this might not work on S3 or Swift.

@pdurbin You are probably right. I did not test it for S3 or Swift, it probably will not work with it. That is why I just replaced getFileSystemDirectory with getExport from ExportService. As I understand it should work for S3 and Swift.

@lubitchv perfect, thanks! I just moved this to QA.

pdurbin · 2019-08-14T11:36:57Z

I feel like this issue is close (screenshots above show it working, even) so I'm leaving it in code review but here's what I'm thinking.

If @lubitchv could merge my suggestion for the button label I'd appreciate it. I made it "DDI HTML Codebook" for consistency and to match the docs.
There a call to the filesystem that I'm not sure will work on S3 or Swift. This should be tested. Maybe @landreev can take a look. It's in HtmlCodeBookExporter.java.

pdurbin · 2019-08-14T15:12:05Z

@landreev nevermind, @lubitchv just made a change to support S3 and Swift too so I'm approved this pull request, which moved it to QA.

djbrooke · 2019-08-16T02:24:13Z

@lubitchv @kcondon @pdurbin - do I need to put anything in the release notes about re-running exports or will existing datasets get this new export automatically? Note we are already advising ReExportall for a citation metadata block change, so we may be covered anyway. Thanks for any guidance!

pdurbin · 2019-08-16T11:27:51Z

@djbrooke I think just talking about the new feature in the release note is enough. The first time someone tries to export metadata as an HTML codebook for a given dataset, it will be created on the fly and cached. That's my understanding anyway. In addition, according to this, missing exports will be created automatically nightly by a timer: http://guides.dataverse.org/en/4.15.1/admin/metadataexport.html#automatic-exports . Apologies if I'm getting any of this wrong.

djbrooke · 2019-08-16T13:14:06Z

@pdurbin perfect, thanks.

lubitchv and others added 7 commits July 15, 2019 12:05

Merge pull request #27 from IQSS/develop

c8462e1

update

html codebook

7c283ad

html codebook

537afe2

html

ff8ba65

fix html codebook reading

9722e8f

commit

6e4e80d

Merge branch 'IQSS-develop' into html-codebook

70e058f

adding documentation of new export format

f6edfdd

Update dataset-management.rst

e53fbb8

pdurbin approved these changes Aug 9, 2019

View reviewed changes

lubitchv added 2 commits August 9, 2019 13:51

capitalize HTML

a2d44d0

Merge branch '739-html-codebook' of https://github.com/lubitchv/datav…

6201c53

…erse into 739-html-codebook

pdurbin assigned lubitchv Aug 9, 2019

lubitchv added 2 commits August 12, 2019 12:29

Unit test html codebook

f8e7874

fix HtmlCodeBookExporter

2163dc6

djbrooke assigned pdurbin and unassigned lubitchv Aug 12, 2019

mheppler mentioned this pull request Aug 13, 2019

Make Dataverse produce valid DDI codebook 2.5 XML #3648

Closed

This was referenced Aug 13, 2019

Make a HTML version of the DDI Codebook XML #739

Closed

ec2-create-instance.sh "branch" and "repo" flags don't seem to work IQSS/dataverse-ansible#97

Closed

pdurbin reviewed Aug 14, 2019

View reviewed changes

pdurbin removed their assignment Aug 14, 2019

Fix for reading ddi cached file

8acc07f

pdurbin approved these changes Aug 14, 2019

View reviewed changes

kcondon self-assigned this Aug 14, 2019

kcondon merged commit 842197d into IQSS:develop Aug 14, 2019

djbrooke added this to the 4.16 milestone Aug 16, 2019

djbrooke mentioned this pull request Aug 22, 2019

Update Codebook Docs #6111

Closed

jggautier mentioned this pull request Jul 29, 2020

Should the Geographic Bounding Box allow optional coordinates and multiple boxes in the XML? #7108

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

739 html codebook #6081

739 html codebook #6081

lubitchv commented Aug 9, 2019 •

edited

Loading

djbrooke commented Aug 9, 2019

coveralls commented Aug 9, 2019 •

edited

Loading

djbrooke commented Aug 9, 2019

lubitchv commented Aug 9, 2019

pdurbin left a comment

pdurbin Aug 9, 2019

pdurbin Aug 9, 2019

pdurbin Aug 9, 2019

lubitchv Aug 9, 2019

pdurbin Aug 9, 2019

lubitchv Aug 9, 2019

pdurbin Aug 9, 2019

djbrooke Aug 12, 2019

lubitchv Aug 12, 2019

lubitchv Aug 12, 2019

djbrooke Aug 12, 2019

pdurbin commented Aug 14, 2019

pdurbin left a comment

pdurbin Aug 14, 2019

pdurbin Aug 14, 2019

lubitchv Aug 14, 2019

pdurbin Aug 14, 2019

pdurbin commented Aug 14, 2019

pdurbin commented Aug 14, 2019

djbrooke commented Aug 16, 2019

pdurbin commented Aug 16, 2019

djbrooke commented Aug 16, 2019

	dataset.exportBtn.itemLabel.html=DDI HTML codebook
	dataset.exportBtn.itemLabel.html=DDI HTML Codebook

739 html codebook #6081

739 html codebook #6081

Conversation

lubitchv commented Aug 9, 2019 • edited Loading

Related Issues

Pull Request Checklist

djbrooke commented Aug 9, 2019

coveralls commented Aug 9, 2019 • edited Loading

djbrooke commented Aug 9, 2019

lubitchv commented Aug 9, 2019

pdurbin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pdurbin commented Aug 14, 2019

pdurbin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pdurbin commented Aug 14, 2019

pdurbin commented Aug 14, 2019

djbrooke commented Aug 16, 2019

pdurbin commented Aug 16, 2019

djbrooke commented Aug 16, 2019

lubitchv commented Aug 9, 2019 •

edited

Loading

coveralls commented Aug 9, 2019 •

edited

Loading