maven build fails because test NewDTAFileReaderTest.testDates() fails #4988

aseeland · 2018-08-21T14:16:32Z

I tried to build dataverse 4.9.2 with maven and got the following error:

Running edu.harvard.iq.dataverse.ingest.tabulardata.impl.plugins.dta.NewDTAFileReaderTest
Tests run: 8, Failures: 1, Errors: 0, Skipped: 4, Time elapsed: 0.036 sec <<< FAILURE!
testDates(edu.harvard.iq.dataverse.ingest.tabulardata.impl.plugins.dta.NewDTAFileReaderTest)  Time elapsed: 0.01 sec  <<< FAILURE!
org.junit.ComparisonFailure: expected:
<...2018-06-20    2018-11-0[5     2018-06-01      2018-01-01      2018-01-01      2018
2595-09-27 06:58:52.032 2018-06-20      2018-11-05      2018-06-01      2018-04-01      2018-01-01      2018
2595-09-27 06:58:52.032 2018-06-20      2018-11-05      2018-06-01      2018-07-01      2018-07-01      2018
2595-09-27 06:58:52.032 2018-06-20      2018-11-05]     2018-06-01      2018-11-...> 
but was:
<...2018-06-20     2018-11-0[4     2018-06-01   2018-01-01       2018-01-01      2018
2595-09-27 06:58:52.032 2018-06-20      2018-11-04      2018-06-01      2018-04-01      2018-01-01      2018
2595-09-27 06:58:52.032 2018-06-20      2018-11-04      2018-06-01      2018-07-01      2018-07-01      2018
2595-09-27 06:58:52.032 2018-06-20      2018-11-04]     2018-06-01      2018-11-...>

The 2nd date seems to be wrong (2018-11-05 instead of 2018-11-04).
I don't use STATA, so I was not able to open the file dataverse/src/test/java/edu/harvard/iq/dataverse/ingest/tabulardata/impl/plugins/dta/dates.dta but I am wondering if you guys also get this failure.

The text was updated successfully, but these errors were encountered:

pdurbin · 2018-08-21T14:26:25Z

@aseeland hi! Thanks for the bug report. If you look at https://travis-ci.org/IQSS/dataverse/branches you can see that we are not (thankfully!) getting that build error. Everything is green:

In pull request #4483 we added export LANG="en_US.UTF-8" for #4482 because tests were failing when the locale was it_IT. Do you think this problem could be similar?

Also, when you say you're trying to build 4.9.2 are you saying you're starting at that tag, which is commit 4ad7bf5 or are you saying you're trying to build the latest from the "develop" branch? If you could let us know which commit you are on, it would be helpful. Thanks!

aseeland · 2018-08-22T09:15:47Z

Hi @pdurbin . Yes, maybe the problem is cause by wrong locale settings... When I check my calendar, the 4th of november is a Sunday, and the 5th is a Monday. So, maybe it is a problem with the definition of the first day of the week?
On the other hand, $LANG is set to en_US.UTF-8 and even if I change it to en_GB.UTF-8 (GB should have Monday as first day of the week) or set in addition $LC_TIME this has no effect on the test failure.
Have you set other locale variables than $LANG ?

Yes, wtih v4.9.2 I mean the tag (commit 4ad7bf5). Thank you for your help!

aseeland · 2018-08-22T11:34:50Z

Hello again.
If I use TZ=/usr/share/zoneinfo/GMT mvn clean package to build dataverse, the build succeeds.
However, with my local time zone (/usr/share/zoneinfo/Europe/Berlin) I get the failure.
Now you can try whether you can reproduce the failure on your system.

I think the problem might be that the calendar that reads the STATA files is set to GMT, but the readFileToString-method in the unit test interprets the dates with the locale time zone.
I'm not sure how one can fix this behaviour.

pdurbin · 2018-08-22T12:07:24Z

@aseeland good troubleshooting. I can confirm that on the v4.9.2 tag when I run...

TZ=/usr/share/zoneinfo/Europe/Berlin mvn clean package

... I get the same error:

Results :

Failed tests: 
  NewDTAFileReaderTest.testDates:77 expected:<...2018-06-20	2018-11-0[5	2018-06-01	2018-01-01	2018-01-01	2018
2595-09-27 06:58:52.032	2018-06-20	2018-11-05	2018-06-01	2018-04-01	2018-01-01	2018
2595-09-27 06:58:52.032	2018-06-20	2018-11-05	2018-06-01	2018-07-01	2018-07-01	2018
2595-09-27 06:58:52.032	2018-06-20	2018-11-05]	2018-06-01	2018-11-...> but was:<...2018-06-20	2018-11-0[4	2018-06-01	2018-01-01	2018-01-01	2018
2595-09-27 06:58:52.032	2018-06-20	2018-11-04	2018-06-01	2018-04-01	2018-01-01	2018
2595-09-27 06:58:52.032	2018-06-20	2018-11-04	2018-06-01	2018-07-01	2018-07-01	2018
2595-09-27 06:58:52.032	2018-06-20	2018-11-04]	2018-06-01	2018-11-...>

Tests run: 545, Failures: 1, Errors: 0, Skipped: 7

[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 30.696 s

I'm not sure how to make the test work for both TZ settings either.

Do you feel like making a pull request? Just yesterday we merged pull request #4989 and you could follow the instructions in the dev guide to annotate the failing test as "non-essential":

https://github.com/IQSS/dataverse/blob/6ad8056b0ff16b2a355e56ea501025729286a208/doc/sphinx-guides/source/developers/testing.rst#running-non-essential-excluded-unit-tests

You would make the pull request from the latest in "develop" rather than the tag you're on, as explained at http://guides.dataverse.org/en/4.9.2/developers/version-control.html#create-a-new-branch-off-the-develop-branch

aseeland · 2018-08-23T09:12:48Z

@pdurbin Do you really want a pull request where the test is annotated as "non-essential"?

Can you maybe help me in thinking about the implications this timezone issue might have?
Are STATA files always (independent where they are created) saved in GMT time zone?
If the users of our dataverse open STATA-files, will they see the right dates?
If users with different time zones use dataverse to share STATA-files, will they have problems with dates?

pdurbin · 2018-08-23T10:57:24Z

@aseeland it's up to you if you want to make a pull request. These tests have only existed for a couple months, added in pull request #4708. Also, annotating a test as non-essential doesn't mean that it's never run. As indicated in the .travis.yml in the root of this repo, Travis CI runs these tests by activating the "all-unit-tests" profile. This happens on every commit. So yes, I'm comfortable with annotating this test a non-essential. That said, I (and probably others) can certainly help you think through the implications you suggest. Perhaps you've found a real bug. I would suggest uploading the file that the test is failing on to your system (you'll probably need to disable the test temporarily to get Dataverse installed) and compare the results to uploading it to https://demo.dataverse.org to see if you see any differences.

poikilotherm · 2018-09-17T14:28:24Z

I can reproduce this bug, too and it hits even more tests on my machine with LANG=de_DE.UTF-8:

$ LANG="de_DE.UTF-8" mvn test
[...]
[ERROR] Failures: 
[ERROR]   FileSizeCheckerTest.testBytesToHumanReadable:34 expected:<[1 B, 1023 B, 1.9 KB, 122.8 KB, 2.6 GB, 10.9 TB]> but was:<[1 B, 1023 B, 1,9 KB, 122,8 KB, 2,6 GB, 10,9 TB]>
[ERROR]   NewDTAFileReaderTest.testDates:77 expected:<...2018-06-20     2018-11-0[5     2018-06-01      2018-01-01      2018-01-01      2018
2595-09-27 06:58:52.032 2018-06-20      2018-11-05      2018-06-01      2018-04-01      2018-01-01      2018
2595-09-27 06:58:52.032 2018-06-20      2018-11-05      2018-06-01      2018-07-01      2018-07-01      2018
2595-09-27 06:58:52.032 2018-06-20      2018-11-05]     2018-06-01      2018-11-...> but was:<...2018-06-20     2018-11-0[4     2018-06-01      2018-01-01      2018-01-01      2018
2595-09-27 06:58:52.032 2018-06-20      2018-11-04      2018-06-01      2018-04-01      2018-01-01      2018
2595-09-27 06:58:52.032 2018-06-20      2018-11-04      2018-06-01      2018-07-01      2018-07-01      2018
2595-09-27 06:58:52.032 2018-06-20      2018-11-04]     2018-06-01      2018-11-...>
[ERROR] Errors: 
[ERROR]   CSVFileReaderTest.testVariableUNFs:350 » ArrayIndexOutOfBounds 1
[INFO] 
[ERROR] Tests run: 539, Failures: 2, Errors: 1, Skipped: 4
[...]

~~Isn't there a way to make Maven use a standard language and time information?~~

Maybe set this to your Harvard based settings and give a hint about this in the dev docs?
https://stackoverflow.com/questions/17774331/maven-default-locale-not-same-with-os-locale

~~@pdurbin What would be the correct settings at your place?~~

Sorry, no proper reading. Anyway I suggest setting the locale in the maven file to get rid of the locale problem. Will do some research on the TZ stuff and how to fix the test...

pdurbin · 2018-09-17T14:39:46Z

@poikilotherm sounds great. If you can add the right settings to pom.xml maybe we can back out of the Docker-specific fix in pull request #4483? Or leave it in, if necessary.

poikilotherm · 2018-09-17T15:58:58Z

The error in #4482 just looks like the issues I get on my machine... As the 1prep.sh seems to fail at the Maven build, this most certainly would be fixed, too.

I am going to send a PR if I get the time today, otherwise on Wednesday.

pdurbin · 2018-09-17T16:10:05Z

@poikilotherm great! I just dragged this issue to our new "Community Dev" column at https://waffle.io/IQSS/dataverse and assigned it to you. Thanks! If you have any questions, please let us know!

pdurbin · 2018-09-17T16:55:31Z

@poikilotherm thanks for pull request #5061! I assume it's ready for code review so I dragged it over in Waffle.

pdurbin · 2018-09-17T17:12:45Z

@poikilotherm hi! I just left a code review. I'm just asking about a tiny change having to do with timezones. Please let me know what you think. Thanks!

pdurbin · 2018-09-17T18:00:16Z

Looks good to me as of 8d3eb8e. Moving to QA.

…uilds Fix #4988. Handle Maven Surefire locale and timezone

…he argLine stuff from IQSS#4988 to a property.

pdurbin assigned poikilotherm Sep 17, 2018

pdurbin added the communitydev label Sep 17, 2018

pdurbin changed the title ~~maven build fails becaue test NewDTAFileReaderTest.testDates() fails~~ maven build fails because test NewDTAFileReaderTest.testDates() fails Sep 17, 2018

poikilotherm mentioned this issue Sep 17, 2018

Fix #4988. Handle Maven Surefire locale and timezone #5061

Merged

5 tasks

pdurbin added Status: Code Review and removed communitydev labels Sep 17, 2018

pdurbin unassigned poikilotherm Sep 17, 2018

pdurbin added communitydev and removed Status: Code Review labels Sep 17, 2018

pdurbin assigned poikilotherm Sep 17, 2018

pdurbin added Status: QA and removed communitydev labels Sep 17, 2018

pdurbin unassigned poikilotherm Sep 17, 2018

kcondon self-assigned this Sep 17, 2018

kcondon closed this as completed in 381eea3 Sep 17, 2018

kcondon added a commit that referenced this issue Sep 17, 2018

Merge pull request #5061 from poikilotherm/4988-fix-locale-tz-maven-b…

cb07a1b

…uilds Fix #4988. Handle Maven Surefire locale and timezone

kcondon removed the Status: QA label Sep 17, 2018

djbrooke added this to the 4.9.3 - Optional File PIDs, Initial Internationalization Work milestone Sep 18, 2018

poikilotherm added a commit to poikilotherm/dataverse that referenced this issue Oct 1, 2018

Fix IQSS#5122. Netbeans does not like the ${argLine} var, thus move t…

89eb58e

…he argLine stuff from IQSS#4988 to a property.

poikilotherm added a commit to poikilotherm/dataverse that referenced this issue Oct 2, 2018

Fix IQSS#5122. Netbeans does not like the ${argLine} var, thus move t…

409c91c

…he argLine stuff from IQSS#4988 to a property.

poikilotherm added a commit to poikilotherm/dataverse that referenced this issue Oct 2, 2018

Fix IQSS#5122. Netbeans does not like the ${argLine} var, thus move t…

cad9cb4

…he argLine stuff from IQSS#4988 to a property.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

maven build fails because test NewDTAFileReaderTest.testDates() fails #4988

maven build fails because test NewDTAFileReaderTest.testDates() fails #4988

aseeland commented Aug 21, 2018

pdurbin commented Aug 21, 2018

aseeland commented Aug 22, 2018

aseeland commented Aug 22, 2018

pdurbin commented Aug 22, 2018

aseeland commented Aug 23, 2018

pdurbin commented Aug 23, 2018

poikilotherm commented Sep 17, 2018 •

edited

Loading

pdurbin commented Sep 17, 2018

poikilotherm commented Sep 17, 2018

pdurbin commented Sep 17, 2018

pdurbin commented Sep 17, 2018

pdurbin commented Sep 17, 2018

pdurbin commented Sep 17, 2018

maven build fails because test NewDTAFileReaderTest.testDates() fails #4988

maven build fails because test NewDTAFileReaderTest.testDates() fails #4988

Comments

aseeland commented Aug 21, 2018

pdurbin commented Aug 21, 2018

aseeland commented Aug 22, 2018

aseeland commented Aug 22, 2018

pdurbin commented Aug 22, 2018

aseeland commented Aug 23, 2018

pdurbin commented Aug 23, 2018

poikilotherm commented Sep 17, 2018 • edited Loading

pdurbin commented Sep 17, 2018

poikilotherm commented Sep 17, 2018

pdurbin commented Sep 17, 2018

pdurbin commented Sep 17, 2018

pdurbin commented Sep 17, 2018

pdurbin commented Sep 17, 2018

poikilotherm commented Sep 17, 2018 •

edited

Loading