Test failures in 4.8.0 wellspring.

[WardF]

Update: Renamed issue, focusing on failures so that we can get 4.8.0 released finally. Adding failure observed in ncdump/tst_nccopy4 as well.

In preparation for the 4.8.0 release, I am testing with Visual Studio 2017 on Windows, using the MSYS shell to provide 'bash'. I am seeing the following failures:

64-bit tests

• ncdump/tst_nccopy4
• nc_test4/tst_filter
• nczarr_test/tst_nccopyz

32-bit tests

• nc_test4/test_filter
• ncdap_tst_remote3
• nczarr_test/run_ut_map
• nczarr_test/run_ut_mapapi
• nczarr_test/tst_nccopyz

Further details added below as I work through this.

[DennisHeimbigner]

Do you have any more info on this?

[WardF]

I've been trying to set up a cygwin+Visual Studio working environment, to see if it's something with my MSYS-based shell, with no luck. The tests that are failing are all shell scripts, let me see if I can get some further information running them by hand.

[WardF]

@DennisHeimbigner For 64-bit tst_filter.sh I'm seeing the following (additional info and test ouput to follow).

# ./tst_filter.sh
findplugin.sh loaded
final HDF5_PLUGIN_PATH=D:/netcdf-c/build/plugins
*** Testing dynamic filters using API

*** Testing API: bzip2 compression.
show parameters for bzip2: level=9
show chunks: chunks=4,4,4,4

*** Testing API: bzip2 decompression.
data comparison: |array|=256
no data errors
*** Pass: API dynamic filter

*** Testing dynamic filters parameter passing
test1: compression.
test: nparams=14: params= 1 239 23 65511 27 77 93 1145389056 3287505826 1097305129 1 2147483648 4294967295 4294967295
dimsizes=4,4,4,4
chunksizes=4,4,4,4
test1: decompression.
data comparison: |array|=256
no data errors
test2: dimsize % chunksize != 0: compress.
test: nparams=14: params= 2 239 23 65511 27 77 93 1145389056 3287505826 1097305129 1 2147483648 4294967295 4294967295
dimsizes=4,4,4,4
chunksizes=4,4,4,4
nbytes = 1024 chunk size = 1024
test2: dimsize % chunksize != 0: decompress.
nbytes = 1024 chunk size = 1024
data comparison: |array|=256
no data errors
nbytes = 1024 chunk size = 1024
*** Pass: parameter passing
*** Testing dynamic filters using ncgen
*** Pass: ncgen dynamic filter
*** Testing dynamic filters using nccopy
        *** Testing simple filter application
16d15
<               var:_Filter = "307,9,4";

[WardF]

Reviving and revamping this issue as part of the concentrated push to get 4.8.0 out the door.

[WardF]

Failure in ncdump/tst_nccopy4:

 *** Creating chunkable test file tst_chunking.nc...
  ok.
  *** Tests successful!
  *** Test that nccopy -c can chunk and unchunk files
  *** Test that nccopy -c works as intended for record dimension default (1)
  *** Test that nccopy -c dim/n works as intended
  *** Test that nccopy -F var1,none works as intended
  NetCDF: Variable not found
  Location: file ; line 2052

[WardF]

ncdump/tst_nccopy4.sh

The issue on Windows (using bash from msys2 at least) is the -c / argument to nccopy.

[DennisHeimbigner]

The only thing I could find was a memory leak. I will post a fix for that.

[DennisHeimbigner]

BTW, are you in a position to do this set of commands?

export SETX=1
sh -x ./tst_nccopy4.sh

[WardF]

@DennisHeimbigner, on Windows when I run the line:

$ nccopy -M0 -4 -F var1,none -c / tst_nofilters.nc tmp_nofilters.nc

It returns

NetCDF: Variable not found
Location: file ; line 2052

If I invoke

$ nccopy -M0 -4 -F var1,none tst_nofilters.nc tmp_nofilters.nc

it returns with no error. The issue, using bash as provided by msys2, appears to be the -c / argument.

[DennisHeimbigner]

What ./configure flags are you using?

[WardF]

The defaults; I'm using cmake on Windows and not specifying anything beyond dependency paths and compiler.

[WardF]

So the commands work fine in the CMD Window, which obviously doesn't work with our bash-based test scripts, but do indicate there is some issue with the way MSYS is interpreting and passing the arguments. This is not enough to hold up the 4.8.0 release, so I will test these 'failures' in a CMD window manually, and figure out which tests are truly failing and which ones are a reflection of this msys bash issue, which can then be resolved later.

[DennisHeimbigner]

I have got to believe it is a \r problem.
You need to add a 'set -x' command to the front of
ncdump/tst_nccopy4
and capture the output to a file in hopes we can see
if extraneous \r characters are being inserted.

[WardF]

Hi Dennis, I have done that but it does not appear to be the issue.

[WardF]

It appears to be a fairly common in msys2 bash; long story short, escaping / with a /, e.g. passing -c // fixes the issue.

[DennisHeimbigner]

Can you check if -c / also works?

[DennisHeimbigner]

You will be amused to know that the -c // case already occurs in tst_nccopy4.sh
some lines above where the current failure occurs. Further, there is already code in
nccopy to handle -c // for linux systems.

[DennisHeimbigner]

this line did not appear correctly

Can you check if -c / also works?

It should be

Can you check if -c \/ also works?

[WardF]

Escaping the slash doesn't work for whatever reason, but the double-slash does. Making the changes in the wellspring branch. Thanks!

[DennisHeimbigner]

Ok, but remember that you do not need to make the fix msys specific.
The nccopy code already handles it.