New3rd party libraries

[sgpearse]

This PR updates Vapor to use ZFP enabled HDF5 and an updated version of Ospray. Users must set the HDF5_PLUGIN_PATH to Vapor's /lib directory to allow for ZFP enabled compression.

Since not all libraries were rebuilt, the newly built libraries had to be targeted at /usr/local/VAPOR-Deps/2019-Aug again.

The new libraries can be found on google drive. They have been named 2019-Aug-[OS]-2022.tar.xz

Links here:
Ubuntu
CentOS
OSX
Windows

d4810cb: clangTidyOutput.txt

Fixes #3179
Fixes #3158
Fixes #3174
Fixes #2816

[shaomeng]

Hi Scott, do you mind updating the glm library too? It's a header-only library so should be easy to deal with, yet updating it is aligned with our ultimate goal. (https://github.com/g-truc/glm)

[clyne]

@sgpearse can you please identify which libraries were updated and their version numbers?

[StasJ]

Is there a reason we will require users to manually set HDF5_PLUGIN_PATH to enable ZFP?

[sgpearse]

@ALL - HDF5 seems to be working on Windows now. I'll request re-reviews once I can verify that the build systems are referencing the new libraries. Answers to your questions are below.

Hi Scott, do you mind updating the glm library too?

@shaomeng - No problem.

@sgpearse can you please identify which libraries were updated and their version numbers?

@clyne
HDF5 - 1.12.2
HDF5 Plugins - 1.12.2
NetCDF - 4.8.1
Ospray - 2.9.0
GLM - 0.9.9.8

Is there a reason we will require users to manually set HDF5_PLUGIN_PATH to enable ZFP?

@StasJ - From what I've read, this is necessary. From the NetCDF documentation:

At the moment, NetCDF uses the existing HDF5 environment variable HDF5_PLUGIN_PATH to locate the directories in which filter wrapper shared libraries are located. This is used both for the HDF5 filter wrappers but also the NCZarr codec wrappers.

There may be a way to use HDF5's API for this but I'm not sure what it entails.

[sgpearse]

Installers have been verified on all platforms with the new libraries. If you'd like to try the installers, you can find them in the artifacts on these builds.

@StasJ I was able to remove the need for HDF5_PLUGIN_PATH on our UNIX systems. I had no luck building SZlib+HDF5+NetCDF together, so I used Unidata's NetCDF binary installer for Windows. This leaves us without the HDF5 library needed to make the programmatic call for finding the plugin path.

@StasJ and @clyne - the programatic call for setting the HDF5 plugin path is currently in apps/vaporgui/main.cpp. Should this be moved elsewhere?

[clyne]

@sgpearse it looks to me like you are calling H5PLreplace() on Mac and Linux platforms, not windows. I'm a little confused because I thought you only need to set the plug path explicitly on Windows systems?

Regardless, setting the path in the vaporgui app. is probably not ideal; our conversion utilities won't work. Any thoughts on this @StasJ? I'm not sure if it is a disaster if we can't convert to/from hdf5 to VDC on Windows platforms. We may just have to live with that.

[shaomeng]

@sgpearse it looks to me like you are calling H5PLreplace() on Mac and Linux platforms, not windows. I'm a little confused because I thought you only need to set the plug path explicitly on Windows systems?

Regardless, setting the path in the vaporgui app. is probably not ideal; our conversion utilities won't work. Any thoughts on this @StasJ? I'm not sure if it is a disaster if we can't convert to/from hdf5 to VDC on Windows platforms. We may just have to live with that.

Here's an idea: in case that we have to distribute VAPOR with different capabilities among 3 major platforms, or even capabilities that are easy to go wrong within the camp of Linux for example, we provide a utility program that detects those variables and report to the user. It could be some added fields to "vaporversion".
The following capabilities can be subject to detection and report:

1. Whether ZFP (or any other) compression is supported
2. Whether OpenMP is supported
3. On tricky grids, whether GPU accelerated DVR is supported

[StasJ]

@sgpearse it looks to me like you are calling H5PLreplace() on Mac and Linux platforms, not windows. I'm a little confused because I thought you only need to set the plug path explicitly on Windows systems?

Regardless, setting the path in the vaporgui app. is probably not ideal; our conversion utilities won't work. Any thoughts on this @StasJ? I'm not sure if it is a disaster if we can't convert to/from hdf5 to VDC on Windows platforms. We may just have to live with that.

Couldn’t we just set the plugin path in the VDC API when attempting to load an HDF5 dataset? We could keep track of whether it had been set or the HDF5 API allows you to query the existing plugin paths.

[clyne]

Couldn’t we just set the plugin path in the VDC API when attempting to load an HDF5 dataset? We could keep track of whether it had been set or the HDF5 API allows you to query the existing plugin paths.

That could be done, but to ensure all of our readers and writers that access NetCDF files (e.g. DCMPAS, DCWRF, VDNetCDF, etc.) are supported we would have to do this in multiple places; there is not one single place where NetCDF is initialized, unfortunately.

Another option might be to not touch the code, and to instead set the HDF5_PLUGIN_PATH. It looks like HDF5 will honor that. I don't what is involved in setting an env variable for our Windows installer.

[clyne]

Here's an idea: in case that we have to distribute VAPOR with different capabilities among 3 major platforms, or even capabilities that are easy to go wrong within the camp of Linux for example, we provide a utility program that detects those variables and report to the user. It could be some added fields to "vaporversion".
The following capabilities can be subject to detection and report:
Whether ZFP (or any other) compression is supported
Whether OpenMP is supported
On tricky grids, whether GPU accelerated DVR is supported

Good idea, @shaomeng. Could you open an issue?

[shaomeng]

Good idea, @shaomeng. Could you open an issue?

Sure. #3142 is opened now.

[clyne]

@sgpearse we need seem clarification. Does the HDF5 plugin path need to be explicitly set on all platforms?

[sgpearse]

@sgpearse we need seem clarification. Does the HDF5 plugin path need to be explicitly set on all platforms?

HDF5_PLUGIN_PATH only needs to be set on Windows. It also needs to be set on our personal development machines because GetResourcePath() only works on installed versions of Vapor. It does not return /usr/local/VAPOR-Deps/2019-Aug during development builds.

[StasJ]

@sgpearse we need seem clarification. Does the HDF5 plugin path need to be explicitly set on all platforms?

HDF5_PLUGIN_PATH only needs to be set on Windows. It also needs to be set on our personal development machines because GetResourcePath() only works on installed versions of Vapor. It does not return /usr/local/VAPOR-Deps/2019-Aug during development builds.

GetResourcePath() does work on development builds. Otherwise many other features such as rendering maps wouldn't work.

[sgpearse]

@StasJ I see it now, thank you.

HDF5_PLUGIN_PATH is now only required on windows dev/installer builds.

[sgpearse]

@sgpearse will look into setting the Windows environment variable within the running executable.

[sgpearse]

@StasJ - Your notes should be addressed. All installers are working except for Windows, #3178. osgl seems to be the problem when we build with the gui. The broken windows build is reported #3178.

[StasJ]

@StasJ - Your notes should be addressed. All installers are working except for Windows, #3178. osgl seems to be the problem when we build with the gui. The broken windows build is reported #3178.

Thanks, looks great. There was one comment you didn't respond to:

This change requires lib-common to link against HDF5. It may be more appropriate for the VDC library to handle this since it handles the use of HDF5 files.

[sgpearse]

@StasJ - Ah, I did miss that. I thought lib common made sense because it handles our resource paths, but no biggie. Fixed.

[sgpearse]

Fixes #3179.

[sgpearse]

Fixes #3158.

[sgpearse]

Fixes #3174

[clyne]

Ready for @StasJ to review again

[clyne]

Fixes #2816

[clyne]

@sgpearse this issue has been closed. Presumably everyone should update their own development libraries to be in sync with the DevOps platforms, correct? Where are these libraries available?

thanks!

cc: @NihanthCW @shaomeng @StasJ

[sgpearse]

Presumably everyone should update their own development libraries to be in sync with the DevOps platforms, correct?

Yes.

Where are these libraries available?

Check out the PR description, which has links to bundles for each platform.

[clyne]

When I click on the link for Mac OSX I get:

Sorry, the file you have requested does not exist.

[sgpearse]

@clyne Broken link is updated on OSX here. PS - I've double checked that the links on CircleCI are targeting the correct libraries and not dead links.