[deepcam][data loading] Use of NVIDIA DALI with Deepcam

[BTMichalowicz]

Hi! I'm currently fiddling around with the deepcam benchmark and plan on using NVIDIA DALI on the deepcam dataset. Right now, I'm using this script to convert the .h5 files to .npy, as DALI does not currently support HDF5 (see NVIDIA/DALI#1252).

Are there any better/more efficient ways to deal with this? Thank you in advance.

[azrael417]

Hello,

A conversion script is here

https://github.com/mlcommons/hpc_results_v1.0/blob/master/NVIDIA/benchmarks/deepcam/implementations/pytorch/src/utils/convert_hdf52npy.py

And the corresponding Dali pipeline (along with an overall optimized benchmark) is here:

https://github.com/mlcommons/hpc_results_v1.0/tree/master/NVIDIA/benchmarks/deepcam/implementations/pytorch/src/deepCam/data

There are a few variants for numpy datasets, with and without manual caching. Just pick whatever works best for you.

Best
Thorsten

[BTMichalowicz]

Awesome. Thank you so much!