Can DALI read from HDF5 file?

[jiankang1991]

Hi,
I would like to read images from pre-saved HDF5 file? With pytorch dataloader, e.g. I create the following data class and put it in dataloader.

class H5Dataset(Dataset):
    def __init__(self, h5_path):
        self.h5_path = h5_path
        self.h5_file = h5py.File(h5_path, 'r')
        self.length = len(h5py.File(h5_path, 'r'))
    
    def __getitem__(self, index):
        record = self.h5_file[str(index)]
        return (
            record['data'].value,
            record['target'].value,
        )
        
    def __len__(self):
        return self.length

Is it possible to use DALI to read HDF5, will it be faster than pytorch dataloader?

Thank you.

[awolant]

Hi, thanks for the question.
Currently there is no native support for HDF5 in DALI.
You can use ExternalSource to read your file using Python and feed it to DALI pipeline.
For better performance, you can try to implement your own reader using our plugin system.
If you are willing to contribute to DALI, we would be happy to accept PR with new op to read HDF5.

[zw0610]

any progress in this issue?

[JanuszL]

Hi, @zw0610,

We haven't got into it yet, nor has anyone from the community decided to contribute to this functionality.
You can read more about our 2021 plans here. We can consider this as an item for 2022.