CSVs take up too much space

[jesteria]

HDF matrix data must be present on the local file system. However, as an experimenter, I might like to be able to specify an S3 file path, with the expectation that this data will be downloaded from and uploaded to S3 as necessary, on my behalf.

[thcrock]

After talking with @nanounanue we decided to repurpose this: CSVs should be compressed by default. The current situation makes us want to use HDFs all the time even when we won't gain anything from them. Compressing CSVs would make it work for more use cases and make HDF-S3 support unnecessary

[jesteria]

👍 👍

I'm not positive that compressed CSV will make this 100% unnecessary, (but yeah certainly might!). Regardless, it shouldn't be a ton of work to read and write CSVs with GZIP compression, by default, and I absolutely agree that that should be done.

We're just talking about something like:

with gzip.open(filename) as fd:
    reader_or_writer = csv.reader_or_writer(fd)

…Which, with S3Fs for example, just becomes:

with s3.open('mybucket/my-file.csv.gz', 'rb') as fd_compressed:
    fd = gzip.GzipFile(fileobj=fd_compressed)
    …

(Simply taken from the S3Fs docs.)