DataLoader with num_workers > 0 increases memory consumption over time

[juancq]

Using the dev branch, I have noticed that when using DataLoader with num_workers greater than 0, the memory consumption increases over time during a single epoch. This problem has been documented here:
https://docs.aws.amazon.com/codeguru/detector-library/python/pytorch-data-loader-with-multiple-workers/
pytorch/pytorch#13246 (comment)

[mmcdermott]

@juancq I just wanted to give you a heads up that what was formerly dev was merged into main and the new code in the dev branch has some changes to this from #74 that will be pulled out or re-worked to address the speed issues you caught, but dev itself now has that code in it. Apologies for any confusion.

[mmcdermott]

@juancq I believe there is a fix now for this and all related issues in #90. If you want to test it, though, right now it requires running the code in the branch in that PR and manually cloning then installing (via pip install -e .) this package: https://github.com/mmcdermott/nested_ragged_tensors as well which handles the manipulation of the ragged tensors that are used to speed things up and reduce memory costs here. This still caches files, so you may need to delete any previously cached files from the old version, but the cached files should be dramatically smaller and the actual runtime should have no memory leaks and be (at worst) competitive with the prior runtime and likely (on some tasks/settings) faster than the prior runtime on a iteration / batch level. No pressure to test for now; I'm going to push the other package to pypi so it can be installed normally and make a few other cosmetic improvements, but that's the state of things for your information.