Implement asynchronous/parallel transfers

[qubixes]

Currently iBridges relies on the parallelism in python-irodsclient for large transfers. This works well for upload/download speed in general, but there are some limitations:

• Multiple smaller files are not transferred in parallel.
• We can't update our progressbar while the transfer is taking place.
• For large files, the time to get the checksum can be very large. It would be nice if this could overlap with other transfers.
• Currently, we are limited in our ability to manage the timeouts (timeout on big files #197) that happen during transfers/checksum calculations. Performing the operations asynchronously might help with this.

If we want to do this properly and solve all the limitations, this would require quite a bit of work (and possibly help from the python-irodsclient). We could also consider solutions that would only resolve part of the limitations.

[trel]

Sounds like a client-side transfer manager. A queue of files-to-send/receive and a set of worker threads to do that work in parallel, and some knobs to control the number of workers / pool size / min/max.

Better progress indicators could be provided by PRC... happy to discuss.

[qubixes]

@trel Yes, that would be a client-side transfer manager. The alternative might be to just send all requests asynchronously, but with millions of files/data objects that might result in some issues (depending on the server configuration).

Thanks for being open to better progress indicators in the PRC! Let's discuss it when we know a little bit better what we actually need. I will also look into some detail how it could be integrated into PRC.

[trel]

The other reason a transfer manager is preferable over async requests.... is the ability for each worker to retry a number of times and/or to have a counter and just throw itself back in the queue instead of waiting for time to pass.

[qubixes]

@trel Yes, I agree.