write 5+ GiB (matrices) to S3Store #687
Conversation
…5+ GiB) Underlying library ``s3fs`` automatically writes objects to S3 in "chunks" or "parts" -- *i.e.* via multipart upload -- in line with S3's *minimum* limit for multipart of 5 MiB. This should, in general, avoid S3's *maximum* limit per (part) upload of 5 GiB. **However**, ``s3fs`` assumes that no *single* ``write()`` might exceed the maximum, and as such fails to chunk out such too-large upload requests prompted by singular writes of 5+ GiB. This can and should be resolved in ``s3fs``. But first, it can, should be and is resolved here in ``S3Store``. resolves #530
Codecov Report
@@ Coverage Diff @@
## master #687 +/- ##
==========================================
+ Coverage 82.18% 82.19% +<.01%
==========================================
Files 93 93
Lines 6271 6279 +8
==========================================
+ Hits 5154 5161 +7
- Misses 1117 1118 +1
Continue to review full report at Codecov.
|
|
A note on the 'further consideration' section: All good points: yes, that change was held back until post-release because it is a larger change and not necessarily needed before feature freeze. I'll also note, though, that in today's compressed-CSV world that the matrix writing is extremely unlikely to be the procedure that surfaces this 5GB problem. It's model writing. https://github.com/dssg/triage/blob/master/src/triage/component/catwalk/storage.py#L246 The way that we write models seems like it is not introducing the single-pass write problem, although I haven't looked at joblib's code so perhaps it is doing so. |
Ensure
S3Storedoes not attempt to write too-large chunks to S3 (5+ GiB)Changes
Underlying library
s3fsautomatically writes objects to S3 in "chunks"or "parts" -- i.e. via multipart upload -- in line with S3's minimum
limit for multipart of 5 MiB.
This should, in general, avoid S3's maximum limit per (part) upload of
5 GiB. However,
s3fsassumes that no singlewrite()mightexceed the maximum, and as such fails to chunk out such too-large upload
requests prompted by singular writes of 5+ GiB.
This can and should be resolved in
s3fs. But first, it can, shouldbe and is resolved here in
S3Store.resolves #530
Testing
In addition to the included unit test, a functional test script was composed – currently uploaded to Gist as test_store.py.
In its default, incremental mode, the functional test found no issue – writing to
s3fsin chunks under 5 GiB is no problem.On the other hand, the following command to test a single 6 GiB write…
$ AWS_PROFILE=dsapp-ddj time nice -n 3 \ python test_store.py --assert-type=S3Store --method=full -l 6GiB \ s3://review.dssg.io/test.6GiB.full.dat…resulted in:
…itself caused by:
Only with the included changeset, the above command succeeded in writing the object to S3, with the timing output:
(It certainly wasn't quick, but it was memory-stable. In the error case, RAM use exploded and threatened system stability – perhaps due to the payload being passed around and even copied by error handlers.)
Further consideration
It is important to cover this edge case and ensure that we can write 5+ GiB strings as necessary. However, it would be best (at least for memory stability) that we not attempt to, as possible.
A cursory look over the codebase brought this back to the fore, in
triage.component.catwalk.storage.CSVMatrixStore.save:I remember that it was attempted to write the above without storing entire payloads at once in memory, (e.g. to pass the file object to
gzipsuch that it could write to it iteratively according to standard buffering); however, this option was put aside, (likely for good reason).That said, we should seriously consider ways to maintain such standard buffering interactions, in lieu of working entirely in RAM, where possible, (at least to better avoid such edge cases and limitations).