cp: fix possible OOM and partial write with large files #6694
Conversation
src/uu/cp/src/platform/linux.rs
Outdated
| @@ -141,6 +141,8 @@ where | |||
| } | |||
| let src_fd = src_file.as_raw_fd(); | |||
| let mut current_offset: isize = 0; | |||
| let step = std::cmp::min(size, 16 * 1024 * 1024) as usize; // 16 MiB | |||
There was a problem hiding this comment.
please add a comment to explain why you are doing this
| current_offset.try_into().unwrap(), | ||
| ) | ||
| }; | ||
| for i in (0..len).step_by(step) { |
There was a problem hiding this comment.
same, please document this a bit more :) (i know it wasn't documented before :)
|
thanks |
|
More docs :) I have "benchmarked" it with my large file (technically not sparse, but heuristic triggered it I suppose). |
|
any idea of the impact on slow harddrive ? |
|
Well, if the copy is between 2 HDDs, then it shouldnt matter. 16MB seems a lot, it should balance out anything with the FS help (just my opinion) |
|
GNU testsuite comparison: |
|
GNU testsuite comparison: |
|
GNU testsuite comparison: |
There are 2 fixes:
pwritedoes not guarantee that all bytes will be written,write_all_atis a proper wrapper over itsparse_copy_without_hole(triggered on ZFS) can eat RAM if file is large and there are not enough holes; now the write happens at max 16 MiB chunks (as far as I tested, it saturates SSD anyways without much CPU)I have tested it with checking sha256sum of a 4 GiB file.