fix: make add_piece taking less time #1707
Conversation
`add_piece` operates in 64 bytes pieces. It's faster to operate in bigger chunks. This commits increases the buffer size to 4KiB. This makes adding a 32GiB piece about more than 2x faster. On the hardware I used it goes down from 14min to about 6min.
| @@ -7,11 +7,13 @@ use rayon::prelude::{ParallelIterator, ParallelSlice}; | |||
|
|
|||
| use crate::{constants::DefaultPieceHasher, pieces::piece_hash}; | |||
|
|
|||
| const BUFFER_SIZE: usize = 4096; | |||
There was a problem hiding this comment.
Have you tried bumping it even further, for example, up to 1MiB-8MiB?
Even with the 4KiB, it is just 64 SHA invocations, the branch predictor is probably just warming up at that point.
There was a problem hiding this comment.
I've tried 8MiB it didn't make a difference.
I think yet another difference would make it if the hashing would be done in parallel. But a simple "par chunker" didn't really help, as then the pieces are to small. One need to implement it manually. I decided that it's not worth it and that this improvement is already good enough for how simple it is.
|
Can we check if lotus is using this method? I recall it not being used, but may be wrong there. In any case, it should help speed up our big tests. |
Why is that important? It's not a breaking change, it just improves things. So if Lotus would use it, it would also be beneficial for them. |
I just wanted to know if we'd see the speed-ups outside of tests. |
add_pieceoperates in 64 bytes pieces. It's faster to operate in bigger chunks. This commits increases the buffer size to 4KiB. This makes adding a 32GiB piece about more than 2x faster. On the hardware I used it goes down from 14min to about 6min.