refactor: revert the pre-serialization and parallel buffer

[MrCroxx]

What's changed and what's your intention?

Please explain IN DETAIL what the changes are in this PR and why they are needed. :D

Let's revert the pre-serialization and parallel buffer design to resolve the memory usage overhead.

Checklist

• I have written the necessary rustdoc comments
• I have added the necessary unit tests and integration tests
• I have passed make all (or make fast instead if the old tests are not modified) in my local environment.

Related issues or PRs (optional)

close #708

[MrCroxx]

Benchmark

command

cargo build --release --features jemalloc && PATH=/home/mrcroxx/.local/bin:$PATH RUST_BACKTRACE=1 RUST_LOG=info flamegraph -- ./target/release/foyer-bench --file /dev/nvme2n1 --mem 64MiB --disk 100GiB --region-size 64MiB --get-range 100000 --flushers 4 --reclaimers 4 --time 60 --writers 256 --w-rate 4MiB --admission-rate-limit 500MiB --readers 32 --r-rate 32MiB --metrics --warm-up 2 --runtime separated --user-runtime-worker-threads 8 --read-runtime-worker-threads 8 --write-runtime-worker-threads 4 --read-runtime-max-blocking-threads 8 --write-runtime-max-blocking-threads 4

main

Total:
disk total iops: 16007.2
disk total throughput: 1.5 GiB/s
disk read iops: 14971.7
disk read throughput: 1.0 GiB/s
disk write iops: 1035.5
disk write throughput: 498.8 MiB/s
insert iops: 15859.0/s
insert throughput: 1.0 GiB/s
insert lat p50: 4us
insert lat p90: 8us
insert lat p99: 14us
insert lat p999: 28us
insert lat p9999: 68us
insert lat p99999: 214us
insert lat pmax: 475us
get iops: 45936.5/s
get miss: 65.49% 
get throughput: 1.0 GiB/s
get hit lat p50: 299us
get hit lat p90: 627us
get hit lat p99: 2239us
get hit lat p999: 2815us
get hit lat p9999: 3167us
get hit lat p99999: 3519us
get hit lat pmax: 4063us
get miss lat p50: 15us
get miss lat p90: 30us
get miss lat p99: 68us
get miss lat p999: 213us
get miss lat p9999: 309us
get miss lat p99999: 461us
get miss lat pmax: 639us

PR

Total:
disk total iops: 22340.9
disk total throughput: 1.4 GiB/s
disk read iops: 14871.8
disk read throughput: 1.0 GiB/s
disk write iops: 7469.0
disk write throughput: 496.0 MiB/s
insert iops: 15858.9/s
insert throughput: 1.0 GiB/s
insert lat p50: 3us
insert lat p90: 6us
insert lat p99: 11us
insert lat p999: 23us
insert lat p9999: 59us
insert lat p99999: 263us
insert lat pmax: 507us
get iops: 49758.2/s
get miss: 68.14% 
get throughput: 1.0 GiB/s
get hit lat p50: 291us
get hit lat p90: 631us
get hit lat p99: 2191us
get hit lat p999: 2751us
get hit lat p9999: 3279us
get hit lat p99999: 5663us
get hit lat pmax: 6367us
get miss lat p50: 14us
get miss lat p90: 28us
get miss lat p99: 62us
get miss lat p999: 150us
get miss lat p9999: 303us
get miss lat p99999: 503us
get miss lat pmax: 1191us

[codecov]

Codecov Report

Attention: Patch coverage is 75.70621% with 43 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
foyer-storage/src/large/flusher.rs	68.75%	20 Missing ⚠️
foyer-storage/src/engine.rs	18.18%	9 Missing ⚠️
foyer-storage/src/serde.rs	75.00%	6 Missing ⚠️
foyer-storage/src/storage/either.rs	0.00%	4 Missing ⚠️
foyer-storage/src/large/batch.rs	95.23%	3 Missing ⚠️
foyer-storage/src/small/generic.rs	0.00%	1 Missing ⚠️

Files with missing lines	Coverage Δ
foyer-storage/src/large/generic.rs	87.77% <100.00%> (-0.30%)	⬇️
foyer-storage/src/storage/noop.rs	73.91% <100.00%> (-3.45%)	⬇️
foyer-storage/src/store.rs	73.12% <100.00%> (-0.51%)	⬇️
foyer-storage/src/small/generic.rs	0.00% <0.00%> (ø)
foyer-storage/src/large/batch.rs	83.20% <95.23%> (-0.57%)	⬇️
foyer-storage/src/storage/either.rs	0.00% <0.00%> (ø)
foyer-storage/src/serde.rs	84.21% <75.00%> (-12.22%)	⬇️
foyer-storage/src/engine.rs	34.45% <18.18%> (-1.20%)	⬇️
foyer-storage/src/large/flusher.rs	85.14% <68.75%> (-10.18%)	⬇️

... and 4 files with indirect coverage changes

[hzxa21]

The disk write iops increases from 1035.5 to 7469.0 with similar disk throughput and insert iops. Is this expected?

[MrCroxx]

The disk write iops increases from 1035.5 to 7469.0 with similar disk throughput and insert iops. Is this expected?

Yes. The disk hit ratio also increased. I'm still checking the reason.

[MrCroxx]

Discussed with @st1page offline. Let's merge it.