Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optimize searcher performance significantly #1072

Merged
merged 5 commits into from
Apr 20, 2024

Conversation

liuchengxu
Copy link
Owner

This PR optimize the searcher performance even further, the hyperfine result indicates it's significantly improved by 60%. The series of optimization efforts have achieved 2x performance than v0.52.

hyperfine --warmup 3 './maple-v0.52 grep --ripgrep --cmd-dir /home/xlc/Data2/github.com/paritytech/polkadot-sdk/ client'  './maple-master grep --lib-ripgrep --cmd-dir /home/xlc/Data2/github.com/paritytech/polkadot-sdk/ client' './target/release/maple grep --lib-ripgrep --cmd-dir /home/xlc/Data2/github.com/paritytech/polkadot-sdk/ client'
Benchmark 1: ./maple-v0.52 grep --ripgrep --cmd-dir /home/xlc/Data2/github.com/paritytech/polkadot-sdk/ client
  Time (mean ± σ):     657.0 ms ±  19.9 ms    [User: 1242.2 ms, System: 461.2 ms]
  Range (min … max):   610.5 ms … 677.2 ms    10 runs
 
Benchmark 2: ./maple-master grep --lib-ripgrep --cmd-dir /home/xlc/Data2/github.com/paritytech/polkadot-sdk/ client
  Time (mean ± σ):     519.0 ms ±   9.2 ms    [User: 1014.1 ms, System: 354.4 ms]
  Range (min … max):   500.4 ms … 535.1 ms    10 runs
 
Benchmark 3: ./target/release/maple grep --lib-ripgrep --cmd-dir /home/xlc/Data2/github.com/paritytech/polkadot-sdk/ client
  Time (mean ± σ):     322.0 ms ±   3.6 ms    [User: 514.1 ms, System: 215.0 ms]
  Range (min … max):   317.4 ms … 326.4 ms    10 runs
 
Summary
  ./target/release/maple grep --lib-ripgrep --cmd-dir /home/xlc/Data2/github.com/paritytech/polkadot-sdk/ client ran
    1.61 ± 0.03 times faster than ./maple-master grep --lib-ripgrep --cmd-dir /home/xlc/Data2/github.com/paritytech/polkadot-sdk/ client
    2.04 ± 0.07 times faster than ./maple-v0.52 grep --ripgrep --cmd-dir /home/xlc/Data2/github.com/paritytech/polkadot-sdk/ client

The flamegraph shows the mpsc in tokio plays an role in the whole
process, I guess it's not efficient to pass the message ProcessedOne to
only deliver the info that one line has been processed, the hyperfine
indicates a significant performance increase in 60+%.
@romgrk
Copy link
Collaborator

romgrk commented Apr 19, 2024

Nice perf improvements, this and the previous one 👍

@liuchengxu liuchengxu merged commit d19e5bb into master Apr 20, 2024
15 checks passed
@liuchengxu liuchengxu deleted the optimize-searcher-further branch April 20, 2024 02:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants