Maintenance on benchmarks #1853
Assignees
Labels
Comments
Merged
wilwade
added a commit
that referenced
this issue
May 22, 2024
# Goal The goal of this PR is to fix an issue with running benchmarks and update our benchmarks with some tests. Part of #1853 ## Issues - When I added the Proxy pallet, the copied weight file worked, but the generated version didn't. - Then when fixing that, the benchmark runners had issues. So updated the benchmark workflow by removing it from the commit commenting system to a manual run. Using that will also automatically trigger verify CI actions. (GitHub actions only run on Manually triggered actions to prevent loops) # Discussion - The additional changes to the templates is from Polkadot-SDK - Actual change is small, as most of this are the generated weights code - Fix issue with the proxy_pallet weights reference - Update the benchmark running system to not use special Github tokens
aramikm
added a commit
that referenced
this issue
Aug 26, 2024
# Goal The goal of this PR is to calculate the DB benchmarks based on current main-net state. Related to #1853 # Warning - **Current PR will change all weights for stable weights** If that is not desirable, we can use defaults for stable weights but it would not be a correct metric for measuring consumed resources. # Discussion - Calculated the benchmark with `1.3` multiplier. If we expect to add more than 30% keys in the next 6 months we need to increase this multiplier. # Checklist - [X] Spec version incremented? # Execution steps 1. download a copy of mainnet chain data 2. build the same version of frequency that is running on mainnet by running `cargo build --release --features frequency,runtime-benchmarks` 3. run the storage benchmark by pointing the path to the copy of mainnet data like following `./target/release/frequency benchmark storage --state-version=1 --chain=frequency --base-path=/home/ubuntu/mainnet_data/ --include-child-trees --warmups=2 --mul=1.3` ```bash 2024-08-22 20:04:37 Warmup round 1/2 2024-08-22 20:09:03 Warmup round 2/2 2024-08-22 20:13:29 Preparing keys from block 0x09b2…00e2 2024-08-22 20:13:50 Reading 7585867 keys 2024-08-22 20:18:48 Reading 1705264 child keys 2024-08-22 20:19:57 Time summary [ns]: Total: 281753393895 Min: 1676, Max: 3967371 Average: 36369, Median: 36876, Stddev: 10368.83 Percentiles 99th, 95th, 75th: 60542, 52621, 43407 Value size summary: Total: 441863634 Min: 0, Max: 891362 Average: 57, Median: 8, Stddev: 363.51 Percentiles 99th, 95th, 75th: 1026, 253, 35 2024-08-22 20:20:18 Warmup round 1/2 2024-08-22 20:24:42 Warmup round 2/2 2024-08-22 20:29:06 Preparing keys from block 0x09b2…00e2 2024-08-22 20:29:29 Writing 7585867 keys 2024-08-22 20:53:04 Writing 1705264 child keys 2024-08-22 20:55:54 Time summary [ns]: Total: 443223318406 Min: 4954, Max: 87545132 Average: 57212, Median: 62141, Stddev: 60272.19 Percentiles 99th, 95th, 75th: 101820, 90948, 75800 Value size summary: Total: 441863634 Min: 0, Max: 891362 Average: 57, Median: 8, Stddev: 363.51 Percentiles 99th, 95th, 75th: 1026, 253, 35 2024-08-22 20:55:54 Writing weights to "/home/ubuntu/frequency/rocksdb_weights.rs" ```
|
Will close with #2131 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
It's time to do some maintenance on our benchmarks
Related: #1795
The text was updated successfully, but these errors were encountered: