Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[WIP] Test out running benchmarks #597

Draft
wants to merge 4 commits into
base: main
Choose a base branch
from
Draft

[WIP] Test out running benchmarks #597

wants to merge 4 commits into from

Conversation

bantonsson
Copy link
Contributor

No description provided.

@pr-commenter
Copy link

pr-commenter bot commented Aug 26, 2024 •
edited
Loading

Benchmarks

Comparison

Benchmark execution time: 2024-09-18 12:59:05

Comparing candidate commit 66cc801 in PR branch ban/tmp-test-benchmarks-bs with baseline commit 89f48d4 in branch main.

Found 7 performance improvements and 31 performance regressions! Performance is the same for 13 metrics, 2 unstable metrics.

scenario:benching deserializing traces from msgpack to their internal representation

  • 🟩 execution_time [-190.491ns; -175.329ns] or [-13.779%; -12.682%]

scenario:benching string interning on wordpress profile

  • 🟩 execution_time [-4.748µs; -4.492µs] or [-3.300%; -3.122%]

scenario:credit_card/is_card_number/

  • 🟥 execution_time [+4.839µs; +4.840µs] or [+400.197%; +400.251%]

scenario:credit_card/is_card_number/ 3782-8224-6310-005

  • 🟥 execution_time [+477.572µs; +478.167µs] or [+430.852%; +431.389%]
  • 🟥 throughput [-538682.654op/s; -519222.516op/s] or [-5.971%; -5.755%]

scenario:credit_card/is_card_number/ 378282246310005

  • 🟥 execution_time [+450.876µs; +452.032µs] or [+435.958%; +437.076%]
  • 🟥 throughput [-672005.809op/s; -644144.515op/s] or [-6.950%; -6.661%]

scenario:credit_card/is_card_number/37828224631

  • 🟥 execution_time [+4.840µs; +4.840µs] or [+400.234%; +400.276%]

scenario:credit_card/is_card_number/378282246310005

  • 🟥 execution_time [+434.669µs; +435.718µs] or [+434.351%; +435.399%]
  • 🟥 throughput [-664630.746op/s; -638279.565op/s] or [-6.651%; -6.387%]

scenario:credit_card/is_card_number/37828224631000521389798

  • 🟥 execution_time [+412.504µs; +412.803µs] or [+431.473%; +431.785%]
  • 🟥 throughput [-633621.615op/s; -612034.863op/s] or [-6.057%; -5.851%]

scenario:credit_card/is_card_number/x371413321323331

  • 🟥 execution_time [+85.216µs; +85.733µs] or [+399.510%; +401.935%]

scenario:credit_card/is_card_number_no_luhn/

  • 🟥 execution_time [+4.841µs; +4.841µs] or [+400.507%; +400.543%]

scenario:credit_card/is_card_number_no_luhn/ 3782-8224-6310-005

  • 🟥 execution_time [+367.169µs; +367.772µs] or [+383.006%; +383.635%]
  • 🟩 throughput [+347924.549op/s; +371650.698op/s] or [+3.335%; +3.563%]

scenario:credit_card/is_card_number_no_luhn/ 378282246310005

  • 🟥 execution_time [+343.217µs; +343.843µs] or [+394.410%; +395.130%]

scenario:credit_card/is_card_number_no_luhn/37828224631

  • 🟥 execution_time [+4.843µs; +4.844µs] or [+400.710%; +400.865%]

scenario:credit_card/is_card_number_no_luhn/378282246310005

  • 🟥 execution_time [+328.882µs; +329.596µs] or [+396.892%; +397.753%]

scenario:credit_card/is_card_number_no_luhn/37828224631000521389798

  • 🟥 execution_time [+413.148µs; +413.503µs] or [+433.945%; +434.318%]
  • 🟥 throughput [-686374.613op/s; -657873.157op/s] or [-6.534%; -6.263%]

scenario:credit_card/is_card_number_no_luhn/x371413321323331

  • 🟥 execution_time [+85.189µs; +85.746µs] or [+399.928%; +402.545%]

scenario:normalization/normalize_name/normalize_name/Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Lo...

  • 🟥 execution_time [+1.137ms; +1.138ms] or [+416.423%; +416.709%]
  • 🟥 throughput [-120605.182op/s; -114561.851op/s] or [-3.292%; -3.127%]

scenario:normalization/normalize_name/normalize_name/bad-name

  • 🟥 execution_time [+112.573µs; +112.669µs] or [+435.965%; +436.339%]
  • 🟥 throughput [-2624986.092op/s; -2597217.733op/s] or [-6.778%; -6.706%]

scenario:normalization/normalize_name/normalize_name/good

  • 🟥 execution_time [+68.087µs; +68.146µs] or [+443.896%; +444.278%]
  • 🟥 throughput [-5308694.975op/s; -5256241.932op/s] or [-8.143%; -8.062%]

scenario:normalization/normalize_service/normalize_service/A0000000000000000000000000000000000000000000000000...

  • 🟥 execution_time [+2.032ms; +2.034ms] or [+400.455%; +400.844%]

scenario:normalization/normalize_service/normalize_service/Data🐨dog🐶 繋がっ⛰てて

  • 🟥 execution_time [+1.849ms; +1.850ms] or [+397.000%; +397.114%]

scenario:normalization/normalize_service/normalize_service/Test Conversion 0f Weird !@#$%^&**() Characters

  • 🟥 execution_time [+698.395µs; +698.625µs] or [+387.242%; +387.369%]
  • 🟩 throughput [+142805.499op/s; +146047.041op/s] or [+2.576%; +2.634%]

scenario:normalization/normalize_service/normalize_service/[empty string]

  • 🟥 execution_time [+172.503µs; +172.615µs] or [+392.834%; +393.088%]

scenario:normalization/normalize_service/normalize_service/test_ASCII

  • 🟥 execution_time [+157.034µs; +157.105µs] or [+313.308%; +313.451%]
  • 🟩 throughput [+4172365.109op/s; +4188844.616op/s] or [+20.912%; +20.995%]

scenario:normalization/normalize_trace/test_trace

  • 🟥 execution_time [+6.388ns; +9.975ns] or [+2.110%; +3.294%]

scenario:sql/obfuscate_sql_string

  • 🟩 execution_time [-5.573µs; -5.518µs] or [-7.472%; -7.397%]

scenario:tags/replace_trace_tags

  • 🟩 execution_time [-61.313ns; -55.508ns] or [-2.291%; -2.074%]

Candidate

Candidate benchmark details

Group 1

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 66cc801 1726663655 ban/tmp-test-benchmarks-bs
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
credit_card/is_card_number/ execution_time 6.044µs 6.049µs ± 0.002µs 6.049µs ± 0.001µs 6.050µs 6.052µs 6.054µs 6.059µs 0.17% 0.798 2.357 0.03% 0.000µs 1 250
credit_card/is_card_number/ throughput 825213761.439op/s 826618016.425op/s ± 283221.847op/s 826617290.564op/s ± 177270.697op/s 826810901.821op/s 827024167.606op/s 827170550.047op/s 827290556.401op/s 0.08% -0.795 2.342 0.03% 17912.522op/s 1 250
credit_card/is_card_number/ 3782-8224-6310-005 execution_time 582.777µs 588.713µs ± 2.232µs 588.916µs ± 1.320µs 590.097µs 592.140µs 593.474µs 595.854µs 1.18% -0.141 0.456 0.38% 0.141µs 1 250
credit_card/is_card_number/ 3782-8224-6310-005 throughput 8391317.336op/s 8493225.353op/s ± 32217.786op/s 8490179.841op/s ± 19030.115op/s 8513206.622op/s 8549231.130op/s 8573706.852op/s 8579605.285op/s 1.05% 0.168 0.447 0.38% 2037.632op/s 1 250
credit_card/is_card_number/ 378282246310005 execution_time 541.218µs 554.876µs ± 4.577µs 555.735µs ± 3.090µs 558.469µs 561.123µs 561.973µs 562.755µs 1.26% -0.690 -0.019 0.82% 0.289µs 1 250
credit_card/is_card_number/ 378282246310005 throughput 8884866.529op/s 9011632.149op/s ± 74766.722op/s 8997095.613op/s ± 49786.348op/s 9053273.446op/s 9156354.086op/s 9212904.286op/s 9238425.018op/s 2.68% 0.727 0.056 0.83% 4728.663op/s 1 250
credit_card/is_card_number/37828224631 execution_time 6.045µs 6.049µs ± 0.002µs 6.049µs ± 0.001µs 6.050µs 6.052µs 6.053µs 6.054µs 0.08% 0.304 0.094 0.03% 0.000µs 1 250
credit_card/is_card_number/37828224631 throughput 825906196.526op/s 826529870.272op/s ± 212401.111op/s 826536796.680op/s ± 139799.327op/s 826678843.679op/s 826859768.067op/s 826982691.223op/s 827093286.625op/s 0.07% -0.302 0.093 0.03% 13433.426op/s 1 250
credit_card/is_card_number/378282246310005 execution_time 522.612µs 535.267µs ± 4.157µs 535.880µs ± 2.479µs 538.126µs 541.045µs 542.217µs 546.605µs 2.00% -0.665 0.371 0.78% 0.263µs 1 250
credit_card/is_card_number/378282246310005 throughput 9147369.553op/s 9341697.502op/s ± 72943.883op/s 9330448.522op/s ± 43169.858op/s 9378476.469op/s 9478178.293op/s 9547240.444op/s 9567323.652op/s 2.54% 0.710 0.440 0.78% 4613.376op/s 1 250
credit_card/is_card_number/37828224631000521389798 execution_time 505.981µs 508.257µs ± 0.912µs 508.125µs ± 0.547µs 508.769µs 509.845µs 511.003µs 511.214µs 0.61% 0.722 0.747 0.18% 0.058µs 1 250
credit_card/is_card_number/37828224631000521389798 throughput 9780632.038op/s 9837568.039op/s ± 17637.501op/s 9840104.760op/s ± 10596.777op/s 9849650.018op/s 9863084.332op/s 9868989.825op/s 9881798.253op/s 0.42% -0.710 0.725 0.18% 1115.493op/s 1 250
credit_card/is_card_number/x371413321323331 execution_time 102.846µs 106.805µs ± 2.052µs 106.625µs ± 1.391µs 108.096µs 110.621µs 111.561µs 112.190µs 5.22% 0.417 -0.423 1.92% 0.130µs 1 250
credit_card/is_card_number/x371413321323331 throughput 44567365.861op/s 46831448.727op/s ± 893265.425op/s 46893216.724op/s ± 613723.389op/s 47436905.307op/s 48248521.026op/s 48467506.979op/s 48616251.830op/s 3.67% -0.337 -0.488 1.90% 56495.066op/s 1 250
credit_card/is_card_number_no_luhn/ execution_time 6.047µs 6.050µs ± 0.001µs 6.050µs ± 0.001µs 6.051µs 6.052µs 6.054µs 6.055µs 0.09% 0.575 0.566 0.02% 0.000µs 1 250
credit_card/is_card_number_no_luhn/ throughput 825756727.547op/s 826479487.203op/s ± 195482.214op/s 826498250.416op/s ± 115824.950op/s 826605300.495op/s 826769451.911op/s 826841423.007op/s 826904023.545op/s 0.05% -0.573 0.563 0.02% 12363.381op/s 1 250
credit_card/is_card_number_no_luhn/ 3782-8224-6310-005 execution_time 458.158µs 463.335µs ± 2.325µs 462.939µs ± 1.452µs 464.654µs 467.575µs 469.556µs 471.099µs 1.76% 0.633 0.162 0.50% 0.147µs 1 250
credit_card/is_card_number_no_luhn/ 3782-8224-6310-005 throughput 10613474.600op/s 10791592.539op/s ± 53982.226op/s 10800548.376op/s ± 33786.776op/s 10831456.006op/s 10864562.865op/s 10888185.187op/s 10913255.521op/s 1.04% -0.607 0.121 0.50% 3414.136op/s 1 250
credit_card/is_card_number_no_luhn/ 378282246310005 execution_time 425.875µs 430.550µs ± 2.427µs 430.562µs ± 1.866µs 432.270µs 434.862µs 435.933µs 437.159µs 1.53% 0.316 -0.626 0.56% 0.153µs 1 250
credit_card/is_card_number_no_luhn/ 378282246310005 throughput 11437486.821op/s 11613425.940op/s ± 65352.614op/s 11612742.545op/s ± 50352.510op/s 11667744.045op/s 11707686.334op/s 11722258.699op/s 11740541.449op/s 1.10% -0.295 -0.650 0.56% 4133.262op/s 1 250
credit_card/is_card_number_no_luhn/37828224631 execution_time 6.045µs 6.052µs ± 0.007µs 6.049µs ± 0.002µs 6.051µs 6.069µs 6.071µs 6.074µs 0.40% 1.641 1.125 0.12% 0.000µs 1 250
credit_card/is_card_number_no_luhn/37828224631 throughput 823246524.121op/s 826174539.231op/s ± 1019156.186op/s 826553666.533op/s ± 230705.609op/s 826747486.696op/s 826987908.299op/s 827108155.116op/s 827181747.024op/s 0.08% -1.639 1.121 0.12% 64457.097op/s 1 250
credit_card/is_card_number_no_luhn/378282246310005 execution_time 407.946µs 412.103µs ± 2.757µs 410.981µs ± 1.851µs 414.496µs 416.721µs 418.308µs 419.239µs 2.01% 0.497 -1.014 0.67% 0.174µs 1 250
credit_card/is_card_number_no_luhn/378282246310005 throughput 11926364.060op/s 12133419.616op/s ± 80914.161op/s 12166013.717op/s ± 55029.227op/s 12200579.975op/s 12229728.778op/s 12246643.425op/s 12256517.985op/s 0.74% -0.483 -1.038 0.67% 5117.461op/s 1 250
credit_card/is_card_number_no_luhn/37828224631000521389798 execution_time 506.306µs 508.533µs ± 0.983µs 508.272µs ± 0.503µs 509.021µs 510.394µs 511.531µs 511.868µs 0.71% 1.025 1.117 0.19% 0.062µs 1 250
credit_card/is_card_number_no_luhn/37828224631000521389798 throughput 9768152.295op/s 9832242.617op/s ± 18977.963op/s 9837251.285op/s ± 9738.192op/s 9844680.628op/s 9855995.772op/s 9865476.340op/s 9875447.157op/s 0.39% -1.013 1.089 0.19% 1200.272op/s 1 250
credit_card/is_card_number_no_luhn/x371413321323331 execution_time 102.493µs 106.769µs ± 2.218µs 106.833µs ± 1.538µs 108.173µs 110.687µs 111.760µs 111.907µs 4.75% 0.152 -0.409 2.07% 0.140µs 1 250
credit_card/is_card_number_no_luhn/x371413321323331 throughput 44679840.349op/s 46850329.822op/s ± 971015.585op/s 46802004.281op/s ± 666400.776op/s 47586968.611op/s 48599809.389op/s 48772673.393op/s 48784006.599op/s 4.23% -0.055 -0.450 2.07% 61412.418op/s 1 250
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
credit_card/is_card_number/ execution_time [6.048µs; 6.049µs] or [-0.004%; +0.004%] None None None
credit_card/is_card_number/ throughput [826582908.526op/s; 826653124.324op/s] or [-0.004%; +0.004%] None None None
credit_card/is_card_number/ 3782-8224-6310-005 execution_time [588.436µs; 588.990µs] or [-0.047%; +0.047%] None None None
credit_card/is_card_number/ 3782-8224-6310-005 throughput [8489231.668op/s; 8497219.037op/s] or [-0.047%; +0.047%] None None None
credit_card/is_card_number/ 378282246310005 execution_time [554.309µs; 555.444µs] or [-0.102%; +0.102%] None None None
credit_card/is_card_number/ 378282246310005 throughput [9002364.140op/s; 9020900.158op/s] or [-0.103%; +0.103%] None None None
credit_card/is_card_number/37828224631 execution_time [6.049µs; 6.050µs] or [-0.003%; +0.003%] None None None
credit_card/is_card_number/37828224631 throughput [826503541.241op/s; 826556199.302op/s] or [-0.003%; +0.003%] None None None
credit_card/is_card_number/378282246310005 execution_time [534.752µs; 535.782µs] or [-0.096%; +0.096%] None None None
credit_card/is_card_number/378282246310005 throughput [9332655.450op/s; 9350739.553op/s] or [-0.097%; +0.097%] None None None
credit_card/is_card_number/37828224631000521389798 execution_time [508.144µs; 508.370µs] or [-0.022%; +0.022%] None None None
credit_card/is_card_number/37828224631000521389798 throughput [9835381.712op/s; 9839754.366op/s] or [-0.022%; +0.022%] None None None
credit_card/is_card_number/x371413321323331 execution_time [106.550µs; 107.059µs] or [-0.238%; +0.238%] None None None
credit_card/is_card_number/x371413321323331 throughput [46720720.433op/s; 46942177.022op/s] or [-0.236%; +0.236%] None None None
credit_card/is_card_number_no_luhn/ execution_time [6.050µs; 6.050µs] or [-0.003%; +0.003%] None None None
credit_card/is_card_number_no_luhn/ throughput [826455255.422op/s; 826503718.984op/s] or [-0.003%; +0.003%] None None None
credit_card/is_card_number_no_luhn/ 3782-8224-6310-005 execution_time [463.047µs; 463.623µs] or [-0.062%; +0.062%] None None None
credit_card/is_card_number_no_luhn/ 3782-8224-6310-005 throughput [10784900.956op/s; 10798284.122op/s] or [-0.062%; +0.062%] None None None
credit_card/is_card_number_no_luhn/ 378282246310005 execution_time [430.249µs; 430.851µs] or [-0.070%; +0.070%] None None None
credit_card/is_card_number_no_luhn/ 378282246310005 throughput [11605324.895op/s; 11621526.985op/s] or [-0.070%; +0.070%] None None None
credit_card/is_card_number_no_luhn/37828224631 execution_time [6.051µs; 6.053µs] or [-0.015%; +0.015%] None None None
credit_card/is_card_number_no_luhn/37828224631 throughput [826048205.643op/s; 826300872.820op/s] or [-0.015%; +0.015%] None None None
credit_card/is_card_number_no_luhn/378282246310005 execution_time [411.762µs; 412.445µs] or [-0.083%; +0.083%] None None None
credit_card/is_card_number_no_luhn/378282246310005 throughput [12123389.577op/s; 12143449.655op/s] or [-0.083%; +0.083%] None None None
credit_card/is_card_number_no_luhn/37828224631000521389798 execution_time [508.411µs; 508.655µs] or [-0.024%; +0.024%] None None None
credit_card/is_card_number_no_luhn/37828224631000521389798 throughput [9829890.128op/s; 9834595.107op/s] or [-0.024%; +0.024%] None None None
credit_card/is_card_number_no_luhn/x371413321323331 execution_time [106.494µs; 107.043µs] or [-0.257%; +0.257%] None None None
credit_card/is_card_number_no_luhn/x371413321323331 throughput [46729963.695op/s; 46970695.949op/s] or [-0.257%; +0.257%] None None None

Group 2

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 66cc801 1726663655 ban/tmp-test-benchmarks-bs
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
benching deserializing traces from msgpack to their internal representation execution_time 1.197µs 1.200µs ± 0.001µs 1.200µs ± 0.000µs 1.200µs 1.201µs 1.205µs 1.208µs 0.67% 3.204 17.365 0.10% 0.000µs 1 250
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
benching deserializing traces from msgpack to their internal representation execution_time [1.199µs; 1.200µs] or [-0.012%; +0.012%] None None None

Group 3

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 66cc801 1726663655 ban/tmp-test-benchmarks-bs
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
concentrator/add_spans_to_concentrator execution_time 8.979ms 9.017ms ± 0.026ms 9.015ms ± 0.007ms 9.023ms 9.035ms 9.052ms 9.343ms 3.64% 9.106 107.689 0.28% 0.002ms 1 250
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
concentrator/add_spans_to_concentrator execution_time [9.014ms; 9.020ms] or [-0.035%; +0.035%] None None None

Group 4

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 66cc801 1726663655 ban/tmp-test-benchmarks-bs
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
normalization/normalize_service/normalize_service/A0000000000000000000000000000000000000000000000000... execution_time 2.537ms 2.541ms ± 0.008ms 2.539ms ± 0.001ms 2.540ms 2.544ms 2.568ms 2.622ms 3.25% 7.581 65.683 0.31% 0.001ms 1 250
normalization/normalize_service/normalize_service/A0000000000000000000000000000000000000000000000000... throughput 1907142.503op/s 1968000.880op/s ± 6009.385op/s 1969068.106op/s ± 602.586op/s 1969589.408op/s 1970334.901op/s 1970760.332op/s 1970854.608op/s 0.09% -7.487 64.170 0.30% 380.067op/s 1 250
normalization/normalize_service/normalize_service/Data🐨dog🐶 繋がっ⛰てて execution_time 2.309ms 2.315ms ± 0.002ms 2.315ms ± 0.001ms 2.317ms 2.318ms 2.319ms 2.328ms 0.54% 0.553 5.488 0.09% 0.000ms 1 250
normalization/normalize_service/normalize_service/Data🐨dog🐶 繋がっ⛰てて throughput 2147711.738op/s 2159512.370op/s ± 1920.719op/s 2159411.888op/s ± 1194.049op/s 2160764.829op/s 2162506.177op/s 2164697.830op/s 2165530.713op/s 0.28% -0.534 5.392 0.09% 121.477op/s 1 250
normalization/normalize_service/normalize_service/Test Conversion 0f Weird !@#$%^&**() Characters execution_time 877.111µs 878.861µs ± 0.844µs 878.822µs ± 0.421µs 879.208µs 880.213µs 881.103µs 884.673µs 0.67% 1.620 8.505 0.10% 0.053µs 1 250
normalization/normalize_service/normalize_service/Test Conversion 0f Weird !@#$%^&**() Characters throughput 5651805.808op/s 5689185.374op/s ± 5454.286op/s 5689434.266op/s ± 2725.033op/s 5692558.906op/s 5696782.096op/s 5699447.940op/s 5700532.272op/s 0.20% -1.597 8.334 0.10% 344.959op/s 1 250
normalization/normalize_service/normalize_service/[empty string] execution_time 215.054µs 216.471µs ± 0.440µs 216.477µs ± 0.279µs 216.746µs 217.152µs 217.534µs 217.785µs 0.60% -0.089 0.420 0.20% 0.028µs 1 250
normalization/normalize_service/normalize_service/[empty string] throughput 22958450.409op/s 23097835.073op/s ± 47008.845op/s 23097112.323op/s ± 29812.351op/s 23126947.257op/s 23175177.128op/s 23212814.291op/s 23249998.450op/s 0.66% 0.104 0.426 0.20% 2973.100op/s 1 250
normalization/normalize_service/normalize_service/test_ASCII execution_time 206.821µs 207.191µs ± 0.244µs 207.141µs ± 0.147µs 207.334µs 207.562µs 207.873µs 208.828µs 0.81% 1.770 7.531 0.12% 0.015µs 1 250
normalization/normalize_service/normalize_service/test_ASCII throughput 23943172.317op/s 24132388.705op/s ± 28418.495op/s 24138196.001op/s ± 17106.799op/s 24151416.825op/s 24168673.572op/s 24172721.250op/s 24175486.959op/s 0.15% -1.748 7.351 0.12% 1797.343op/s 1 250
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
normalization/normalize_service/normalize_service/A0000000000000000000000000000000000000000000000000... execution_time [2.540ms; 2.542ms] or [-0.039%; +0.039%] None None None
normalization/normalize_service/normalize_service/A0000000000000000000000000000000000000000000000000... throughput [1967255.963op/s; 1968745.797op/s] or [-0.038%; +0.038%] None None None
normalization/normalize_service/normalize_service/Data🐨dog🐶 繋がっ⛰てて execution_time [2.315ms; 2.316ms] or [-0.011%; +0.011%] None None None
normalization/normalize_service/normalize_service/Data🐨dog🐶 繋がっ⛰てて throughput [2159274.280op/s; 2159750.460op/s] or [-0.011%; +0.011%] None None None
normalization/normalize_service/normalize_service/Test Conversion 0f Weird !@#$%^&**() Characters execution_time [878.757µs; 878.966µs] or [-0.012%; +0.012%] None None None
normalization/normalize_service/normalize_service/Test Conversion 0f Weird !@#$%^&**() Characters throughput [5688509.266op/s; 5689861.482op/s] or [-0.012%; +0.012%] None None None
normalization/normalize_service/normalize_service/[empty string] execution_time [216.417µs; 216.526µs] or [-0.025%; +0.025%] None None None
normalization/normalize_service/normalize_service/[empty string] throughput [23092007.903op/s; 23103662.243op/s] or [-0.025%; +0.025%] None None None
normalization/normalize_service/normalize_service/test_ASCII execution_time [207.160µs; 207.221µs] or [-0.015%; +0.015%] None None None
normalization/normalize_service/normalize_service/test_ASCII throughput [24128865.977op/s; 24135911.434op/s] or [-0.015%; +0.015%] None None None

Group 5

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 66cc801 1726663655 ban/tmp-test-benchmarks-bs
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
redis/obfuscate_redis_string execution_time 37.370µs 37.873µs ± 0.847µs 37.465µs ± 0.030µs 37.527µs 39.635µs 40.076µs 40.803µs 8.91% 1.619 0.845 2.23% 0.054µs 1 250
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
redis/obfuscate_redis_string execution_time [37.768µs; 37.977µs] or [-0.277%; +0.277%] None None None

Group 6

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 66cc801 1726663655 ban/tmp-test-benchmarks-bs
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
normalization/normalize_trace/test_trace execution_time 309.494ns 310.960ns ± 0.912ns 310.788ns ± 0.280ns 311.133ns 311.793ns 313.822ns 321.393ns 3.41% 6.879 69.496 0.29% 0.058ns 1 250
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
normalization/normalize_trace/test_trace execution_time [310.847ns; 311.073ns] or [-0.036%; +0.036%] None None None

Group 7

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 66cc801 1726663655 ban/tmp-test-benchmarks-bs
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
normalization/normalize_name/normalize_name/Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Lo... execution_time 1.403ms 1.410ms ± 0.003ms 1.410ms ± 0.001ms 1.411ms 1.414ms 1.418ms 1.431ms 1.47% 2.211 17.083 0.19% 0.000ms 1 250
normalization/normalize_name/normalize_name/Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Lo... throughput 3494171.877op/s 3545631.669op/s ± 6555.820op/s 3545529.184op/s ± 2557.703op/s 3548276.770op/s 3554212.874op/s 3562235.315op/s 3563264.736op/s 0.50% -2.133 16.454 0.18% 414.626op/s 1 250
normalization/normalize_name/normalize_name/bad-name execution_time 137.614µs 138.443µs ± 0.388µs 138.411µs ± 0.283µs 138.700µs 139.107µs 139.415µs 139.593µs 0.85% 0.340 -0.174 0.28% 0.025µs 1 250
normalization/normalize_name/normalize_name/bad-name throughput 35818539.372op/s 36116349.421op/s ± 101170.803op/s 36124189.721op/s ± 73881.008op/s 36194753.439op/s 36271124.040op/s 36315762.970op/s 36333624.590op/s 0.58% -0.325 -0.193 0.28% 6398.603op/s 1 250
normalization/normalize_name/normalize_name/good execution_time 82.878µs 83.455µs ± 0.234µs 83.391µs ± 0.108µs 83.533µs 83.927µs 84.110µs 84.332µs 1.13% 1.165 1.039 0.28% 0.015µs 1 250
normalization/normalize_name/normalize_name/good throughput 59289531.936op/s 59912909.592op/s ± 167713.939op/s 59958528.149op/s ± 77652.068op/s 60029799.117op/s 60098570.766op/s 60159730.002op/s 60329995.291op/s 0.62% -1.151 1.007 0.28% 10607.161op/s 1 250
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
normalization/normalize_name/normalize_name/Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Lo... execution_time [1.410ms; 1.411ms] or [-0.023%; +0.023%] None None None
normalization/normalize_name/normalize_name/Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Lo... throughput [3544819.016op/s; 3546444.322op/s] or [-0.023%; +0.023%] None None None
normalization/normalize_name/normalize_name/bad-name execution_time [138.394µs; 138.491µs] or [-0.035%; +0.035%] None None None
normalization/normalize_name/normalize_name/bad-name throughput [36103808.389op/s; 36128890.453op/s] or [-0.035%; +0.035%] None None None
normalization/normalize_name/normalize_name/good execution_time [83.426µs; 83.484µs] or [-0.035%; +0.035%] None None None
normalization/normalize_name/normalize_name/good throughput [59892119.939op/s; 59933699.245op/s] or [-0.035%; +0.035%] None None None

Group 8

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 66cc801 1726663655 ban/tmp-test-benchmarks-bs
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
tags/replace_trace_tags execution_time 2.606µs 2.618µs ± 0.004µs 2.618µs ± 0.002µs 2.620µs 2.625µs 2.630µs 2.630µs 0.48% 0.236 1.167 0.16% 0.000µs 1 250
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
tags/replace_trace_tags execution_time [2.617µs; 2.618µs] or [-0.020%; +0.020%] None None None

Group 9

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 66cc801 1726663655 ban/tmp-test-benchmarks-bs
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
two way interface execution_time 17.312µs 34.977µs ± 14.223µs 33.367µs ± 1.315µs 35.478µs 45.618µs 85.768µs 173.532µs 420.07% 6.526 55.981 40.58% 0.900µs 1 250
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
two way interface execution_time [33.214µs; 36.740µs] or [-5.041%; +5.041%] None None None

Group 10

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 66cc801 1726663655 ban/tmp-test-benchmarks-bs
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
sql/obfuscate_sql_string execution_time 68.882µs 69.045µs ± 0.112µs 69.028µs ± 0.039µs 69.064µs 69.183µs 69.460µs 70.181µs 1.67% 5.248 44.063 0.16% 0.007µs 1 250
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
sql/obfuscate_sql_string execution_time [69.031µs; 69.059µs] or [-0.020%; +0.020%] None None None

Group 11

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 66cc801 1726663655 ban/tmp-test-benchmarks-bs
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
benching string interning on wordpress profile execution_time 138.173µs 139.262µs ± 0.444µs 139.203µs ± 0.187µs 139.407µs 139.896µs 140.819µs 142.679µs 2.50% 2.491 15.163 0.32% 0.028µs 1 250
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
benching string interning on wordpress profile execution_time [139.207µs; 139.317µs] or [-0.039%; +0.039%] None None None

Group 12

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 66cc801 1726663655 ban/tmp-test-benchmarks-bs
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
write only interface execution_time 1.366µs 3.190µs ± 2.236µs 2.984µs ± 0.029µs 3.014µs 3.399µs 6.425µs 30.487µs 921.70% 10.681 117.801 69.94% 0.141µs 1 250
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
write only interface execution_time [2.913µs; 3.467µs] or [-8.687%; +8.687%] None None None

Baseline

Omitted due to size.

@codecov-commenter
Copy link

codecov-commenter commented Aug 26, 2024 •
edited
Loading

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 73.04%. Comparing base (89f48d4) to head (66cc801).

Additional details and impacted files 
@@            Coverage Diff             @@
##             main     #597      +/-   ##
==========================================
- Coverage   73.05%   73.04%   -0.01%     
==========================================
  Files         252      252              
  Lines       36093    36076      -17     
==========================================
- Hits        26366    26352      -14     
+ Misses       9727     9724       -3
Components Coverage Δ
crashtracker 20.53% <ø> (ø)
datadog-alloc 98.73% <ø> (ø)
data-pipeline 90.12% <ø> (ø)
data-pipeline-ffi 0.00% <ø> (ø)
ddcommon 83.08% <ø> (ø)
ddcommon-ffi 69.52% <ø> (ø)
ddtelemetry 59.10% <ø> (ø)
ipc 83.63% <ø> (ø)
profiling 84.26% <ø> (ø)
profiling-ffi 77.42% <ø> (ø)
serverless 0.00% <ø> (ø)
sidecar 40.12% <ø> (ø)
sidecar-ffi 0.00% <ø> (ø)
spawn-worker 50.36% <ø> (ø)
tinybytes 91.66% <ø> (ø)
trace-mini-agent 72.49% <ø> (ø)
trace-normalization 98.25% <ø> (ø)
trace-obfuscation 95.73% <ø> (ø)
trace-protobuf 77.67% <ø> (ø)
trace-utils 93.30% <ø> (+0.41%) ⬆️

@github-actions github-actions bot added common and removed common labels Aug 26, 2024
@bantonsson bantonsson closed this Aug 27, 2024
@bantonsson bantonsson deleted the ban/tmp-test-benchmarks-bs branch August 27, 2024 09:17
@bantonsson bantonsson restored the ban/tmp-test-benchmarks-bs branch August 27, 2024 10:43
@bantonsson bantonsson reopened this Aug 27, 2024
@bantonsson bantonsson force-pushed the ban/tmp-test-benchmarks-bs branch 4 times, most recently from b830d93 to 7a0d6b9 Compare August 29, 2024 14:19
@bantonsson bantonsson force-pushed the ban/tmp-test-benchmarks-bs branch 2 times, most recently from e2baa4a to ff63eb3 Compare September 12, 2024 14:49
@github-actions github-actions bot added profiling Relates to the profiling* modules. data-pipeline labels Sep 12, 2024
@bantonsson bantonsson force-pushed the ban/tmp-test-benchmarks-bs branch 3 times, most recently from 373d6d1 to 746194a Compare September 17, 2024 06:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
ci-build data-pipeline mini-agent profiling Relates to the profiling* modules.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@bantonsson @codecov-commenter