Switch to the new version of OLTPBench. #1632

lmwnshn · 2021-07-26T09:59:07Z

Heading

Switch to the new version of OLTPBench.

Description

Blocked on merging:

Does NOT need OLTPBench itself to merge oltpbench_tim into master.

noisepage-checks · 2021-07-26T13:58:42Z

Major Decrease in Performance

STOP: this PR has a major negative performance impact

tps (%change)	benchmark_type	wal_device	details
-6.16%	tpcc	RAM disk	Details master tps=22847.0, commit tps=21440.41, query_mode=extended, benchmark_type=tpcc, scale_factor=32.0000, terminals=32, client_time=60, weights={'Payment': 43, 'Delivery': 4, 'NewOrder': 45, 'StockLevel': 4, 'OrderStatus': 4}, wal_device=RAM disk, max_connection_threads=32
-2.19%	tpcc	None	Details master tps=29796.25, commit tps=29142.8, query_mode=extended, benchmark_type=tpcc, scale_factor=32.0000, terminals=32, client_time=60, weights={'Payment': 43, 'Delivery': 4, 'NewOrder': 45, 'StockLevel': 4, 'OrderStatus': 4}, wal_device=None, max_connection_threads=32
-1.94%	tpcc	HDD	Details master tps=21697.7, commit tps=21276.38, query_mode=extended, benchmark_type=tpcc, scale_factor=32.0000, terminals=32, client_time=60, weights={'Payment': 43, 'Delivery': 4, 'NewOrder': 45, 'StockLevel': 4, 'OrderStatus': 4}, wal_device=HDD, max_connection_threads=32
2.64%	tatp	RAM disk	Details master tps=6574.02, commit tps=6747.38, query_mode=extended, benchmark_type=tatp, scale_factor=1.0000, terminals=16, client_time=60, weights={'GetAccessData': 35, 'UpdateLocation': 14, 'GetNewDestination': 10, 'GetSubscriberData': 35, 'DeleteCallForwarding': 2, 'InsertCallForwarding': 2, 'UpdateSubscriberData': 2}, wal_device=RAM disk, max_connection_threads=32
-7.81%	tatp	None	Details master tps=7537.73, commit tps=6949.0, query_mode=extended, benchmark_type=tatp, scale_factor=1.0000, terminals=16, client_time=60, weights={'GetAccessData': 35, 'UpdateLocation': 14, 'GetNewDestination': 10, 'GetSubscriberData': 35, 'DeleteCallForwarding': 2, 'InsertCallForwarding': 2, 'UpdateSubscriberData': 2}, wal_device=None, max_connection_threads=32
5.52%	tatp	HDD	Details master tps=6546.74, commit tps=6907.92, query_mode=extended, benchmark_type=tatp, scale_factor=1.0000, terminals=16, client_time=60, weights={'GetAccessData': 35, 'UpdateLocation': 14, 'GetNewDestination': 10, 'GetSubscriberData': 35, 'DeleteCallForwarding': 2, 'InsertCallForwarding': 2, 'UpdateSubscriberData': 2}, wal_device=HDD, max_connection_threads=32

…tpbench

This might be the bug?

noisepage-checks · 2021-08-07T06:24:57Z

Performance Boost!

Nice job! This PR has increased the throughput of the system.

Could not find any performance results to compare for this commit.

lmwnshn · 2021-08-10T00:41:42Z

benchbase

noisepage-checks · 2021-08-11T08:08:46Z

Major Decrease in Performance

STOP: this PR has a major negative performance impact

tps (%change)	benchmark_type	wal_device	details
-54.65%	tpcc	RAM disk	Details master tps=22928.11, commit tps=10398.17, query_mode=extended, benchmark_type=tpcc, scale_factor=32.0000, terminals=32, client_time=60, weights={'Payment': 43, 'Delivery': 4, 'NewOrder': 45, 'StockLevel': 4, 'OrderStatus': 4}, wal_device=RAM disk, max_connection_threads=32
-60.3%	tpcc	None	Details master tps=29712.44, commit tps=11794.9, query_mode=extended, benchmark_type=tpcc, scale_factor=32.0000, terminals=32, client_time=60, weights={'Payment': 43, 'Delivery': 4, 'NewOrder': 45, 'StockLevel': 4, 'OrderStatus': 4}, wal_device=None, max_connection_threads=32
-57.62%	tpcc	HDD	Details master tps=21952.81, commit tps=9302.72, query_mode=extended, benchmark_type=tpcc, scale_factor=32.0000, terminals=32, client_time=60, weights={'Payment': 43, 'Delivery': 4, 'NewOrder': 45, 'StockLevel': 4, 'OrderStatus': 4}, wal_device=HDD, max_connection_threads=32
-30.74%	tatp	RAM disk	Details master tps=6435.38, commit tps=4456.95, query_mode=extended, benchmark_type=tatp, scale_factor=1.0000, terminals=16, client_time=60, weights={'GetAccessData': 35, 'UpdateLocation': 14, 'GetNewDestination': 10, 'GetSubscriberData': 35, 'DeleteCallForwarding': 2, 'InsertCallForwarding': 2, 'UpdateSubscriberData': 2}, wal_device=RAM disk, max_connection_threads=32
-34.53%	tatp	None	Details master tps=7211.2, commit tps=4720.9, query_mode=extended, benchmark_type=tatp, scale_factor=1.0000, terminals=16, client_time=60, weights={'GetAccessData': 35, 'UpdateLocation': 14, 'GetNewDestination': 10, 'GetSubscriberData': 35, 'DeleteCallForwarding': 2, 'InsertCallForwarding': 2, 'UpdateSubscriberData': 2}, wal_device=None, max_connection_threads=32
-32.15%	tatp	HDD	Details master tps=6349.03, commit tps=4308.11, query_mode=extended, benchmark_type=tatp, scale_factor=1.0000, terminals=16, client_time=60, weights={'GetAccessData': 35, 'UpdateLocation': 14, 'GetNewDestination': 10, 'GetSubscriberData': 35, 'DeleteCallForwarding': 2, 'InsertCallForwarding': 2, 'UpdateSubscriberData': 2}, wal_device=HDD, max_connection_threads=32

codecov · 2021-08-11T08:08:47Z

Codecov Report

Merging #1632 (af0592c) into master (25b264e) will increase coverage by 0.01%.
The diff coverage is n/a.

❗ Current head af0592c differs from pull request most recent head 714253a. Consider uploading reports for the commit 714253a to get more accurate results

@@            Coverage Diff             @@
##           master    #1632      +/-   ##
==========================================
+ Coverage   81.11%   81.13%   +0.01%     
==========================================
  Files         761      761              
  Lines       55258    55258              
==========================================
+ Hits        44825    44835      +10     
+ Misses      10433    10423      -10

Impacted Files	Coverage Δ
src/include/storage/block_access_controller.h	`88.23% <0.00%> (-5.89%)`	⬇️
src/execution/sql/vector.cpp	`85.76% <0.00%> (-0.34%)`	⬇️
src/include/storage/index/bplustree.h	`90.98% <0.00%> (-0.11%)`	⬇️
src/execution/sema/sema_builtin.cpp	`60.54% <0.00%> (-0.09%)`	⬇️
src/execution/vm/bytecode_generator.cpp	`84.00% <0.00%> (+0.08%)`	⬆️
src/parser/postgresparser.cpp	`84.99% <0.00%> (+0.14%)`	⬆️
src/storage/arrow_serializer.cpp	`86.87% <0.00%> (+0.62%)`	⬆️
src/execution/sema/sema_expr.cpp	`77.27% <0.00%> (+1.13%)`	⬆️
...self_driving/modeling/data/grouped_op_unit_data.py	`46.85% <0.00%> (+1.14%)`	⬆️
src/storage/index/hash_index.cpp	`91.13% <0.00%> (+1.26%)`	⬆️
... and 3 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 25b264e...714253a. Read the comment docs.

BenchBase can test itself.

…tpbench

noisepage-checks · 2021-08-12T00:31:12Z

Major Decrease in Performance

STOP: this PR has a major negative performance impact

tps (%change)	benchmark_type	wal_device	details
-51.43%	tpcc	RAM disk	Details master tps=23246.39, commit tps=11291.67, query_mode=extended, benchmark_type=tpcc, scale_factor=32.0000, terminals=32, client_time=60, weights={'Payment': 43, 'Delivery': 4, 'NewOrder': 45, 'StockLevel': 4, 'OrderStatus': 4}, wal_device=RAM disk, max_connection_threads=32
-57.08%	tpcc	None	Details master tps=29097.48, commit tps=12488.65, query_mode=extended, benchmark_type=tpcc, scale_factor=32.0000, terminals=32, client_time=60, weights={'Payment': 43, 'Delivery': 4, 'NewOrder': 45, 'StockLevel': 4, 'OrderStatus': 4}, wal_device=None, max_connection_threads=32
-54.54%	tpcc	HDD	Details master tps=21521.21, commit tps=9783.1, query_mode=extended, benchmark_type=tpcc, scale_factor=32.0000, terminals=32, client_time=60, weights={'Payment': 43, 'Delivery': 4, 'NewOrder': 45, 'StockLevel': 4, 'OrderStatus': 4}, wal_device=HDD, max_connection_threads=32
-26.29%	tatp	RAM disk	Details master tps=6454.45, commit tps=4757.84, query_mode=extended, benchmark_type=tatp, scale_factor=1.0000, terminals=16, client_time=60, weights={'GetAccessData': 35, 'UpdateLocation': 14, 'GetNewDestination': 10, 'GetSubscriberData': 35, 'DeleteCallForwarding': 2, 'InsertCallForwarding': 2, 'UpdateSubscriberData': 2}, wal_device=RAM disk, max_connection_threads=32
-33.08%	tatp	None	Details master tps=7294.7, commit tps=4881.72, query_mode=extended, benchmark_type=tatp, scale_factor=1.0000, terminals=16, client_time=60, weights={'GetAccessData': 35, 'UpdateLocation': 14, 'GetNewDestination': 10, 'GetSubscriberData': 35, 'DeleteCallForwarding': 2, 'InsertCallForwarding': 2, 'UpdateSubscriberData': 2}, wal_device=None, max_connection_threads=32
-25.31%	tatp	HDD	Details master tps=6349.03, commit tps=4741.83, query_mode=extended, benchmark_type=tatp, scale_factor=1.0000, terminals=16, client_time=60, weights={'GetAccessData': 35, 'UpdateLocation': 14, 'GetNewDestination': 10, 'GetSubscriberData': 35, 'DeleteCallForwarding': 2, 'InsertCallForwarding': 2, 'UpdateSubscriberData': 2}, wal_device=HDD, max_connection_threads=32

This reverts commit dac568c.

noisepage-checks · 2021-08-12T21:45:42Z

Major Decrease in Performance

STOP: this PR has a major negative performance impact

tps (%change)	benchmark_type	wal_device	details
-10.48%	tpcc	RAM disk	Details master tps=23070.67, commit tps=20653.66, query_mode=extended, benchmark_type=tpcc, scale_factor=32.0000, terminals=32, client_time=60, weights={'Payment': 43, 'Delivery': 4, 'NewOrder': 45, 'StockLevel': 4, 'OrderStatus': 4}, wal_device=RAM disk, max_connection_threads=32
-6.35%	tpcc	None	Details master tps=30175.57, commit tps=28259.21, query_mode=extended, benchmark_type=tpcc, scale_factor=32.0000, terminals=32, client_time=60, weights={'Payment': 43, 'Delivery': 4, 'NewOrder': 45, 'StockLevel': 4, 'OrderStatus': 4}, wal_device=None, max_connection_threads=32
-5.71%	tpcc	HDD	Details master tps=21604.76, commit tps=20370.56, query_mode=extended, benchmark_type=tpcc, scale_factor=32.0000, terminals=32, client_time=60, weights={'Payment': 43, 'Delivery': 4, 'NewOrder': 45, 'StockLevel': 4, 'OrderStatus': 4}, wal_device=HDD, max_connection_threads=32
-4.84%	tatp	RAM disk	Details master tps=6501.3, commit tps=6186.63, query_mode=extended, benchmark_type=tatp, scale_factor=1.0000, terminals=16, client_time=60, weights={'GetAccessData': 35, 'UpdateLocation': 14, 'GetNewDestination': 10, 'GetSubscriberData': 35, 'DeleteCallForwarding': 2, 'InsertCallForwarding': 2, 'UpdateSubscriberData': 2}, wal_device=RAM disk, max_connection_threads=32
-14.14%	tatp	None	Details master tps=7233.86, commit tps=6210.83, query_mode=extended, benchmark_type=tatp, scale_factor=1.0000, terminals=16, client_time=60, weights={'GetAccessData': 35, 'UpdateLocation': 14, 'GetNewDestination': 10, 'GetSubscriberData': 35, 'DeleteCallForwarding': 2, 'InsertCallForwarding': 2, 'UpdateSubscriberData': 2}, wal_device=None, max_connection_threads=32
-3.28%	tatp	HDD	Details master tps=6262.88, commit tps=6057.52, query_mode=extended, benchmark_type=tatp, scale_factor=1.0000, terminals=16, client_time=60, weights={'GetAccessData': 35, 'UpdateLocation': 14, 'GetNewDestination': 10, 'GetSubscriberData': 35, 'DeleteCallForwarding': 2, 'InsertCallForwarding': 2, 'UpdateSubscriberData': 2}, wal_device=HDD, max_connection_threads=32

noisepage-checks · 2021-08-16T05:46:26Z

Major Decrease in Performance

STOP: this PR has a major negative performance impact

tps (%change)	benchmark_type	wal_device	details
-9.47%	tpcc	RAM disk	Details master tps=23070.67, commit tps=20886.88, query_mode=extended, benchmark_type=tpcc, scale_factor=32.0000, terminals=32, client_time=60, weights={'Payment': 43, 'Delivery': 4, 'NewOrder': 45, 'StockLevel': 4, 'OrderStatus': 4}, wal_device=RAM disk, max_connection_threads=32
-7.8%	tpcc	None	Details master tps=30175.57, commit tps=27821.55, query_mode=extended, benchmark_type=tpcc, scale_factor=32.0000, terminals=32, client_time=60, weights={'Payment': 43, 'Delivery': 4, 'NewOrder': 45, 'StockLevel': 4, 'OrderStatus': 4}, wal_device=None, max_connection_threads=32
-7.08%	tpcc	HDD	Details master tps=21604.76, commit tps=20074.48, query_mode=extended, benchmark_type=tpcc, scale_factor=32.0000, terminals=32, client_time=60, weights={'Payment': 43, 'Delivery': 4, 'NewOrder': 45, 'StockLevel': 4, 'OrderStatus': 4}, wal_device=HDD, max_connection_threads=32
-7.12%	tatp	RAM disk	Details master tps=6501.3, commit tps=6038.61, query_mode=extended, benchmark_type=tatp, scale_factor=1.0000, terminals=16, client_time=60, weights={'GetAccessData': 35, 'UpdateLocation': 14, 'GetNewDestination': 10, 'GetSubscriberData': 35, 'DeleteCallForwarding': 2, 'InsertCallForwarding': 2, 'UpdateSubscriberData': 2}, wal_device=RAM disk, max_connection_threads=32
-12.44%	tatp	None	Details master tps=7233.86, commit tps=6333.97, query_mode=extended, benchmark_type=tatp, scale_factor=1.0000, terminals=16, client_time=60, weights={'GetAccessData': 35, 'UpdateLocation': 14, 'GetNewDestination': 10, 'GetSubscriberData': 35, 'DeleteCallForwarding': 2, 'InsertCallForwarding': 2, 'UpdateSubscriberData': 2}, wal_device=None, max_connection_threads=32
-2.39%	tatp	HDD	Details master tps=6262.88, commit tps=6113.22, query_mode=extended, benchmark_type=tatp, scale_factor=1.0000, terminals=16, client_time=60, weights={'GetAccessData': 35, 'UpdateLocation': 14, 'GetNewDestination': 10, 'GetSubscriberData': 35, 'DeleteCallForwarding': 2, 'InsertCallForwarding': 2, 'UpdateSubscriberData': 2}, wal_device=HDD, max_connection_threads=32

lmwnshn · 2021-08-17T14:12:33Z

I don't foresee a risk of merge conflicts, so this can sit out here for a while. The plan is for me and @jkosh44 to poke at BenchBase next week to do a little light profiling work.

lmwnshn added 2 commits July 26, 2021 04:45

Refactor support for SHOW commands in DESCRIBE.

45d90cf

Adapt CI framework to use oltpbench_tim.

4ce4411

lmwnshn self-assigned this Jul 26, 2021

lmwnshn added the blocked This issue or pull request is in progress, but dependent on another task being completed first. label Jul 26, 2021

lmwnshn added 4 commits July 26, 2021 06:00

Format...

2191d55

Merge branch 'fix_oltpbench' into switch_oltpbench

0885064

Add format header...

e06c2ff

Merge branch 'fix_oltpbench' into switch_oltpbench

408c4d6

Merge branch 'master' into switch_oltpbench

bdbbf3f

mbutrovich and others added 9 commits July 27, 2021 14:55

Merge branch 'master' into switch_oltpbench

779c516

Bunch of fixes/hacks around the test framework stuff.

ee64eb3

Merge remote-tracking branch 'origin/switch_oltpbench' into switch_ol…

3509bf1

…tpbench

More test framework hackery.

3d206c8

Replace one stupid hack with another stupid hack.

8ef0e01

Print out result.

782ef89

Version is being weird.

e1be29b

Suppress Maven download progress.

016eec9

Change folder for Jenkins to benchbase.

e0a35d4

This might be the bug?

lmwnshn added 3 commits August 10, 2021 15:11

Flip over to benchbase.

03a9d77

Merge branch 'master' into switch_oltpbench

bdfd4fe

Fix typo (oltpbench2 -> benchbase).

9d37fec

Don't run BenchBase tests.

7f8d68c

BenchBase can test itself.

Merge remote-tracking branch 'origin/switch_oltpbench' into switch_ol…

c48c3c2

…tpbench

lmwnshn added 2 commits August 12, 2021 11:14

Temporary test with Wan's branch of benchbase.

dac568c

Revert "Temporary test with Wan's branch of benchbase."

714253a

This reverts commit dac568c.

Merge branch 'master' into switch_oltpbench

af0592c

lmwnshn added ready-for-review This PR passes all checks and is ready to be reviewed. Mark PRs with this. and removed in-progress This PR is being actively worked on and not ready to be reviewed or merged. Mark PRs with this. labels Aug 16, 2021

lmwnshn added in-progress This PR is being actively worked on and not ready to be reviewed or merged. Mark PRs with this. and removed ready-for-review This PR passes all checks and is ready to be reviewed. Mark PRs with this. labels Aug 17, 2021

lmwnshn assigned jkosh44 Aug 17, 2021

lmwnshn removed the ready-for-ci Indicate that this build should be run through CI. label Aug 17, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Switch to the new version of OLTPBench. #1632

Switch to the new version of OLTPBench. #1632

lmwnshn commented Jul 26, 2021 •

edited by mbutrovich

Loading

noisepage-checks bot commented Jul 26, 2021

noisepage-checks bot commented Aug 7, 2021

lmwnshn commented Aug 10, 2021

noisepage-checks bot commented Aug 11, 2021

codecov bot commented Aug 11, 2021 •

edited

Loading

noisepage-checks bot commented Aug 12, 2021

noisepage-checks bot commented Aug 12, 2021

noisepage-checks bot commented Aug 16, 2021

lmwnshn commented Aug 17, 2021

Switch to the new version of OLTPBench. #1632

Are you sure you want to change the base?

Switch to the new version of OLTPBench. #1632

Conversation

lmwnshn commented Jul 26, 2021 • edited by mbutrovich Loading

Heading

Description

noisepage-checks bot commented Jul 26, 2021

Major Decrease in Performance

noisepage-checks bot commented Aug 7, 2021

Performance Boost!

lmwnshn commented Aug 10, 2021

noisepage-checks bot commented Aug 11, 2021

Major Decrease in Performance

codecov bot commented Aug 11, 2021 • edited Loading

Codecov Report

noisepage-checks bot commented Aug 12, 2021

Major Decrease in Performance

noisepage-checks bot commented Aug 12, 2021

Major Decrease in Performance

noisepage-checks bot commented Aug 16, 2021

Major Decrease in Performance

lmwnshn commented Aug 17, 2021

lmwnshn commented Jul 26, 2021 •

edited by mbutrovich

Loading

codecov bot commented Aug 11, 2021 •

edited

Loading