test: Benchmark contract state insertions with DB vs. DB transactions

[bvrooman]

Related issues:

• Closes State performance tests #1134

This PR adds additional benches to compare the performance of inserting state when the existing data is 1) written to the database, and 2) stored in-memory in a transaction. The results demonstrate that when preexisting data is held in a transaction, inserting state has constant complexity and is generally faster, and performance degradation comes from the logarithmic complexity of Merkle tree insertion.

insert state single contract database/insert state with 0 preexisting entries
                        time:   [3.0283 µs 3.0349 µs 3.0417 µs]
insert state single contract database/insert state with 1 preexisting entry
                        time:   [6.4704 µs 6.9960 µs 7.6233 µs]
insert state single contract database/insert state with 10 preexisting entries
                        time:   [15.750 µs 16.668 µs 17.636 µs]
insert state single contract database/insert state with 100 preexisting entries
                        time:   [29.102 µs 30.128 µs 31.204 µs]
insert state single contract database/insert state with 1,000 preexisting entries
                        time:   [45.264 µs 46.447 µs 47.666 µs]
insert state single contract database/insert state with 10,000 preexisting entries
                        time:   [63.839 µs 65.277 µs 66.784 µs]
insert state single contract database/insert state with 100,000 preexisting entries
                        time:   [85.863 µs 87.430 µs 89.119 µs]

insert state single contract transaction/insert state with 0 preexisting entries
                        time:   [2.4829 µs 2.8342 µs 3.4867 µs]
insert state single contract transaction/insert state with 1 preexisting entry
                        time:   [5.4261 µs 5.8455 µs 6.2946 µs]
insert state single contract transaction/insert state with 10 preexisting entries
                        time:   [11.104 µs 11.721 µs 12.352 µs]
insert state single contract transaction/insert state with 100 preexisting entries
                        time:   [20.425 µs 21.213 µs 21.987 µs]
insert state single contract transaction/insert state with 1,000 preexisting entries
                        time:   [28.998 µs 29.852 µs 30.714 µs]
insert state single contract transaction/insert state with 10,000 preexisting entries
                        time:   [38.645 µs 39.401 µs 40.206 µs]
insert state single contract transaction/insert state with 100,000 preexisting entries
                        time:   [49.969 µs 50.776 µs 51.611 µs]

insert state multiple contracts database/insert state with 0 preexisting entries
                        time:   [2.7846 µs 2.7901 µs 2.7962 µs]
insert state multiple contracts database/insert state with 1 preexisting entry
                        time:   [2.9599 µs 2.9688 µs 2.9782 µs]
insert state multiple contracts database/insert state with 10 preexisting entries
                        time:   [3.1822 µs 3.2080 µs 3.2350 µs]
insert state multiple contracts database/insert state with 100 preexisting entries
                        time:   [3.7841 µs 3.9514 µs 4.1838 µs]
insert state multiple contracts database/insert state with 1,000 preexisting entries
                        time:   [4.1301 µs 4.3894 µs 4.7190 µs]
insert state multiple contracts database/insert state with 10,000 preexisting entries
                        time:   [4.4221 µs 4.5124 µs 4.6034 µs]
insert state multiple contracts database/insert state with 100,000 preexisting entries
                        time:   [4.8611 µs 4.9762 µs 5.0904 µs]

insert state multiple contracts transaction/insert state with 0 preexisting entries
                        time:   [2.8071 µs 2.8121 µs 2.8175 µs]
insert state multiple contracts transaction/insert state with 1 preexisting entry
                        time:   [2.9778 µs 2.9842 µs 2.9912 µs]
insert state multiple contracts transaction/insert state with 10 preexisting entries
                        time:   [2.9502 µs 2.9587 µs 2.9677 µs]
insert state multiple contracts transaction/insert state with 100 preexisting entries
                        time:   [2.9706 µs 2.9779 µs 2.9854 µs]
insert state multiple contracts transaction/insert state with 1,000 preexisting entries
                        time:   [2.9824 µs 2.9910 µs 2.9999 µs]
insert state multiple contracts transaction/insert state with 10,000 preexisting entries
                        time:   [2.9799 µs 2.9887 µs 2.9985 µs]
insert state multiple contracts transaction/insert state with 100,000 preexisting entries
                        time:   [2.9731 µs 2.9846 µs 2.9963 µs]

[bvrooman]

Originally, I was exploring if I could implement From<DatabaseTransaction> for VmDatabase to simplify this logic. The implementation meant using the default for GeneratedConsensusFields, but that is only available under the test or test-helpers configuration. During benchmarking, neither of this flags are automatically available, and I did not think it would be valuable to require users to add a test configuration to their benchmarking at this time.

[Voxelot]

Hmm this seems strange, why is there such a big performance difference when there is a single contract vs multiple contract insert when using transactions?

insert state single contract transaction/insert state with 100,000 preexisting entries
                        time:   [49.969 µs 50.776 µs 51.611 µs]
..
insert state multiple contracts transaction/insert state with 100,000 preexisting entries
                        time:   [2.9731 µs 2.9846 µs 2.9963 µs]

[xgreenx]

It is SMT. In the case of one contract with 100_000 entries, we must do many iterations over the tree.

[xgreenx]

Thank you for the benchmarks!=)

It seems the worst case is the insertion of a new state/balance into RocksDB with many entries.

We need to update our benchmark for all opcodes that insert something into the storage to include a huge pre-existing state. I think about 100_000_000 entries as a default value for benchmarks because after this value, the cost slows down, and it is almost impossible to have so many state/balance entries=)

[bvrooman]

Hmm this seems strange, why is there such a big performance difference when there is a single contract vs multiple contract insert when using transactions?

insert state single contract transaction/insert state with 100,000 preexisting entries
                        time:   [49.969 µs 50.776 µs 51.611 µs]
..
insert state multiple contracts transaction/insert state with 100,000 preexisting entries
                        time:   [2.9731 µs 2.9846 µs 2.9963 µs]

Yes, as Green mentioned, the performance difference comes from the different size of SMTs at the time of insertion. In the "single contract" test cases, we are inserting an entry for a contract with preexisting leaves (i.e. 100,000 leaves for the above test case) and accumulating the Merkle tree. The biggest cost here is the tree traversal and update. In the "multiple contract" test cases, we are inserting entries for 100,000 new contracts, each with their own Merkle tree. Because these trees are new, updating them is very fast.