Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ms.mempool locking #9050

Merged
merged 92 commits into from
Nov 4, 2021
Merged
Show file tree
Hide file tree
Changes from 17 commits
Commits
Show all changes
92 commits
Select commit Hold shift + click to select a range
f545bed
Prority locking to consensus
mariano54 Oct 31, 2021
01f3688
Remove pstats
mariano54 Oct 31, 2021
1ffe17e
Linting
mariano54 Oct 31, 2021
fe5e528
Do some stuff outside of lock
mariano54 Oct 31, 2021
a27a049
Fix startup
mariano54 Oct 31, 2021
bf4a33d
Add log timings
mariano54 Nov 1, 2021
c44e676
Try some different locking
mariano54 Nov 1, 2021
95f1c5c
Add limit
mariano54 Nov 1, 2021
e5ee15a
catch excp
mariano54 Nov 1, 2021
e23be2f
CLVM inside lock
mariano54 Nov 1, 2021
05da9c3
Try using a semaphore instead
mariano54 Nov 1, 2021
629185d
use events for lock queue
mariano54 Nov 1, 2021
175397f
test
mariano54 Nov 1, 2021
abf2690
Add logging for message types
mariano54 Nov 1, 2021
54c0fcb
type
mariano54 Nov 1, 2021
2748641
remove seed
mariano54 Nov 1, 2021
ce61397
check new peak waiters
mariano54 Nov 1, 2021
5d65024
correct FullNodeAPI self.full_node.new_peak._waiters typo
altendky Nov 1, 2021
9b8a08f
correct logging string typos
altendky Nov 1, 2021
632a9f2
only warn about new_peak Waiters if there is at least 1
altendky Nov 1, 2021
f39553b
remove no-longer-accepted parameter to FullNode.peak_post_processing()
altendky Nov 1, 2021
4b179a4
only warn about respond_transaction Waiters if there is at least 1
altendky Nov 1, 2021
9662e7c
lint
altendky Nov 1, 2021
574c065
Change some constants
mariano54 Nov 1, 2021
f8475f2
Small fix and logging changes
mariano54 Nov 1, 2021
849d185
Put message types outside
mariano54 Nov 2, 2021
223d137
Change some log levels so we can test with info
mariano54 Nov 2, 2021
30c7f1b
More logging
mariano54 Nov 2, 2021
ea2a6d4
Increase rate limits but decrease paralelism
mariano54 Nov 2, 2021
13ff27b
tweaks
mariano54 Nov 2, 2021
20290ba
Log dropped tx
mariano54 Nov 2, 2021
3d2c3a0
Fix pool rpc test
mariano54 Nov 2, 2021
a6e87bc
Test fixes
mariano54 Nov 2, 2021
4ce93f0
Mempool optimization
mariano54 Nov 2, 2021
6bab531
Remove from seen if fails
mariano54 Nov 2, 2021
b94ad72
Increase queue sizes
mariano54 Nov 2, 2021
9e4e329
Message types info
mariano54 Nov 2, 2021
ec6aa45
More test and logging
mariano54 Nov 2, 2021
2ae32c6
Small changes to networking just in case
mariano54 Nov 2, 2021
4ac5bd7
Decrease logging
mariano54 Nov 2, 2021
65888df
Decrease logging even further
mariano54 Nov 2, 2021
0332be3
Decrease logging even further even further
mariano54 Nov 2, 2021
b07ccea
Decrease logging 3
mariano54 Nov 2, 2021
fc159bf
Transaction queue
mariano54 Nov 2, 2021
0d0bdb1
Don't cancel tasks or close connection
mariano54 Nov 2, 2021
01f972b
Cancel tasks on disconnect (for shutdown purposed)
mariano54 Nov 2, 2021
e62bd38
Fix typo
mariano54 Nov 2, 2021
b1b7357
Catch cancelled
mariano54 Nov 2, 2021
d1aedcd
Do multiple at a time
mariano54 Nov 2, 2021
4d90340
More accurate farmer response time
mariano54 Nov 2, 2021
bd9605c
More efficiently create tasks
mariano54 Nov 2, 2021
e7304b0
Increase queue size and priority by fee
mariano54 Nov 2, 2021
0aa1032
Revert priority
mariano54 Nov 2, 2021
9d42e0a
Don't re-request too many times for dropped TX
mariano54 Nov 2, 2021
31bf911
Handle cancelled error so we don't go into a bad state
mariano54 Nov 3, 2021
7ebfb1f
Catch cancelled in syncing tasks
mariano54 Nov 3, 2021
5daf545
Reduce new_peak_sem to improve performance
mariano54 Nov 3, 2021
2fbd6c7
Less bytes conversion
mariano54 Nov 3, 2021
a55ca22
Missing file, and 2 workers for CLVM
mariano54 Nov 3, 2021
0ca7b35
Validate BLS in a new thread
mariano54 Nov 3, 2021
b750477
tests
mariano54 Nov 3, 2021
5d92019
Merge branch 'main' into ms.mempool_locking
altendky Nov 3, 2021
cf8901e
Change semaphore constants
mariano54 Nov 3, 2021
e6c6b92
correct a cancellation triggered exception and assertion
altendky Nov 3, 2021
0eaecfc
Fix send_transaction, dont use BaseException, fix tests
mariano54 Nov 3, 2021
f4489aa
Fix more tests
mariano54 Nov 3, 2021
6c0cb36
only log transaction handler cancellation in debug
altendky Nov 3, 2021
5e28b75
typing in log
altendky Nov 3, 2021
551d52d
move unfinished validation to diff proc
Yostra Nov 3, 2021
caba9f9
it is asyncio.CancelledError
altendky Nov 3, 2021
516b7cf
Add a test for bad signature
mariano54 Nov 3, 2021
6f60f4a
Fix more tests, reduce logging, lint
mariano54 Nov 3, 2021
fabfa6d
One more lint
mariano54 Nov 3, 2021
10ff6b6
blockchain tests, pass bytes directly, single call
Yostra Nov 3, 2021
571df63
Try to fix rl_wallet failures
mariano54 Nov 3, 2021
dd6e486
Fix mempool test
mariano54 Nov 3, 2021
3b561b6
catch everything
Yostra Nov 3, 2021
bee7128
Don't test RL wallet
mariano54 Nov 3, 2021
69e1a8f
Merge branch 'main' into ms.mempool_locking
altendky Nov 4, 2021
3a0d8e0
Fix more tests and return error code
mariano54 Nov 4, 2021
9b9e988
Improve error handling in multiprocess
mariano54 Nov 4, 2021
903357c
Add pre-validation time
mariano54 Nov 4, 2021
6f80227
Add pre-validation time in logs, and revert pytest.ini changes
mariano54 Nov 4, 2021
b4aa321
Add log correctly
mariano54 Nov 4, 2021
a8c1d58
Ms.bls cache experiment (#9115)
mariano54 Nov 4, 2021
7380e99
formalize LockQueue shutdown
altendky Nov 4, 2021
632e480
Comments
mariano54 Nov 4, 2021
3b6a9a0
Fix blockchain test
mariano54 Nov 4, 2021
8f0afdd
Improve cache
mariano54 Nov 4, 2021
686afb7
Remove logs
mariano54 Nov 4, 2021
e8ef4ab
Fix sign_coin_spends
mariano54 Nov 4, 2021
5a9d291
Fix pool wallet
mariano54 Nov 4, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 21 additions & 9 deletions chia/consensus/block_body_validation.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
import collections
import logging
import time
from typing import Dict, List, Optional, Set, Tuple, Union, Callable

from chiabip158 import PyBIP158
Expand Down Expand Up @@ -59,6 +60,7 @@ async def validate_block_body(
the result of running the generator with the previous generators refs. It is only present for transaction
blocks which have spent coins.
"""
start = time.time()
if isinstance(block, FullBlock):
assert height == block.height
prev_transaction_block_height: uint32 = uint32(0)
Expand All @@ -82,7 +84,8 @@ async def validate_block_body(
return Err.NOT_BLOCK_BUT_HAS_DATA, None

return None, None # This means the block is valid

log.debug(f"block_body_validation timings 1: {time.time() - start}")
start = time.time()
# All checks below this point correspond to transaction blocks
# 2. For blocks, foliage block, transactions info must not be empty
if block.foliage_transaction_block is None or block.transactions_info is None:
Expand Down Expand Up @@ -150,7 +153,8 @@ async def validate_block_body(

if len(block.transactions_info.reward_claims_incorporated) != len(expected_reward_coins):
return Err.INVALID_REWARD_COINS, None

log.debug(f"block_body_validation timings 2: {time.time() - start}")
start = time.time()
removals: List[bytes32] = []
coinbase_additions: List[Coin] = list(expected_reward_coins)
additions: List[Coin] = []
Expand Down Expand Up @@ -191,13 +195,16 @@ async def validate_block_body(
return Err.TOO_MANY_GENERATOR_REFS, None
if any([index >= height for index in block.transactions_generator_ref_list]):
return Err.FUTURE_GENERATOR_REFS, None

log.debug(f"block_body_validation timings 3: {time.time() - start}")
start = time.time()
if block.transactions_generator is not None:
# Get List of names removed, puzzles hashes for removed coins and conditions created

assert npc_result is not None
cost = calculate_cost_of_program(block.transactions_generator, npc_result, constants.COST_PER_BYTE)
npc_list = npc_result.npc_list
log.debug(f"block_body_validation timings 4: {time.time() - start}")
start = time.time()

# 7. Check that cost <= MAX_BLOCK_COST_CLVM
log.debug(
Expand Down Expand Up @@ -244,7 +251,8 @@ async def validate_block_body(
)
if root_error:
return root_error, None

log.debug(f"block_body_validation timings 5: {time.time() - start}")
start = time.time()
# 12. The additions and removals must result in the correct filter
byte_array_tx: List[bytes32] = []

Expand Down Expand Up @@ -286,7 +294,8 @@ async def validate_block_body(
# timestamp of the block in which it was confirmed
additions_since_fork: Dict[bytes32, Tuple[Coin, uint32, uint64]] = {} # This includes coinbase additions
removals_since_fork: Set[bytes32] = set()

log.debug(f"block_body_validation timings 6: {time.time() - start}")
start = time.time()
# For height 0, there are no additions and removals before this block, so we can skip
if height > 0:
# First, get all the blocks in the fork > fork_h, < block.height
Expand Down Expand Up @@ -342,7 +351,8 @@ async def validate_block_body(
break
curr = reorg_blocks[curr.height - 1]
assert curr is not None

log.debug(f"block_body_validation timings 7: {time.time() - start}")
start = time.time()
removal_coin_records: Dict[bytes32, CoinRecord] = {}
for rem in removals:
if rem in additions_dic:
Expand Down Expand Up @@ -388,7 +398,8 @@ async def validate_block_body(
if rem in removals_since_fork:
# This coin was spent in the fork
return Err.DOUBLE_SPEND_IN_FORK, None

log.debug(f"block_body_validation timings 8: {time.time() - start}")
start = time.time()
removed = 0
for unspent in removal_coin_records.values():
removed += unspent.coin.amount
Expand Down Expand Up @@ -446,7 +457,8 @@ async def validate_block_body(

# create hash_key list for aggsig check
pairs_pks, pairs_msgs = pkm_pairs(npc_list, constants.AGG_SIG_ME_ADDITIONAL_DATA)

log.debug(f"block_body_validation timings 9: {time.time() - start}")
start = time.time()
# 22. Verify aggregated signature
# TODO: move this to pre_validate_blocks_multiprocessing so we can sync faster
if not block.transactions_info.aggregated_signature:
Expand All @@ -462,5 +474,5 @@ async def validate_block_body(
pairs_pks, pairs_msgs, block.transactions_info.aggregated_signature, force_cache
):
return Err.BAD_AGGREGATE_SIGNATURE, None

log.debug(f"block_body_validation timings 10: {time.time() - start}")
return None, npc_result
Loading