wip try to implement scanblocks rpc to speed up full wallet scans

[chrisguida]

Description

This PR uses the scanblocks RPC, which uses bitcoind's internal block filter index to find block hashes relevant to a descriptor wallet. This boosts the performance of wallet rescans by several orders of magnitude, especially when scanning from genesis to tip. I can scan mainnet wallets in about 2 minutes using this code, whereas the prior version of the code takes (probably?) several hours.

However, I didn't find a way to "connect" the blocks to the wallet once fetched, as this error is thrown when attempting to do so:

Error: CannotConnectError { try_include_height: 450 }

According to @evanlinjin this is because:
"In LocalChain, we represent blocks as checkpoints (height, blockhash). When we update the LocalChain with a Block, this creates an update with two checkpoints: (prev_height, prev_block_hash), (height, block_hash). Either prev_block_hash or block_hash needs to exist in the LocalChain for it to "connect"."

So we either need to download the full header chain in order to start connecting blocks, or we need to change the structures to allow connecting blocks without downloading the full header chain first.

Checklists

All Submissions:

• I've signed all my commits
• I followed the contribution guidelines
• I ran cargo fmt and cargo clippy before committing

New Features:

• I've added tests for the new feature
• I've added docs for the new feature
• I've updated CHANGELOG.md

Bugfixes:

• This pull request breaks the existing API
• I've added tests to reproduce the issue which are now passing
• I'm linking the issue being fixed by this PR

[LLFourn]

Hey @chrisguida -- scanblocks is a great idea. The error you are seeing is because you need to assert that the block you are adding is connected to existing block in the LocalChain which @vladimirfomene's apply_block doesn't do. So the question is what should the be API be for this? The big questions are:

1. Should we assume the block you are adding is in the main chain?

2. Should we assume that all the existing blocks we have are in the same chain as the block you are adding? What if this block contradicts one of the others (i.e. prev block hash differs from height before). Otherwise we have to force you to be explicit about which of our existing blocks is valid.
   My first guess is we have:

3. wallet.apply_block_conncted_to(block, block_id_I_say_its_connected_to) -

4. wallet.apply_block_assume_connected(block) - this calls to (1) but with the most appropriate block id to neatly fit it into the chain.

I imagine both of these return an error if what you're doing is contradictory.

cc @vladimirfomene @evanlinjin

[chrisguida]

Hi @LLFourn , thanks for the message! Apologies for the delay, I've been traveling for the last week.

I think your solution in 3-4 seems reasonable, though I'm still unsure of all the implications.

@vladimirfomene Do you plan to implement this in your PR? If not, do you mind if I start?

I think maybe it would be best if a few of us could get on a call and figure out a strategy. Would love to have something working here in the next week or two so I can use this downstream by EOY.

cc @evanlinjin

[vladimirfomene]

@chrisguida I don't mind you going ahead with it. I'm a little bit held up at the afrobitcoin conference here in Accra.

[chrisguida]

@vladimirfomene your force push from a couple of weeks ago doesn't build, so I made a new branch with the original commits

bitcoindevkit/bdk@master...chrisguida:bdk:old/rpcwallet-scanblocks

Will update again once it builds.

Or maybe I should split this procedure out into its own example?

[chrisguida]

@LLFourn I believe my new branch's updates to the apply_block_relevant function (ie passing in prev_block_id) incorporate your suggestion number 3 here #2 (comment)

Can you go more into detail about why we need your 4th suggestion? LocalChain doesn't seem to mind if there are only 3 blocks in the chain, at very distant heights, with this code.

[LLFourn]

(4) is just what you had in mind where prev_block_id is None.

[evanlinjin]

I'll respond to the conversations here by the end of the week. This problem has been on my mind. However I've only been working part-time for the past two weeks.