Use REST interface for calling get_raw_transaction

[andrewtoth]

Closes https://github.com/casey/ord/issues/1558.

Use the REST endpoint for fetching raw transactions instead of RPC endpoint. Only really useful when implementing https://github.com/casey/ord/issues/1364 by reverting https://github.com/casey/ord/pull/1357.

When combined with https://github.com/casey/ord/pull/1516, I synced to 775812 (without --index-sats) in 1 hour 6 minutes. And no more pesky JSON-RPC errors.

The progress bar is kind of silly now though. It should really show progress from first inscription block to current height if no --index-sats.

This needs to have bitcoind run with -rest though, otherwise painful slowness and rpc errors.

[andrewtoth]

The library oxhttp creates a new TcpStream connection for every request, which unfortunately breaks on Macs (apoelstra/rust-jsonrpc#67). I tested a rewritten version copying SimpleHttpTransport from rust-jsonrpc, but it unfortunately has degraded performance. From apoelstra/rust-jsonrpc#76 (comment), it looks like it's much faster to create a new socket connection to bitcoind for every request. Benchmarking a sync with the same setup that I got 1:06 with using a single socket connection, I get ~5hrs :(.

Benchmarking RPC vs REST, it is about ~10-20% faster using REST. However, being able to use a new connection per request for either method is definitely what we want to do. I think the performance is so much better we might want to support reusing connections if not using Mac? I know a lot of users are running this in cloud instances on linux, so might be worth it?

[andrewtoth]

Ok update on this, I switched out oxhttp with hyper. This does two things - hyper pools connections and reuses them for requests if they are going to the same host. So in theory this should work on Mac. I still need someone on Mac to confirm for me. Edit: Confirmed working on Mac 🚀 https://github.com/casey/ord/issues/1648#issuecomment-1427946904.

Second, it is async, so while we don't batch requests like JSON-RPC can, we can execute requests concurrently. This patch now executes up to 15 requests concurrently, since the default -rpcworkqueue is set to 16 and if we exceed that it breaks. With this configuration I managed to sync to block 776260 in 80 minutes.

But there's more. If you configure bitcoind to have a higher rpcworkqueue, you can sync even faster! I set it to 120 and managed to sync in just 39 minutes.

[tzarebczan]

This worked for me to index in about 2 hours last night, thanks for the effort! Did have to restart it once, but not sure what caused this issue:

[indexing blocks] ████████████████████████████████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 114991/776385[indexing blocks] 
█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 480982/776385error: JSON-RPC error: transport error: Couldn't connect to host: Broken pipe (os error 32)

[andrewtoth]

@tzarebczan awesome! Did you also have #1357 reverted? Otherwise it doesn't use this feature.

What about #1516, did you try with that? If not, that would make sense why you have the JSON-RPC error, because it would still be fetching blocks via JSON-RPC.

[tzarebczan]

I used speedup improvements branch from your repo

[andrewtoth]

Closing in favor of https://github.com/casey/ord/pull/1759