Get raw blocks

[darosior]

Hi,

Reading through the issues this is not the only request but here it is anyway.

I'm currently putting up a Bitcoin backend plugin for C-lightning [1] which would hit an esplora instance's API. C-lightning needs sendrawtransaction, gettxout, getblock, estimatesmartfee and some fields of getblockchainfo.
Almost everything was provided by esplora, but not raw blocks (verbosity 0 in getblock). It would be great to be able to get them, say, the same way we access the raw transactions ("GET /tx/:txid/hex" ==> "GET /block/:blockhash/hex").

Anyway, thanks for open sourcing such a great API for working with Bitcoin !

[1] To potentially replace bitcoind

[lvaccaro]

Electrum protocol should not support rawblock information. This feature should require to use bitcoind to fetch block data, as made in spruned project.

[darosior]

@lvaccaro Given some error messages from the Esplora API I assumed it relied on the bitcoind API for some of its calls so a getblock <blockhash> 1 seems possible

[shesek]

Electrum protocol should not support rawblock information.

It should not, but the esplora electrs fork also provides an HTTP REST API where this could definitely make sense.

so a getblock <blockhash> 1 seems possible

It is technically possible, but esplora's electrs is designed in a way that minimizes its use of bitcoind rpc calls. None of the current endpoints result in bitcoind rpc calls, everything is read directly out of electrs's own rocksdb database.

However, rocksdb does not currently store the full raw blocks, and storing them would add significant storage requirements. There are two other alternatives that don't involve using bitcoind's rpc:

1. Reconstruct the raw block using its header and the list of txids (which can be used to fetch full txs from rocksdb). This reconstruction would be somewhat expensive because it requires de-serializing and re-serializing all the transactions in the block, in order to use rust-bitcoin's block encoding. (an alternative is to treat them as byte blobs and stitch them together manually, but this might be more prone to errors compared to using rust-bitcoin.)

2. Read the raw blocks directly from bitcoind's blkfiles. This would require storing the blkfile and position where the block was found during the indexing process, which we don't currently do. Alongside this, we would also need an rpc-based fallback for users running with --import-json. This would be the most efficient, but somewhat complicated implementation-wise.

Because of the relative complexity this has, this feature is not currently prioritized. But it definitely is something that I'd like to eventually implement and will gladly accept a PR for it!

[shesek]

Well, actually, after giving this some more thought... the alternative outlined at the end of (1) - treating them as byte blobs and manually stitching them together - isn't really that complicated or error prone, its just <raw header><varint tx count><raw txs>.

I went ahead and implemented this in Blockstream/electrs@de1ffb9, will become available with the next deployment.

[darosior]

Thanks !

[shesek]

Thank you for bringing up this suggestion :)

Note that it returns the blocks in binary and not in hex as you requested, to reduce bandwidth. But converting this on your side should be easy enough.

[shesek]

For reference, this is the new endpoint: https://github.com/blockstream/esplora/blob/master/API.md#get-blockhashraw

[darosior]

But converting this on your side should be easy enough.

Yeah absolutely