experimental redb database backend support

[antonilol]

depends on #1043 because fixed size datatypes made this pr easier

a bit inspired by #765, only not trying to get a dozen DBs to work at once, and not using any dynamic dispatch

still experimental because it performed a bit worse than rocksdb in my initial try (indexing on the signet), but that could be due to other factors as well (both were running from the same bitcoind so one might have gotten prioritized maybe?)

there is a few TODOs still (like metrics from the db) and probably some configuration still needed to get more speed out of the hardware

the Database trait defines all things the rest of electrs can do with a database backend, so more databases can be added later, even a built in in-memory db maybe for automated testing or just quick regtest setup

[antonilol]

redb needs at least rustc 1.66, and the Database trait in the pr uses generic associated types which were stabilized in 1.65 (redb also uses them)
1.66 is almost 1 and a half year old and debian stable still has 1.63

[yancyribbens]

why not sled?

[antonilol]

stats so far:
db size on signet (block height 197272), rocksdb: 152M, redb: 447M (rocksdb uses compression, right?)
indexing time, rocksdb: 1 minute, redb: 6 minutes on ssd
on hdd rocksdb takes about the same time and appears to commit only 3 times, while redb commits way more often and was really slow

clearly I am doing something wrong here because redb's performance is comparable to that of rocksdb according to the project maintainer's benchmarks

could signet just be the wrong thing to test with? will redb perform better (compared to rocksdb) once rocksdb can't do (almost) everything in ram?

[antonilol]

why not sled?

I initially considered starting with redb or sled, both are in beta, but I chose redb because its file format is stable, contrary to sled which has warnings about possible manual migration in its readme

I will try sled right now, hopefully I get it to work with similar performance compared to rocksdb

[yancyribbens]

fwiw sled claims to be in alpha now

[antonilol]

fwiw sled claims to be in alpha now

they name beta in their readme, anyway this should only be an issue if there are bugs in there, and the manual migrations only become an issue when updating sled in the future, if they put a breaking change in a patch version (semver people wont be happy then) we just pin it to a previous (working) patch version

[antonilol]

good news for sled: its minimum rust version is below that of electrs (redb would require a higher one) and I can work around not being able to use generic associated types, there is a workaround for that for old versions of rust
(if I get it working I will revert the change to rust-toolchain.toml, it's set higher now for redb)

[Kixunil]

I hope we don't end up with sled. Manual migration would be very annoying for the users.

[antonilol]

It shouldn't be annoying for electrs users, there is a bit of doc from sled on how to do migration, (import and export functions), in electrs we can make this an automatic migration. I think "manual" refers to users of sled (rust projects that use it as library, like electrs) need to manually add/change code for some updates.

[Kixunil]

Still no migration is better than migration which takes time and resources.