Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Save on allocations by using fixed size types for database rows #1043

Merged
merged 12 commits into from
Jul 13, 2024

Conversation

antonilol
Copy link
Contributor

@antonilol antonilol commented May 24, 2024

In the WriteBatch struct, I changed the Vec<Box<[u8]>> fields to Vec<[u8, N]>, which saves on memory allocations when indexing blocks and putting the results of that in the db, before this pr that adds up to 1 per unique output address per block (funding), 1 per spent utxo per block (spending) and 1 per transaction (txid) (and a bit more for tip_row and header_rows).

When retrieving from the db currently Boxes are still used, but this can be fixed by using rocksdb's raw iterators. (The optimizer might be able to even remove the Boxes now that they are directly dropped, but as with almost all optimizations, they are not guaranteed).
I also managed to fix this, 800000+ less Boxes are now made when retrieving the headers on startup, and a lot too on electrum protocol requests (i think the amount here depends on the amount of blocks the address is used in, for history and balance at least)

And when loading headers, the iterator is not .collect().into_iter()'ed anymore (found that 2 times, so another 2 intermediate allocations saved).

I was experimenting with redb a bit, redb uses fixed size types, unlike rocksdb that only allows &[u8], this change would also make it easier to use redb as db with electrs in the future, I plan on making a pr for adding redb if I get it to work.

Copy link
Contributor

@Kixunil Kixunil left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks nice!

src/db.rs Outdated Show resolved Hide resolved
src/db.rs Outdated Show resolved Hide resolved
src/db.rs Outdated Show resolved Hide resolved
src/db.rs Outdated Show resolved Hide resolved
src/index.rs Show resolved Hide resolved
src/db.rs Outdated Show resolved Hide resolved
Copy link
Contributor

@Kixunil Kixunil left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good!

Copy link
Owner

@romanz romanz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Many thanks for the contribution!
A single comment regarding DB scan:

src/db.rs Outdated Show resolved Hide resolved
@romanz romanz merged commit b101e30 into romanz:master Jul 13, 2024
4 checks passed
@romanz
Copy link
Owner

romanz commented Jul 13, 2024

Many thanks!

@antonilol antonilol deleted the fixed-size-rows branch July 13, 2024 10:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants