-
Notifications
You must be signed in to change notification settings - Fork 394
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Save on allocations by using fixed size types for database rows #1043
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks nice!
logic is kept unchanged this saves on a memory allocation for every database row that is retrieved
looks cleaner now!
485b9bc
to
231a943
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Many thanks for the contribution!
A single comment regarding DB scan:
Many thanks! |
In the WriteBatch struct, I changed the
Vec<Box<[u8]>>
fields toVec<[u8, N]>
, which saves on memory allocations when indexing blocks and putting the results of that in the db, before this pr that adds up to 1 per unique output address per block (funding), 1 per spent utxo per block (spending) and 1 per transaction (txid) (and a bit more for tip_row and header_rows).When retrieving from the db currentlyBox
es are still used, but this can be fixed by using rocksdb's raw iterators. (The optimizer might be able to even remove theBox
es now that they are directly dropped, but as with almost all optimizations, they are not guaranteed).I also managed to fix this, 800000+ less
Box
es are now made when retrieving the headers on startup, and a lot too on electrum protocol requests (i think the amount here depends on the amount of blocks the address is used in, for history and balance at least)And when loading headers, the iterator is not
.collect().into_iter()
'ed anymore (found that 2 times, so another 2 intermediate allocations saved).I was experimenting with redb a bit, redb uses fixed size types, unlike rocksdb that only allows
&[u8]
, this change would also make it easier to use redb as db with electrs in the future, I plan on making a pr for adding redb if I get it to work.