Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: rebuild tree_r_last trees from a replica file #1170

Merged
merged 1 commit into from
Jun 16, 2020

Conversation

cryptonemo
Copy link
Collaborator

@cryptonemo cryptonemo commented Jun 16, 2020

This PR adds a tool for rebuilding tree_r_last trees from a replica file. The intended usage is for replacing existing trees with an updated rows_to_discard setting, as trees with different values for that setting are incompatible.

OPTIONAL (one time sanity check, but can be skipped entirely): To verify that the tool works as expected, let's use the 'old' default of rows_to_discard as 3 for 32GiB sectors and simply rebuild them to make sure the data produced exactly matches what's already present:

time RUST_LOG=trace RUSTFLAGS="-C target-cpu=native" FIL_PROOFS_ROWS_TO_DISCARD=3 cargo run --release --bin update_tree_r_cache rebuild --size 34359738368 --cache /some/new/output/dir --replica /path/to/existing/sealed-file

OPTIONAL (continued): After this completes, use md5sum to match the tree_r_last files produced in /some/new/output/dir against the ones already present in the path containing the existing tree_r_last files. If these match exactly, the real conversion step is run as follows:

Example usage:

time RUST_LOG=trace RUSTFLAGS="-C target-cpu=native" FIL_PROOFS_ROWS_TO_DISCARD=2 cargo run --release --bin update_tree_r_cache rebuild --size 34359738368 --cache /some/new/output/dir --replica /path/to/existing/sealed-file

Note that in this case, FIL_PROOFS_ROWS_TO_DISCARD is changed from 3 to 2. Alternatively, you could unset that env var, as 2 is the new default value.

The output trees produced in this step should be bigger, as less data is discarded. While it's recommended that you preserve the old tree_r_last files (just until we're sure everything starts up properly with the new code), the newly produced files should be placed where the 'old' ones were present.

Copy link
Collaborator

@porcuquine porcuquine left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. The suggestion in the description require generating the tree twice for every sector to be replaced, right? This seems worth checking at least once, to make sure the user is doing what they intend. Once that is established, I wonder if just checking that the input size is as expected, and perhaps that the generated root (last 32 bytes) of the old and new trees agree would suffice. If so, consider updating the instructions with the lighter options.

@cryptonemo cryptonemo merged commit d49dcb8 into master Jun 16, 2020
@cryptonemo cryptonemo deleted the update-tree-r-cache branch June 16, 2020 17:20
@porcuquine
Copy link
Collaborator

FYI, @travisperson.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants