Skip to content

Commit

Permalink
feat: split up pc1/pc2 in the window post bench
Browse files Browse the repository at this point in the history
docs: add usage notes in the readme

Resolves #1399
  • Loading branch information
cryptonemo committed Mar 8, 2021
1 parent dbb5bdd commit dbb0193
Show file tree
Hide file tree
Showing 3 changed files with 60 additions and 10 deletions.
31 changes: 31 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,37 @@ The main benchmarking tool is called `benchy`. `benchy` has several subcommands
> cargo run --release --bin benchy -- prodbench
```

### Window PoSt Bench usages

The Window PoSt bench can be used a number of ways, some of which are detailed here.

First, you can run the benchmark and preserve the working directory like this:
```
cargo run --release --bin benchy -- window-post --size 2KiB --cache window-post-2KiB-dir --preserve-cache
```

Then if you want to run the benchmark again to test commit-phase2, you can quickly run it like this:
```
cargo run --release --bin benchy -- window-post --size 2KiB --skip-precommit-phase1 --skip-precommit-phase2 --skip-commit-phase1 --cache window-post-2KiB-dir
```

Alternatively, if you want to test just GPU tree building, you can run it like this:
```
cargo run --release --bin benchy -- window-post --size 2KiB --skip-precommit-phase1 --skip-commit-phase1 --skip-commit-phase2 --cache window-post-2KiB-dir
```

Note that some combinations of arguments will cause destructive changes to your cached directory. For larger benchmark sector sizes, it is recommended that once you create an initial cache, that it be saved to an alternate location in the case that it is corrupted by a test run. For example, the following run sequence will be guaranteed to corrupt your cache:

```
# Do NOT run this sequence. For illustrative purposes only:
# Generate clean cache
cargo run --release --bin benchy -- window-post --size 2KiB --cache window-post-2KiB-dir --preserve-cache
# Skip all stages except the first
cargo run --release --bin benchy -- window-post --size 2KiB --skip-precommit-phase2 --skip-commit-phase1 --skip-commit-phase2 --cache broken-cache-dir
```

The reason this fails is because new random piece data is generated (rather than loaded from disk from a previous run) in the first step, and then we attempt to use it in later sealing steps using data from previously preserved run. This cannot work.

There is also a bench called `gpu-cpu-test`:

```
Expand Down
12 changes: 2 additions & 10 deletions fil-proofs-tooling/src/bin/benchy/main.rs
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,6 @@ fn main() -> Result<()> {
.help("Preserve the directory where cached files are persisted")
.takes_value(false),
)
/*
.arg(
Arg::with_name("skip-precommit-phase1")
.long("skip-precommit-phase1")
Expand All @@ -44,13 +43,6 @@ fn main() -> Result<()> {
.required(false)
.help("Skip precommit phase 2")
.takes_value(false),
)*/
.arg(
Arg::with_name("skip-precommit")
.long("skip-precommit")
.required(false)
.help("Skip precommit phase 1 & 2")
.takes_value(false),
)
.arg(
Arg::with_name("skip-commit-phase1")
Expand Down Expand Up @@ -191,8 +183,8 @@ fn main() -> Result<()> {
("window-post", Some(m)) => {
let preserve_cache = m.is_present("preserve-cache");
// For now these options are combined.
let skip_precommit_phase1 = m.is_present("skip-precommit");
let skip_precommit_phase2 = m.is_present("skip-precommit");
let skip_precommit_phase1 = m.is_present("skip-precommit-phase1");
let skip_precommit_phase2 = m.is_present("skip-precommit-phase2");
let skip_commit_phase1 = m.is_present("skip-commit-phase1");
let skip_commit_phase2 = m.is_present("skip-commit-phase2");
let test_resume = m.is_present("test-resume");
Expand Down
27 changes: 27 additions & 0 deletions storage-proofs-porep/src/stacked/vanilla/proof.rs
Original file line number Diff line number Diff line change
Expand Up @@ -671,6 +671,18 @@ impl<'a, Tree: 'static + MerkleTreeTrait, G: 'static + Hasher> StackedDrg<'a, Tr
assert_eq!(tree_len, config.size.expect("config size failure"));

// Persist the base and tree data to disk based using the current store config.
let tree_c_store_path = StoreConfig::data_path(&config.path, &config.id);
let tree_c_store_exists = Path::new(&tree_c_store_path).exists();
trace!(
"tree_c store path {:?} -- exists? {}",
tree_c_store_path,
tree_c_store_exists
);
if tree_c_store_exists {
std::fs::remove_file(&tree_c_store_path)
.expect("failed to remove tree_c_store_path");
}

let tree_c_store =
DiskStore::<<Tree::Hasher as Hasher>::Domain>::new_with_config(
tree_len,
Expand Down Expand Up @@ -790,6 +802,7 @@ impl<'a, Tree: 'static + MerkleTreeTrait, G: 'static + Hasher> StackedDrg<'a, Tr
}

assert_eq!(tree_count, trees.len());

create_disk_tree::<
DiskTree<Tree::Hasher, Tree::Arity, Tree::SubTreeArity, Tree::TopTreeArity>,
>(configs[0].size.expect("config size failure"), &configs)
Expand Down Expand Up @@ -1135,6 +1148,20 @@ impl<'a, Tree: 'static + MerkleTreeTrait, G: 'static + Hasher> StackedDrg<'a, Tr
i + 1,
tree_count
);

// Remove the tree_r_last store if it exists already
let tree_r_last_store_path = StoreConfig::data_path(&config.path, &config.id);
let tree_r_last_store_exists = Path::new(&tree_r_last_store_path).exists();
trace!(
"tree_r_last store path {:?} -- exists? {}",
tree_r_last_store_path,
tree_r_last_store_exists
);
if tree_r_last_store_exists {
std::fs::remove_file(&tree_r_last_store_path)
.expect("failed to remove tree_r_last_store_path");
}

LCTree::<Tree::Hasher, Tree::Arity, U0, U0>::from_par_iter_with_config(
encoded_data,
config.clone(),
Expand Down

0 comments on commit dbb0193

Please sign in to comment.