Skip to content

Commit

Permalink
docs: storage update (#80)
Browse files Browse the repository at this point in the history
reflects update in storage plugin fs
  • Loading branch information
cmeesters committed Jun 10, 2024
1 parent 06a1555 commit 7e19560
Showing 1 changed file with 20 additions and 6 deletions.
26 changes: 20 additions & 6 deletions docs/further.md
Original file line number Diff line number Diff line change
Expand Up @@ -198,8 +198,7 @@ Snakemake will check the status of your jobs 40 seconds after submission. Anothe

When using [profiles](https://snakemake.readthedocs.io/en/stable/executing/cli.html#profiles), a command line may become shorter. A sample profile could look like this:

```console
__use_yte__: true
```YAML
executor: slurm
latency-wait: 60
default-storage-provider: fs
Expand All @@ -208,18 +207,33 @@ shared-fs-usage:
- software-deployment
- sources
- source-cache
local-storage-prefix: "<your node local storage prefix>"
remote-job-local-storage-prefix: "<your node local storage prefix>"
local-storage-prefix: "<your local storage prefix, e.g. on login nodes>"
```

The entire configuration will set the executor to SLURM executor, ensures sufficient file system latency and allow automatic stage-in of files using the [file system storage plugin](https://github.com/snakemake/snakemake-storage-plugin-fs).

On a cluster with a scratch directory per job id, the prefix within jobs might be:

```YAML
remote-job-local-storage-prefix: "<scratch>/$SLURM_JOB_ID"
```

On a cluster with a scratch directory per user, the prefix within jobs might be:

```YAML
remote-job-local-storage-prefix: "<scratch>/$USER"
```

It will set the executor to be this SLURM executor, ensure sufficient file system latency, and allow automatic stage-in of files using the [file system storage plugin](https://github.com/snakemake/snakemake-storage-plugin-fs).
Note, that the path `<scratch>` needs to be taken from a specific cluster documentation.

Note, that you need to set the `SNAKEMAKE_PROFILE` environment variable in your `~/.bashrc` file, e.g.:
Further note, that you need to set the `SNAKEMAKE_PROFILE` environment variable in your `~/.bashrc` file, e.g.:

```console
export SNAKEMAKE_PROFILE="$HOME/.config/snakemake"
```

Further note, that there is further development ongoing to enable differentiation of file access patterns.
==This is ongoing development. Eventually you will be able to annotate different file access patterns.==

## Retries - Or Trying again when a Job failed

Expand Down

0 comments on commit 7e19560

Please sign in to comment.