Setting memory twice when submitting with slurm executor #75
Comments
Your requirement should be translated to |
|
PS would you be interested contributing your workflows to the snakemake-workflows catalogue? see https://snakemake.github.io/snakemake-workflow-catalog/ - some of your look pretty interesting! |
|
Thanks for helping with this!
And the output from the job |
I could definitely do that! I'll add it to my todo list! |
|
Bad news: I cannot reproduce this behaviour. edit: My SLURM version is 23.02.7. I noticed that you are overwriting of My Snakefile is My profile: Does this produce the observed error, too? |
|
That's unfortunate that you can't reproduce it. You are completely correct, using your profile (changing the partition and account) and your |
That is not what I wanted to read ;-) Assuming you have this script: and you run |
|
You are good, that produced the exact same error. Seems to be an issue with my system and not I'll reach out to our system administrators. Thank you so much for all of your help! |
Not really. It is some sort of relief, though. I know that it takes effort to update SLURM, if my colleagues are bitten by a bug — but then again, I would be surprised if you are the first to report. Thanks for the feedback. I will keep this issue open, if you don't mind, and await further feedback. Perhaps, it turns out to be a corner case, we can mitigate. |
|
Sounds great, we are working on it and have so far figured out that this works I will let you know if we make any progress. |
|
urgh, is redundancy a new hobby of SchedMD or is there a technical reason behind it (just a rhetorical question!)? I need to check a couple (read: two, for I do not have more and ask colleagues to do the same) of SLURM versions when I contribute the duplication into the code. I am not sure whether or where there might be side effects. Also, as “my” most current version of SLURM is slightly more up to date than yours, I have to presume, that this is a quirk of your cluster. |
|
This is likely a quirk of my cluster. We will definitely keep working on our side to see if there are good fixes. Again, thanks so much for your help! |
|
I might have found the problem... Our cluster is currently going through some growing pains so the best way to get an interactive job is by staring an interactive So the slurm integration seems to work well as long as I'm not running through |
|
Ah, the issue is that you submit whilst working within job context. I'm afraid, that's not what we designed the plugin for. It should not be an issue either, at least that issue of yours should not arise. Now, we can certainly detect this and program a fat warning. I wonder, however, whether falling back on the actual SLURM executor instead of the jobstep executor is possible as a reaction. Either way, I will keep this issue open until I have an answer to this question. |
cmeesters
added a commit
that referenced
this issue
Jun 25, 2024
Already, two issues (#75 and #22) seem to result from running Snakemake in a SLURM job context and using this executor plugin. This PR introduces detecting if triggered within a SLURM job and issuing a warning accordingly. In principle, the plugin may work in job context. Submitting jobs from jobs has always been a highlight of SLURM. However, settings may lead to unintended behaviour (and would do so without Snakemake, presumably). Hence, we can only warn from the executor.
cmeesters
added a commit
that referenced
this issue
Jul 5, 2024
Already, two issues (#75 and #22) seem to result from running Snakemake in a SLURM job context and using this executor plugin. This PR introduces detecting if triggered within a SLURM job and issuing a warning accordingly. In principle, the plugin may work in job context. Submitting jobs from jobs has always been a highlight of SLURM. However, settings may lead to unintended behaviour (and would do so without Snakemake, presumably). Hence, we can only warn from the executor.
cmeesters
added a commit
that referenced
this issue
Jul 12, 2024
Already, two issues (#75 and #22) seem to result from running Snakemake in a SLURM job context and using this executor plugin. This PR introduces detecting if triggered within a SLURM job and issuing a warning accordingly. In principle, the plugin may work in job context. Submitting jobs from jobs has always been a highlight of SLURM. However, settings may lead to unintended behaviour (and would do so without Snakemake, presumably). Hence, we can only warn from the executor.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Versions
snakemakeversion 8.10.7snakemake-executor-plugin-slurmversion 0.4.4snakemake-executor-plugin-slurm-jobstepversion 0.2.1The problem
I am working on getting
snakemakeversion 8 to work on my slurm server and keep getting the following error:I can see that two resource arguments are being passed when looking at the rule description:
However, I don't know where the
mem_mbis being passed.Profile
My rule
my command
snakemake \ --snakefile Snakefile \ --configfile config.yaml \ --jobs 12 \ --latency-wait 60 \ --rerun-incomplete \ --use-singularity \ --workflow-profile profiles/defaultAttempted fix 1: use
mem_mbI have also tried this using the
mem_mbargument insteadI get the same error, but the double memory request is less obvious
Attempted fix 2 remove profile and specify within rule
I have also tried this where I deleted my profile and just assigned the resources within the rule:
Submit with:
But this also fails with the same srun error:
Attempted fix 3 - submit with sbatch
I've also submitted the job per this issue but that gave the same error as above.
Conclusion
mem_mbis obviously specified somewhere but I am not sure where to look beyond the profile, rules, andsnakemakecommand. Do you have any ideas what I may be missing? Thanks so much for your help!The text was updated successfully, but these errors were encountered: