snakemake-profiles / lsf Goto Github PK
View Code? Open in Web Editor NEWSnakemake profile for running jobs on an LSF cluster
License: MIT License
Snakemake profile for running jobs on an LSF cluster
License: MIT License
Profiles are a better general solution for cluster configuration. However, I still feel there is a need to allow for configuring special cluster settings for specific rules.
For example, on my cluster, if I want to use GPUs for a job I need to add a collection of parameters to the bsub
call that are not relevant to normal jobs. It would be great if there was a way of specifying this.
Given that the cluster configuration is now deprecated it seems like a better way of doing per-rule configuration is to have a profile-specific way as each cluster will likely have different ways of configuring the "same" thing - e.g. GPU usage.
Allow for an LSF-specific config file that effectively has much the same functionality as the deprecated cluster configuration YAML file. To ensure that it is obvious that this file is LSF-specific it will be required to be named lsf.yaml
. Then, within the job submission process of this profile, we will look for the presence of this file. If there are specific settings for the current rule being submitted, then those settings will be applied.
To make this mirror LSF as much as possible, I will endeavour to make the configuration mirror the LSF bsub
options as closely as possible, and of course, produce thorough documentation on usage.
EDIT: I think the best way to allow for the full suite of bsub
commands is by just required the user to provide strings for the commands, otherwise I would end up implementing an entire API for bsub
(which I am not keen on). So it would look something like
__default__:
- "-P project"
- "-q queue"
my_rule:
- "-P gpu"
- "-m gpu-host"
other_rule: "-q special-queue -gpu 'num=2'"
Which allows either a list or a single string. I prefer the list as it is "neater" but I will support both.
@johanneskoester do you have any problems with this or any points you would like to raise/discuss?
I no longer use LSF and @leoisl's institue will soon move to Slurm. As such we would like to hand over maintenance of this respoitory to others who are actively using LSF. Please comment below if you are willing to take on this role.
See new snakemake
feature: snakemake/snakemake#1395
It seems that when a job is submitted or running in an unreachable host (e.g. host was reachable when job was submitted to it, and while it was executing, it became unreachable), its status becomes UNKWN
: https://www.ibm.com/support/pages/how-requeue-lsf-jobs-unknwn-status-when-remote-execution-host-becomes-unreachable
The current profile will consider that an UNKWN
job is still running, so will not try to kill it:
UNKWN
job should be done with bkill -r
: https://www.ibm.com/support/pages/how-requeue-lsf-jobs-unknwn-status-when-remote-execution-host-becomes-unreachable . Simply bkill
won't do anything. This will take the job from UNKWN
to ZOMBI
, and then EXIT
. Currently, the pipeline does not know the ZOMBI
status:[Predicted exception] Unknown job status: 'ZOMBI'
There are several approaches we can handle UNKWN
and ZOMBI
:
ZOMBI
to STATUS_TABLE
and put as RUNNING
(UNKWN
is already like this), as eventually the ZOMBI
job will become EXIT
, and then we recognise it failed;UNKWN
, we bkill -r
it. When we see ZOMBI
, we say that the job FAILED
;Option 1 seems to need manual intervention though... An UNKWN
job might return to a valid state if the execution host becomes reachable again (I think execution hosts become unreachable when there is an actual issue with the host, and thus they need to be rebooted anyway? So the job is lost anyway?). So the user might want to wait for the UNKWN
job to return to a valid state, or bkill -r
it, and then it becomes ZOMBI
and EXIT
.
Option 2 is more automatic, but requires more development and is more aggressive: as soon as we see UNKWN
, we bkill -r
it and resubmit it. I prefer option 2, as if the execution host became unreachable, I usually prefer to kill the job and submit to a healthy host than waiting for an unknown period of time to maybe it become reachable again.
In any case, Option 1 is already what is sort of implemented. User has to manually kill these jobs, and ZOMBI
state is not recognised, but if everything fails we eventually go look at LSF log. So this is not an urgent issue, but maybe sth nice to fix at some point.
PS: there is a more annoying case where some jobs had the status of RUN
for almost 1 day, and not a single line was executed. I think it might be related with this issue, but somehow LSF did not manage to tag these jobs as UNKWN
. We can retrieve how much computing time was done in a job with bjobs -l
:
Sun Aug 30 16:29:56: Resource usage collected.
The CPU time used is 71 seconds.
I am sure there is a better way that would allow us to query just the CPU time.
It would be nice also to deal with this, as my pipelines actually got stuck, and I thought they were just taking long, but actually nothing was being run... It seems to me that this issue happens when the execution host somehow can't execute anything. It might be solvable also with a LSF preexec command (on the hypothesis that the execution can't execute anything, it won't be able to execute a simple echo
), or with this constant resource usage querying
Hey!
Not quite sure if this is the right place for my issue, as I suspect it's more of a cluster issue than a problem with the profile. But maybe someone can still help.
I'm getting randomly failing job submissions on pipelines that usually work fine. The tracebacks are something along the lines of:
Traceback (most recent call last):
File "/homes/lukasw/.config/snakemake/lsf_short/lsf_submit.py", line 230, in submit
external_job_id = self._submit_cmd_and_get_external_job_id()
File "/homes/lukasw/.config/snakemake/lsf_short/lsf_submit.py", line 216, in _submit_cmd_and_get_external_job_id
output_stream, error_stream = OSLayer.run_process(self.submit_cmd)
File "/homes/lukasw/.config/snakemake/lsf_short/OSLayer.py", line 40, in run_process
completed_process = subprocess.run(
File "[..]/lib/python3.9/subprocess.py", line 528, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command 'bsub -M 1000 -n 1 -R 'select[mem>1000] rusage[mem=1000] span[hosts=1]' -o "logs/cluster/[..]/jobid140_5dad74cf-fda6-41db-9965-ac8a0c18ec25.out" -e "logs/cluster/[..]/jobid140_5dad74cf-fda6-41db-9965-ac8a0c18ec25.err" -J "[..]" -q short [..]/.snakemake/tmp.4tc5r3ou/snakejob.core_metaquast.140.sh' returned non-zero exit status 255.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/homes/lukasw/.config/snakemake/lsf_short/lsf_submit.py", line 259, in <module>
lsf_submit.submit()
File "/homes/lukasw/.config/snakemake/lsf_short/lsf_submit.py", line 236, in submit
raise BsubInvocationError(error)
__main__.BsubInvocationError: Command 'bsub -M 1000 -n 1 -R 'select[mem>1000] rusage[mem=1000] span[hosts=1]' -o "logs/cluster/[..]/jobid140_5dad74cf-fda6-41db-9965-ac8a0c18ec25.out" -e "logs/cluster/[..]/jobid140_5dad74cf-fda6-41db-9965-ac8a0c18ec25.err" -J "[..]" -q short [..]/.snakemake/tmp.4tc5r3ou/snakejob.core_metaquast.140.sh' returned non-zero exit status 255.
Error submitting jobscript (exit code 1):
So bsub
returns exit code 255, which leads the profile to raise a BsubInvocationError
. Since this issue appears sporadically, I am wondering if this could be caused by file system latency (which is quite high on this system). I.e. the jobscript is not available yet at the time of trying to run bsub, would that make sense? Any ideas how I might try and debug this?
Lastly, could the profile help with mitigating this by e.g. waiting a few seconds between creating the jobscript and submitting it or an option to retry submission after a few seconds instead of raising an error at the first failure to submit?
Cheers!
Thanks for putting together the config docs! It looks like things diverged at one point, and the rule-specific yaml file gets parsed as a dict rather than a list. Sorry I should have saved the error message!
I was getting errors until looking at the sge profile and seeing how they specified their keys. I think the example in the lsf README
__default__:
- "-P project2"
- "-W 1:05"
foo:
- "-P gpu"
- "-gpu 'gpu resources'"
should become soemthing like
__default__:
P: "project2"
W: "1:05"
foo:
P: gpu
gpu: 'gpu resources'
Because we emit the log path and the job ID together, setting cluster cancel to bkill
fails with
Terminating processes on user request, this might take some time.
Traceback (most recent call last):
File "/homes/mbhall88/.config/snakemake/lsf/lsf_status.py", line 232, in <module>
print(lsf_status_checker.get_status())
File "/homes/mbhall88/.config/snakemake/lsf/lsf_status.py", line 156, in get_status
status = self._query_status_using_bjobs()
File "/homes/mbhall88/.config/snakemake/lsf/lsf_status.py", line 99, in _query_status_using_bjobs
output_stream, error_stream = OSLayer.run_process(self.bjobs_query_cmd)
File "/homes/mbhall88/.config/snakemake/lsf/OSLayer.py", line 40, in run_process
completed_process = subprocess.run(
File "/hps/software/users/iqbal/mbhall/miniconda3/envs/who-correspondence/lib/python3.10/subprocess.py", line 503, in run
stdout, stderr = process.communicate(input, timeout=timeout)
File "/hps/software/users/iqbal/mbhall/miniconda3/envs/who-correspondence/lib/python3.10/subprocess.py", line 1149, in communicate
stdout, stderr = self._communicate(input, endtime, timeout)
File "/hps/software/users/iqbal/mbhall/miniconda3/envs/who-correspondence/lib/python3.10/subprocess.py", line 2000, in _communicate
ready = selector.select(timeout)
File "/hps/software/users/iqbal/mbhall/miniconda3/envs/who-correspondence/lib/python3.10/selectors.py", line 416, in select
fd_event_list = self._selector.poll(timeout)
KeyboardInterrupt
3399781 logs/cluster/download_data/proj=PRJNA650381.sample=SAMN27765622.run=SRR18914164/jobid258_ff07f69d-f434-4b59-99f1-3f65c066bf08.out: Illegal job ID.
1 out of 1 calls to --cluster-cancel failed. This is safe to ignore in most cases.
I guess we just can't emit the log path with the job ID.
@leoisl maybe we could emit the log path to stderr?
I suppose GitHub actions is the preferred supplier for most snakemake organisation repos? Am I able to set that up myself or is there special organisation access I need @johanneskoester?
Hello,
I am not sure whether there's something wrong in my Snakefile setup: When I use
resources:
mem_mb=32, time_min = 300
The time limit is correctly passed to LSF (bsub) jobs, but memory
configue is still using the memory limit set in the snake-profiles.
Is there someway to change the memory requirement for the run, without changing the snakemake-profile? If there's a quick way to edit the profile/to make the change, that would be a valuable solution for me as well (right now using cookie cutter to create a new profile works for me, but it seems a direct editing on the profile, could be quicker/more direct?)
Thanks a lot
Isaac
Hi I am facing this issue also posted on stackoverflow. Can this be achieved? Thanks!
Hello,
thanks for this plug-in first and foremost.
I noticed an Issue with my workflow and could not create a minimal not working example. The logs of the jobs and the error messages do not line up:
Take job 5767 for example:
[Mon Sep 28 14:14:25 2020]
rule mash_sketch:
input: data/species/genomes/GCA_900290415.1.fna
output: data/species/sketch/GCA_900290415.1.msh
jobid: 5767
wildcards: gca=GCA_900290415.1
resources: mem_mb=4000
Submitted job 5767 with external jobid '9873015 logs/cluster/mash_sketch/gca=GCA_900290415.1/jobid5767_7192d265-413d-4ae5-8d21-1ac836805741.out'.
The log file "logs/cluster/mash_sketch/gca=GCA_900290415.1/jobid5767_7192d265-413d-4ae5-8d21-1ac836805741.err"
is for a different rule "predict proteins":
Building DAG of jobs...
Traceback (most recent call last):
File "[...]/miniconda3/envs/drep_euk/lib/python3.7/site-packages/snakemake/__init__.py", line 709, in snakemake
keepincomplete=keep_incomplete,
File "[...]/miniconda3/envs/drep_euk/lib/python3.7/site-packages/snakemake/workflow.py", line 670, in execute
dag.init()
File "[...]/miniconda3/envs/drep_euk/lib/python3.7/site-packages/snakemake/dag.py", line 177, in init
job = self.update(self.file2jobs(file), file=file, progress=progress)
File "[...]/miniconda3/envs/drep_euk/lib/python3.7/site-packages/snakemake/dag.py", line 715, in update
progress=progress,
File "[...]/miniconda3/envs/drep_euk/lib/python3.7/site-packages/snakemake/dag.py", line 792, in update_
file.inventory()
File "[...]/miniconda3/envs/drep_euk/lib/python3.7/site-packages/snakemake/io.py", line 210, in inventory
self._local_inventory(cache)
File "[...]/miniconda3/envs/drep_euk/lib/python3.7/site-packages/snakemake/io.py", line 224, in _local_inventory
with os.scandir(path) as scan:
FileNotFoundError: [Errno 2] No such file or directory: 'data/species/gmes/compute/GCA_014235955.1'
The out log file is correct and shows there was an issue with the job, but not which
------------------------------------------------------------
Sender: LSF System <lsf@hx-noah-11-07>
Subject: Job 9873015: <mash_sketch.gca=GCA_900290415.1> in cluster <EBI> Exited
Job <mash_sketch.gca=GCA_900290415.1> was submitted from host <hx-noah-39-01> by user <$USER> in cluster <EBI> at Mon Sep 28 14:14:25 2020
Job was executed on host(s) <hx-noah-11-07>, in queue <research-rh74>, as user <$USER> in cluster <EBI> at Mon Sep 28 14:14:26 2020
</homes/$USER> was used as the home directory.
</hps/research/$GROUP/$USER/projects/drep/snakemake> was used as the working directory.
Started at Mon Sep 28 14:14:26 2020
Terminated at Mon Sep 28 14:15:25 2020
Results reported at Mon Sep 28 14:15:25 2020
Your job looked like:
------------------------------------------------------------
# LSBATCH: User input
/hps/research/$GROUP/$USER/projects/drep/snakemake/.snakemake/tmp.hrw102kd/snakejob.mash_sketch.5767.sh
------------------------------------------------------------
Exited with exit code 1.
Resource usage summary:
CPU time : 1.63 sec.
Max Memory : 61 MB
Average Memory : 58.50 MB
Total Requested Memory : 4000.00 MB
Delta Memory : 3939.00 MB
Max Swap : 468 MB
Max Processes : 4
Max Threads : 5
Run time : 59 sec.
Turnaround time : 60 sec.
The output (if any) is above this job summary.
I dont know what is getting messed up, but log files (always just the err file) do not line up with jobs, so debugging is complicated and I suspect this messes with snakemake job success/fail evaluations.
Hi,
Thank you for this wonderful profile. I was wondering whether it is possible to set "immediate-submit: true" in config.yaml file and still use job dependencies setting for LSF profile.
This kind of setting worked well in PBS profile but seems to trouble LSF profile as lsf-submit.py file cannot deal with -w options of LSF.
How can I use this profile so that my snakemake script submits multiple job command with specified dependency options?
For testing, the ideal scenario would be to have a CI setup that executes an ad-hoc LSF cluster (e.g. via docker) to which we would automatically submit a few toy jobs. I am not sure whether such container images are available though.
We have lots of small unit tests where we mock LSF behaviour to ensure our code behaves as we expect. This is really nice as it is very easy to add new features or change the behaviour of existing features without having to test in a real LSF system, and check if it works. With mocking, testing new code is very easy.
However, we can have issues if we incorrectly mock LSF behaviour. Also, having some end-to-end tests in a real LSF system (e.g. in a container) could also be valuable. The only other snakemake profile that has tests is slurm, and their tests are these end-to-end tests in a slurm container: https://github.com/Snakemake-Profiles/slurm/tree/master/tests
End-to-end tests in a container with proper setup and proper tests do take time and effort (way more complicated than using mock). I am unsure if we should add these tests, and what is the priority.
I am receving the following error:
Resuming incomplete job 9 with external jobid '3.8.13 | packaged by conda-forge | (default, Mar 25 2022, 06:04:10) '.
Traceback (most recent call last):
File ".config/snakemake/lsf/lsf_status.py", line 207, in <module>
jobid = int(split_args[0])
ValueError: invalid literal for int() with base 10: '3.8.13'
WorkflowError:
Failed to obtain job status. See above for error message.
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: .snakemake/log/2022-05-04T193619.896311.snakemake.log
SnakeMake version: 7.5.0
It can be related to issue #45
Hi,
I would suggest changing the order of rule-specific parameters to first rule_params
and then default_params
. bsub takes the first argument and ignores any other occurrences, hence the default is always taken.
Thanks for the work and best,
Nico
Hello,
I was trying to instruct snakemake to ignore a specific lsf host which is faulty, so wrote this in my lsf.yaml
:
__default__:
- "-R \"select[hname!='hl-codon-32-02']\""
(This gets parsed as {'__default__': ['-R "select[hname!=\'hl-codon-32-02\']"']}
by pyyaml)
The reason I do this is that if the hostname to ignore has special characters, the shell command needs to look like -R "select[hname!='hname']"
(see IBM docs : If you need to include a hyphen (-) or other non-alphabetic characters within the string, enclose the text in single quotation marks, for example, bsub -R "select[hname!='host06-x12']"
)
However this gets passed at submission time as -R select[hname!='hl-codon-32-02']
but lsf cannot submit that (try bsub -Is -R select[hname!='hl-codon-32-02'] bash
for eg)
Pfew- long story short, I have found a simple solution to this, and putting in a PR with unit tests.
Hello,
Group jobids seem to be string uuids which causes an error in lsf_submit.py
https://snakemake.readthedocs.io/en/stable/_modules/snakemake/jobs.html
See snakemake
new feature: snakemake/snakemake#1397
This is a more efficient and better way to query for job statuses than bjobs <jobid>
, but rather complex to implement. An implementation for slurm cluster can be found here: https://github.com/holtgrewe/snakemake-profiles-slurm/blob/slurm-sidecar/%7B%7Bcookiecutter.profile_name%7D%7D/slurm-sidecar.py and can be used as base
This profile is incompatible with Snakemake version 8 because the --cluster
command line arguments were deprecated (more information here).
Here is this same issue being discussed on the slurm profile Github.
I understand this package is no longer being actively maintained, but I thought I would leave a note in case it's helpful to someone in the future.
Snakemake version 7 still works fine with this profile (more specifically, I'm using v7.32.4).
Hello,
In order to specify e.g. a time limit (which can be needed to select an appropriate queue) for a certain rule, one needs to create lsf.yaml
(see also #7 and #13). This file needs to be in the working directory of the pipeline.
This can be difficult/slightly annoying to handle when lsf.yaml
needs to be moved manually to the correct working directory before executing the pipeline is possible.
Is there a better way of doing this that I am missing?
Or asked differently, what is the difference between standard and non-standard resources?
When the default cluster log directory does not exist, LSF won't be able to redirect the error and output stream to the log directory, causing it to fallback to the default behaviour, which is to e-mail the output of these streams (at least in EBI cluster). I guess it would be nice to either i) print a warning about this, or ii) error out saying that the log directory does not exist or iii) create the log directory automatically. Not sure which would be the best option, but I think ii) or iii) are fine, since I guess it will be common for users to forget to create the cluster log dir.
Or maybe there a better option?
Thanks!
Hi,
I find myself running several workflows with the same rule name, leading to the log directory with that rule name holding log files for multiple workflows.
Currently I just rename rules to avoid that, but what do you think of prefixing the log dir name with the workflow file name?
See #45 (comment) and #45 (comment)
I donβt like the format the names of the cluster log files have been changed to. It makes it impossible to figure out what log file relates to what job without digging into the snakemake stderr log (which is one of the main things I hate about nextflow).
self.logdir / "{jobid}_{random_string}.err".format(jobid=self.jobid, random_string=self.random_string)
self.logdir / self.rule_name / self.wildcards_str / "jobid{jobid}_{random_string}.err".format(jobid=self.jobid, random_string=self.random_string)
# current
'logdir/2_random.out'
# proposed
'logdir/search_fasta_on_index/i=0/jobid2_random.out'
There are two major advantages I see to the new naming scheme.
I use LSF in two different clusters. In one, this is the LSF version:
$ lsid
IBM Spectrum LSF Standard 10.1.0.6, May 25 2018
Explicitly specifying memory units in this LSF version works just fine, e.g. this command works fine:
bsub -M 1000MB -n 1 -R 'select[mem>1000MB] rusage[mem=1000MB] span[hosts=1]' -o ls.out -e ls.err -J ls_job ls
In the other cluster, the version is a bit older:
$ lsid
IBM Spectrum LSF Standard 10.1.0.0, Jul 08 2016
Explicitly specifying memory units in this cluster fails:
$ bsub -M 1000MB -n 1 -R 'select[mem>1000MB] rusage[mem=1000MB] span[hosts=1]' -o ls.out -e ls.err -J ls_job ls
1000MB: MEMLIMT value should be a positive integer. Job not submitted.
It works if MB
is removed:
$ bsub -M 1000 -n 1 -R 'select[mem>1000] rusage[mem=1000] span[hosts=1]' -o ls.out -e ls.err -J ls_job ls
Job <9491039> is submitted to default queue <standard>.
I did not track which update of LSF between these two versions enabled memory units to be specified in -M
.
I wonder if:
Hello, it would be nice to be able to set the job time limit (-W).
It seems like this would require adding an entry to a rule's resources, but it's not clear to me that there is a standard key for this in snakemake.
See #41 (comment)
I am trying to select some specific nodes on lsf cluster by making a new cluster.json file:
{
"__default__": {
"queue": "normal",
"nodes": "lz-gpu lf-gpu ln-gpu lx-gpu ly-gpu ll-gpu",
"extra": "",
"threads": 4 # Default thread count, can be overridden per rule
}
}
But i am getting this error?
File "/home/ryanr2/data/anaconda3/envs/sopa/lib/python3.10/subprocess.py", line 526, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command 'bsub -M 200 -n 1 -R 'select[mem>200] rusage[mem=200] span[hosts=1]' -W 300 -o "logs/cluster/patch_segmentation_baysor/index=53/jobid55_6637d0e7-670b-4ba2-ad88-3580dba86228.out" -e "logs/cluster/patch_segmentation_baysor/index=53/jobid55_6637d0e7-670b-4ba2-ad88-3580dba86228.err" -J "patch_segmentation_baysor.index=53" -q normal /lila/data/chanjlab/ryanr2/sopa/workflow/.snakemake/tmp.5z0azw4k/snakejob.patch_segmentation_baysor.55.sh' returned non-zero exit status 255.
any idea?
Hi!
I was trying to set queue for a rule by adding
cluster:
queue='long'
but snakemake (5.22.1) gave me a SyntaxError Unexpected keyword cluster in rule definition (test.smk, line 4)
I also tried editing lsf.yaml
file by adding test: '-q long'
, but the job was submitted to the default queue.
As I understand, the problem is with this piece of code
https://github.com/Snakemake-Profiles/snakemake-lsf/blob/2e6f23cbea58bb07bde5eff873be6bc87f2a4018/%7B%7Bcookiecutter.profile_name%7D%7D/lsf_submit.py#L63-L64
And I was able to hack it by replacing cluster
with params
in lsf-submit.py
and adding this to the rule:
params:
queue='long'
Then the job was submitted to the long queue as expected.
Sometimes we want to try running a job with 100 GB on 1st try, 300 GB on 2nd try and 1 TB of RAM on 3rd try. The third attempt might be needed to be submitted to a big mem queue, while the first two could be to the standard queue. The user should be able to specify which queue the profile should use at each attempt
lsf_status.py
is sometimes unable to retrieve the status of a running job and fails with
[Predicted exception] Error calling bjobs: Command 'bjobs -o 'stat' -noheader 212651182' returned non-zero exit status 255.
By default, this is tried once and then makes Snakemake crash. Changing the default value from 1 to, e.g., 10 in the line
Is there a reason why this is set to 1 by default although the retry mechanism exists? How about setting it to a higher number or making this configurable via the cookiecutter template.
I work on a cluster where I need to do module load singularity
to add singularity to the path. I must do this within every new job, not once at the start -- module load singularity && bsub "singularity"
fails. Based on my experience with a solution similar to what you have here, I suspect that this setup won't allow --use-singularity
. I predict it will fail with the same sort of error, which will manifest like this:
Building DAG of jobs...
WorkflowError:
The singularity command has to be available in order to use singularity integration.
My current workaround is to do load module singularity
as a prefix to the variable you call jobscript
in the wrapper (line 98 of lsf-submit.py).
thinking it could be because not all params required by the bsub is in the profile e.g. -G. How can I provide those?
Error:
Submitted group job 29137fca-fc8a-5511-88d3-2149b00a8b5e with external jobid '5412741 logs/cluster/group_1/unique/jobid29137fca_07818911-9e6a-460c-a46e-b0f3f0bdb675.out'.
Traceback (most recent call last):
File "/homes/leandro/.config/snakemake/lsf/lsf_status.py", line 201, in <module>
jobid = int(sys.argv[1])
ValueError: invalid literal for int() with base 10: '5412741 logs/cluster/group_1/unique/jobid29137fca_07818911-9e6a-460c-a46e-b0f3f0bdb675.out'
WorkflowError:
Failed to obtain job status. See above for error message.
The reason is that on snakemake 7.1.1 the job id is now quoted. LSF profile works on snakemake 7.1.0 though. Will provide a fix soon
@johanneskoester would you be willing to add @leoisl as a contributor on this repository? He is a software engineer in my group and has been helping improve the profile. It would be great to be able to request PR reviews and have him be able to deal with issues too.
I've tried using lsf.yaml
to define rule-specific options multiple times over the last year, but it never has worked. Unfortunately, it is always faster to just edit my Snakefile, but I thought I'd post an Issue this time because it really would be more appropriate, vis-Γ -vis reusability, to have these settings in the YAML. In the most recent attempt, I am trying
__default__:
- "-W 3:59"
foo:
- "-W 24:00"
but all my foo
jobs still get created with the default settings. Has anyone else had success using this?
snakemake
accepts a --default-resources
option that allows us to specify the default mem_mb
, disk_mb
and tmpdir
if a rule did not specify these directives. We can ask for these values in the cookie-cutter, and in config.yaml
they can be specified using the following format:
default-resources: ["mem_mb=7777", "disk_mb=8888", "tmpdir='/hps/nobackup/research/zi/leandro/snakemake_test/temp'"]
Adding this to my config.yaml
, and submitting this dummy job, which does not specify any of these directives:
rule make_file:
output:
"new_file.txt"
shell:
"touch {output}"
I got the following log:
[Tue Apr 19 11:39:19 2022]
rule make_file:
output: new_file.txt
jobid: 1
resources: mem_mb=7777, disk_mb=8888, tmpdir=/hps/nobackup/research/zi/leandro/snakemake_test/temp
Job script is:
#!/bin/sh
# properties = {"type": "single", "rule": "make_file", "local": false, "input": [], "output": ["new_file.txt"], "wildcards": {}, "params": {}, "log": [], "threads": 1, "resources": {"mem_mb": 7777, "disk_mb": 8888, "tmpdir": "/hps/nobackup/research/zi/leandro/snakemake_test/temp"}, "jobid": 1, "cluster": {}}
cd /hps/nobackup/research/zi/leandro/snakemake_test && /hps/nobackup/research/zi/leandro/miniconda3/bin/python -m snakemake --snakefile '/hps/nobackup/research/zi/leandro/snakemake_test/Snakefile' 'new_file.txt' --allowed-rules 'make_file' --cores 'all' --attempt 1 --force-use-threads --wait-for-files '/hps/nobackup/research/zi/leandro/snakemake_test/.snakemake/tmp.ftj3136o' --force --keep-target-files --keep-remote --max-inventory-time 0 --nocolor --notemp --no-hooks --nolock --ignore-incomplete --skip-script-cleanup --conda-frontend 'mamba' --wrapper-prefix 'https://github.com/snakemake/snakemake-wrappers/raw/' --latency-wait 10 --scheduler 'greedy' --scheduler-solver-path '/hps/nobackup/research/zi/leandro/miniconda3/bin' --default-resources 'mem_mb=7777' 'disk_mb=8888' "tmpdir='/hps/nobackup/research/zi/leandro/snakemake_test/temp'" --mode 2 && exit 0 || exit 1
if we instead specify mem_mb
in the job:
rule make_file:
output:
"new_file.txt"
resources: mem_mb=100
shell:
"touch {output}"
we get this log:
[Tue Apr 19 11:49:00 2022]
rule make_file:
output: new_file.txt
jobid: 1
resources: mem_mb=100, disk_mb=8888, tmpdir=/hps/nobackup/research/zi/leandro/snakemake_test/temp
with this job script:
#!/bin/sh
# properties = {"type": "single", "rule": "make_file", "local": false, "input": [], "output": ["new_file.txt"], "wildcards": {}, "params": {}, "log": [], "threads": 1, "resources": {"mem_mb": 100, "disk_mb": 8888, "tmpdir": "/hps/nobackup/research/zi/leandro/snakemake_test/temp"}, "jobid": 1, "cluster": {}}
cd /hps/nobackup/research/zi/leandro/snakemake_test && /hps/nobackup/research/zi/leandro/miniconda3/bin/python -m snakemake --snakefile '/hps/nobackup/research/zi/leandro/snakemake_test/Snakefile' 'new_file.txt' --allowed-rules 'make_file' --cores 'all' --attempt 1 --force-use-threads --wait-for-files '/hps/nobackup/research/zi/leandro/snakemake_test/.snakemake/tmp.iq82hnq_' --force --keep-target-files --keep-remote --max-inventory-time 0 --nocolor --notemp --no-hooks --nolock --ignore-incomplete --skip-script-cleanup --conda-frontend 'mamba' --wrapper-prefix 'https://github.com/snakemake/snakemake-wrappers/raw/' --latency-wait 10 --scheduler 'greedy' --scheduler-solver-path '/hps/nobackup/research/zi/leandro/miniconda3/bin' --default-resources 'mem_mb=7777' 'disk_mb=8888' "tmpdir='/hps/nobackup/research/zi/leandro/snakemake_test/temp'" --mode 2 && exit 0 || exit 1
so it indeed works
A declarative, efficient, and flexible JavaScript library for building user interfaces.
π Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. πππ
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google β€οΈ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.