resources assignment when perform parallel jobs about bcbio-nextgen HOT 1 OPEN

Hi @wangpenhok !

I suspect that here you have an indentation issue: you have 4 spaces instead of 2 after resources, and you specifications have not been parsed.

For a one-node non-distributed run, bcbio's logic in allocating resources with (-n 64) is

• try to run all tools with 64 cores, if memory spec is allows for that
• for example the default spec says 4G/ core, so 64 would need 64 x 4G = 256G RAM. If your server does not have this amount of RAM, bcbio is trying to decrease ncores, next would be 32 cores x 128G RAM
• https://github.com/bcbio/bcbio-nextgen/blob/master/config/bcbio_system.yaml

After these calculations, bcbio uses: 32 cores each with 192.1g

When bcbio runs a pipe, it accounts for the fact that every command in the pipe consumes RAM, so it has to decrease cores to fit into the RAM which happened in the command:

bwa mem -t 32 | bamsormadup threads=24

Still, these values are very high for this server. The memory is also consumed for the IO buffers.
You need to try running bcbio with -n 7 or -n10, maximum with -n20.

Large core numbers -n only make sense in a distributed bcbio runs, when these cores are requested across many servers.

SN

from bcbio-nextgen.