Too many jobs created if reference genome has a large number of contigs about grenepipe HOT 2 CLOSED

Hi @J-Wall,

that exact same issue recently came up with a collaborator as well, and so I have already thought about adding it. So, great idea!

Back then, I decided to not implement it immediately, as I had many other things on my plate, and the collaborator was working with only 400 contigs, so it was still manageable. But your case is even more extreme, and so yes, I can see the usefulness of this!

I'd be happy to work on this at some point, and sure, if you already have an implementation that does something in that direction, let's continue working on this, or maybe you can even prepare a PR for this if you want :-)

Cheers
Lucas

from grenepipe.

Hi @J-Wall,

finally got to work on this. In the latest set of commits, I've added the functionality that you proposed to grenepipe.

My first draft was based on your idea, thank you very much for this! I did not have a pull request from you to work off, but mentioned you in the commit. I hope that this works as recognition :-)

Then, I started refining your solution. It now uses a simple (greedy) bin packing solver to achieve a slightly more optimal distribution of contigs per group, allows to use the other calling tools as well, and has generally a "less hacky" snakemake setup. Maybe that is also interesting for you - you can back-port it to your fork, or just use the main repo instead, as this should solve your problem as well (and has some more bug fixes by now that are not part of your fork).

Cheers, and thanks again for both the suggestion and your initial solution!
Lucas

PS: Closing this now. Feel free to re-open should there be any issues with the new code.

from grenepipe.