Have a single sourceable environment/configuration script about rnaseq-pipeline HOT 4 CLOSED

That's a quite nice choice of tool @mbelmadani !

from rnaseq-pipeline.

There's a very nice support for configuration built-in into luigi.

https://luigi.readthedocs.io/en/stable/configuration.html

If we can have all the configuration localized in a single spot (i.e. luigi.cfg), then we can avoid using shell scripts and pass the relevant env entries directly to the component.

In addition, luigi has an external program module that can run the tasks without invoking manually subprocess.call and with the possibility of generating the relevant environment variables.

from rnaseq-pipeline.

Just a quick comment; the main reason for using shell-based configurations was because it was sometimes useful to run a single script from the pipeline without having to go through luigi. Also probably because the scheduler came in after the base scripts were written, mainly as a way to track which jobs were done or not and not to launch the same job multiple times concurrently, so there's probably a lot of useful features that are un/under-utilized (I'm thinking of how distributed processed is called for example, currently with GNU parallel over ssh). In general it does sound like a good idea to leverage more of the luigi built-ins.

from rnaseq-pipeline.

Fixed in f1559f3

We now store the configuration in luigi.cfg and I will progressively adjust the rest of the pipeline. I also updated the Wiki to describe the new way of running specific tasks for users (with luigi CLI).

from rnaseq-pipeline.