File structure about modules HOT 8 CLOSED

Yes, I think that file structure makes sense 👍 Maybe worth adding a command here that can be run in different ways so we can get an idea as to how we will need to deal with this. For most instances though, Im assuming the command will evolve based on updates provided by users of the module - backwards compatibility will have to be a must here.

from modules.

Ah, sorry what I meant by backwards compatibility here is that the command should still work as it did before you changed it e.g. If I add in an additional option to the command as part of a revision does it still work? More to do with the module tests we perform with actual data I guess.

Id imagine there will be instances where these changes arent "backward compatible" e.g. having to remove a parameter to be able to use another one in which case we would need a different module file.

from modules.

backwards compatibility will have to be a must here.

I disagree a bit - git histories provide backwards compatibility! 😉

from modules.

If we release a pipeline / update it using submodules, this will refer to https://github.com/nf-core/modules@HASH with a specific set of tools/methods at that point in time, thus not breaking things. Once you update to a newer version of modules, you might experience differences (e.g. because certain modules were updated), but that is fine/should be fine I suppose?

The only thing you can't do with that approach is using different module revisions at one time, e.g. two different fastqc module revisions at once in a single pipeline, which you probably won't want to do anyways (or only very rarely?).

Phil's on an example, guess that we can test/clarify/set things there then easily and gain some extra experience quickly :-)

from modules.

If I add in an additional option to the command as part of a revision does it still work?

Yes, this will break pipelines, but only when the pipelines update their hash reference to the modules repo (eg. before a release). So then the tests will fail and the pipeline can be fixed.

The previous stable versions of the pipeline will still refer to the previous hash before the change so should still be fine.

from modules.

We will need to update this structure because tools used in this context can also cause confusion with nf-core/tools 😏 Should we replace tools with software? Also, given the preliminary pipeline template for DSL 2 as added here I dont think we will need to host any specific modules for the pipeline template on this repository so could maybe remove the nf-core folder too?

Newly proposed structure:

.
├── .github
│   └── wokflows
│       └── test-processes.yml
├── README.md
└── software
    ├── bwa
    │   └── mem
    │       ├── main.nf
    │       ├── meta.yml
    │       └── test-action.yml
    ├── fastqc
    │   ├── main.nf
    │   ├── meta.yml
    │   └── test-action.yml
    └── samtools
        ├── index
        │   ├── main.nf
        │   ├── meta.yml
        │   └── test-action.yml
        └── sort
            ├── main.nf
            ├── meta.yml
            └── test-action.yml

from modules.

Agreed, software is nicer 👍

I think we should change to this structure ASAP, but maybe worth doing a PR mergeathon first to avoid all of the conflicts that it will cause...

from modules.

Ok, I've been a complete merge-cowboy this morning and just bulldozed through a load of stuff so that we're vaguely ready for the hackathon next week.

I've restructured everything to fit the above proposed file hierarchy in #36

from modules.