Batch module updates - Summer 2024 about modules HOT 9 OPEN

Can we automatically swap the test datasets paths too?

from modules.

@SPPearce can you elaborate / open an issue about this please? (Or point to an existing one)

We have swapped the file paths used in tests from:
file(params.test_data['sarscov2']['genome']['genome_fasta'], checkIfExists: true)
to direct paths like:
file(params.modules_testdata_base_path + 'genomics/sarscov2/genome/genome.fasta', checkIfExists: true)

This is in the guidelines now, but could do with a bulk update.

Made an issue: #5848

from modules.

My concern is that several of the changes require Nextflow 24.04. It means that from this point forward, nf-core will only support Nextflow 24.04+. Is there an analysis of 24.04 vs 22.04-23.10 and what developers have to do to make their pipelines compatible with 24.04 ? Any pipeline developer who is unable to move to 24.04 would remain stuck on pre-June 2024 modules and sub-workflows.

from modules.

Oh yes @ewels , I don't deny the advantages of moving over. There will be a lot of benefits for people to move. Hopefully we can get away with it by just updating the Nextflow version number and can do pipeline template updates later.

from modules.

Should this also include adding stubs? (That is not a breaking change but might still be worth to keep in mind?) Related to #4570.

from modules.

Should this also include adding stubs? (That is not a breaking change but might still be worth to keep in mind?) Related to #4570.

Added, thanks!

from modules.

@SPPearce can you elaborate / open an issue about this please? (Or point to an existing one)

from modules.

@muffato this has been policy for a long time now in nf-core. We are not adopting anything that requires an edge release of Nextflow here, everything is stable. There are other updates that are in 24.04 that we also want to update for pipelines, such as the use of resourceLimits. Some of those changes have been requested for ~6 years, and have been present in the Nextflow edge release for ~6 months.

We will soon bump the minimum required version of Nextflow in the pipeline template, and many / most of these changes should come through automatically via template syncs and module updates. We will do our best to write lint tests to cover most of them so that any required changes don't slip through the gaps. We've also recently started the practice of writing blog posts and doing bytesize talks where we talk over and demonstrate the updates that are required for new versions of the template.

I appreciate that it can be frustrating to keep updating pipelines with these kinds of changes. But that's also part of what makes nf-core great. We are a large pool of feature requests to Nextflow which makes it better. Stability is still possible through using previous versions of Nextflow and pipelines. I don't think that we should be held back from implementing new features once they are available in stable versions of Nextflow.

Any pipeline developer who is unable to move to 24.04

The hope is that no-one falls into this camp. If you can see any potential reasons for this, then please say and we can try to resolve them.

from modules.

Can we add, including arity: to modules or is that out of scope? It'll require a more recent minimum version.

from modules.