hi, while you have a Cambridge profile listed under https://github.com/nf-core/configs#documentation
and is in the https://github.com/nf-core/configs/blob/master/conf/cambridge.config it is not added under the https://github.com/nf-core/configs/blob/master/nfcore_custom.config which subsequently gives an error:
Unknown configuration profile: 'cambridge'
could you please add it.

Do we need the aws_tower config? It just appears to set the executor which isn't necessary in Tower. Can we remove it @ggabernet?

When running Sarek on IRMA with updated uppmax.config

Command output:
sbatch: error: Batch job submission failed: Invalid feature specification.

#SBATCH -A ngi2016003 -p node -C mem512GB

This probably creates issues running other nf-core pipelines on Irma as well.

Examples are always like this:

Launching `nf-core/scrnaseq` [focused_einstein] - revision: ef3f49479f [dev]
NOTE: Your local project version looks outdated - a different revision is available in the remote repository [c8d04dc40a]
WARNING: Could not load nf-core/config profiles: https://raw.githubusercontent.com/nf-core/configs/master/nfcore_custom.config
No such file: Config file does not exist: https://raw.githubusercontent.com/nf-core/configs/master/conf/conf/igenomes.config

A hint what could go wrong is the second URL, that is of course not correct. I am a bit wondering what happens here and why this is duplicating the conf part of the URL.

@alexblaessle and I got hit by this a couple of times already

Make a logo and write some documentation explaining what this is, how it works and guidelines / instructions for how to add to it.

I've been trying to run the smrnaseq pipeline on the HPC. It doesn't work if I try to use the imperial profile. The error I get is relating to job sizing/resource selection. When I looked at the file, it turned out that the job was being submitted larger than the HPC can handle, and with a submission time of 0. The pipeline works fine if I just run it on one large node.

We might also move from:

-    config_profile_contact = 'Denis O\'Meally (@drejom) or HPRCC Helpdesk ([email protected])'
+    config_profile_contact = 'Denis O\'Meally'
+    config_profile_contact_github = '@drejom'
+    config_profile_contact_email = "[email protected]"

Originally posted by @emiller88 in #577 (comment)

Basically some modules have all of the info, some modules don't. It's fine if they don't want to include an email for example. The real problem is they're not in a consistant format.

So we also need to go through and clean up the CODEOWNERS errors.

It would be good to either move or render the documentation in this repo on the main nf-core website somehow.

The stuff on the readme is quite good and general, and the docs for the profiles are very difficult to find.

Maybe a new subsection of the website dedicated for institutional configs?

Hi,

The link to config file for GIS is broken:
https://github.com/nf-core/configs/blob/master/docs/gis.md

As suggested by @piotr-faba-ardigen , we should check this in the CI stuff.

The documentation for e.g. adding a new config (and what to put in it) is currently quite dense and sometimes skips over things.

We should do a clean-up pass of the README to make it more accessible, and maybe consider splitting it up into different pages.

Custom configs are now hosted centrally at nf-core/configs. We need to think about where we host the documentation for running these config files. These are currently added on a per-pipeline basis but it would be good to also host them centrally so they are available to all pipelines.

This will also probably involve refactoring the current docs to deal with this.

Documention to-do list:

• BINAC (Alex P)
• CCGA
• CFC (Alex P)
• GIS
• hebbe
• phoenix (Alex P)
• uct_hex
• uppmax_devel
• uppmax
• uzh (Alex P)

Hello there! I am a bioinformatics scientist contractor working out of CDC (U.S.) for a group named SciComp. We manage the scientific computing infrastructure at CDC, including the Rosalind HPC cluster. One of our users, Gregory Sprenger, submitted the "rosalind_uge" config to nf-core a while back. Our group never approved of this, so we request that it be taken off of the repository. We have decided for the time being to maintain our own configuration files that users are instructed to use, so we do not want any configuration specific to CDC compute environments in nf-core.

Config in question: conf/rosalind_uge.config

Thank you for taking time to consider this issue, we appreciate it greatly.

Hi @marcmtk, would it be possible to update computerome documentation at https://github.com/nf-core/configs/blob/master/docs/computerome.md

Thank you!

It would be good to get Travis to run some basic tests, even if it's only checking that nextflow can load these configs properly.

Will need to set the base URL sensibly to ensure that the correct remote config files are loaded.

hello,
what is the trick to obtain help from you via slack. I tried to join the nf-core on slack under https://nfcore.slack.com/
choose continue with google and ended up with the message:
"The email address must match one of the domains listed below. Please try another email."
I can not the workspace administrator at nf-core for an invitation either.
Please suggest a working way for joining the nf-core on slack.
Many Thanks & looking forward to reading from you.

While composing the config file for HPC, I found that the slurm executor scope listed inside of profile got ignored (example 3) , so by default the pollInterval will be 5s, which is very unfriendly for multiple-users cluster, but the setting works in process scope outside of profile (example 2), is it a bug or feature?

e.g.1 worked example nextflow.conf:

executor {
      pollInterval = '2 min'
}
profile1 { 
      ...
}

run with

nextflow run -profile profile1 xxx

.nextflow.log

pollInterval: 2m

e.g. 2 worked example nextflow.conf:

process { 
     pollInterval = '2 min'
      ...
}

run with

nextflow run -c nextflow.conf xxx

.nextflow.log

pollInterval: 2m

e.g. 3 failed example nextflow.conf:

profile2 {
     process { 
           pollInterval = '2 min'
           ...
           }
}

run with

nextflow run -profile profile2 xxx

.nextflow.log

pollInterval: 5s

At the moment, we mostly use a single queue per defined cluster environment. However, it might be that we require more than just a single queue and instead should be able to e.g. define

process A uses short, process B afterward needs much memory mem and process C runs very long, e.g. uses long.

At the moment this is not addressed, but we should be flexible in terms of being able to address this 👍

Some users that were using pipelines already using the centralized configs repository had issues after yesterdays functional update in here:

N E X T F L O W  ~  version 19.01.0
Launching `nf-core/rnafusion` [sleepy_goldstine] - revision: b2fb212d31 [dev]
ERROR ~ Unable to parse config file: '/home-link/qeaga01/.nextflow/assets/nf-core/rnafusion/nextflow.config'

 Illegal character in scheme name at index 0: %5B:%5D/conf/cfc.config


-- Check '.nextflow.log' file for details

This affected all pipelines already using the centralized configs, as the ${params.custom_config_base} isn't defined in the previous version of this repository, thus making all changes in this repository propagate to eager, rnaseq, ampliseq, rnafusion, hlatyping, methylseq (?)and potentially others as well.

We should maybe consider having something in nf-core/tools that checks that the pipeline releases are pinning against a specific tag (can be something fixed other than latest), thus making any changes permanent. Thoughts on this?

This code block breaks the functionality of nextflow config -show-profiles with all nf-core pipelines:

Gives:

$ nextflow config -a nf-core/rnaseq
ERROR: Environment variable NXF_GLOBAL_CONFIG is missing. Set it to point to global.config file.
WARNING: Could not load nf-core/config profiles: https://raw.githubusercontent.com/nf-core/configs/master/nfcore_custom.config
No such file: Config file does not exist: https://raw.githubusercontent.com/nf-core/configs/master/conf/conf/test.config

I thought that this was a nextflow bug at first, see nextflow-io/nextflow#1639

@apeltzer - what's going on in this code? Can we refactor it somehow?

Currently this is part of the Azure specific config for Sarek, but can be optimized.

azure {
    batch {
        pools {
            auto {
               autoScale = true

               sku = "batch.node.centos 7"

               offer = "centos-container"

               publisher = "microsoft-azure-batch"

               vmType = 'Standard_E64_v3'

               vmCount = 1

            }
        }
    }
}

Hi,
Do you have any step by step tutorial for processing viral metagenomic samples?
it would be great if you share it with me
Regards

A point I thought of earlier today is that we need to think about how these configs will work offline.

It could be simply that we update the nf-core download command to also always download this repo. That may be sufficient.

Originally posted on the Nextflow repo: nextflow-io/nextflow#792 but was auto-closed due to inactivity. I still think it's a cool idea, so reposting here.

Suggestion is to make a PR to Nextflow to add functionality to config profiles, to allow them to be automatically assigned when a certain condition is met. That logic could be anything, but my initial suggestion was to match the system hostname.

In short, the aim is to make Nextflow support the following syntax:

profiles {
  uppmax {
    when = host.hostname.contains('.uppmax.uu.se')
    includeConfig 'conf/uppmax.config'
  }
  system_two {
    when = host.hostname.contains('.system_two.com')
    includeConfig 'conf/system_two.config'
  }
  system_three {
    when = System.getEnv('SOME_VAR') == 'myvalue'
    includeConfig 'conf/system_two.config'
  }
}

Paolo gave some advice back on the original thread. The most pertinent bit is probably the following:

The config object is parsed using the ConfigParser. The first goal is to understand how the when definition is mapped in the config object. The ConfigParserTest can help to make some experiments.

Then once we have the auto profile information it should be enforce in the ConfigBuilder at this point.

@jfy133, should debug be specified somehow?

nf-core/scrnaseq feature request

Hi nf-core/scrnaseq developer!

I got interested in your very promising package and am trying to set it up on aurora - our HPC system in Lund.

This system (aurora) uses /projects as mount point for the storage. When using singularity container to store and run the analysis software I can not access my data if I do not "mount" this path inside the container. To do so I would need a /projects path in the singularity container and this path would also need to be actively filled (-B /projects:/projects) in the singularity startup.

I think this is a very simple fix - just I am not familiar with nextflow. Is there a configuration that enables me to do this or do you need to change the singularity image in order to become compatible with the Lund University HPC system?

Thank you for your time and help!

Hey,

Not sure if you are aware but the large med-bio queue (as used in the imperial_mb.config) has been deprecated but annoyingly you can still submit jobs to it and they will just stay queued.

Alan.

Some of my processes were set to walltime=00:00:00. Solution below:

original:

withLabel:process_low {
                queue  = 'v1_throughput72'
                cpus   = { 2	 * task.attempt }
                memory = { 48.GB * task.attempt }
                time   = { 8 * task.attempt  }
}

change it to:

withLabel:process_low {
                queue  = 'v1_throughput72'
                cpus   = { 2	 * task.attempt }
                memory = { 48.GB * task.attempt }
                time   = { 8.h * task.attempt  }
}

Currently we always test all configs with every PR.

We should do something similar to module and only check for changes in nfcore_custom.config.

Would be good to include the small and week queue to take some of the pressure off of the basement.

The current documentation is quite sparse on what sort of things you need to consider when trying to make an institutional profile.

I would suggest we make a 'step-by-step how-to' guide on how to do so, with extra 'best practises' and 'tips and 'tricks' for different set up environments.

This would help with lowering the bar for new users to get nextflow/nf-core adopted in their institute.

I was running nf-core/mag with -profile mpcdf,raven,test_full and was getting failed to execute to grid schedular.

Manually submitting the failed job resulted in

sbatch: error: job memory limit for shared nodes exceeded. Must be <= 120000 MB
sbatch: error: Batch job submission failed: Invalid feature specification

We need to play with settings to correctly set this.

I recently discovered the existence of the azure.storage.tokenDuration parameter in the wrong way -- that is, by having a large and expensive workflow fail spectacularly 2/3 of way through compute due to token expiration. 🙃 The nature of this failure appears to be a particular bummer, as it seems to have interfered with cacheing, requiring me to bodge together a fix to prevent needless re-execution of most of the expensive steps.

I'd propose exposing this parameter in the configuration file!

On our UPPMAX clusters, nodes have a given scaling factor where a certain amount of memory is allowed per CPU. Previously, if you submitted a job which requests both CPUs and memory and these values didn't match the scaling factor, SLURM would just silently bump up whichever value was needed to make the scaling factor match. However, apparently this behaviour has now changed and now SLURM simply refuses to run these jobs with the error Requested node configuration is not available.

Instead of writing new configuration files for every pipeline for each UPPMAX cluster, I wonder if we can instead just specify the relevant scaling factor (memory per cpu) for each UPPMAX cluster and adjust the job requests before submission to the cluster.

We already run job requirements through the check_max function, so I perhaps we could tie in to this somehow from the institutional configs? Then run a custom function to round up either the cpus or memory as needed.

The main problem I can see with this is that the check_max function takes one resource type at a type. Here, we need to know both the cpus and memory simultaneously. As such I think that this is going to require new modifications to every pipeline 😞

On the Swedish UPPMAX clusters you can now do modules load nf-core-pipelines to get access to pre-downloaded and cached pipelines complete with their singularity containers.

This is basically undocumented currently, it would be great to add mention of it to the configs UPPMAX docs. It is now the recommended way to run nf-core pipelines on UPPMAX (especially offline clusters, eg. Bianca)

cf https://nbis-workshop-epigenomics.readthedocs.io/en/latest/content/tutorials/nextflow.html