elizabethmcd / genomes-mags-database Goto Github PK

Total number of jobs exceeds max number of jobs that can be submitted in a session
So probably have to split up with a DAG, or manually split submissions by lines in the metadata file
Probably best for reproducibility to do the first option, but somewhat more painful

NCBI wget downloads seem to poop out after around downloading 100 genomes, and right now have per job 500 genomes, so could split up each metadata file so instead has 50 genome split in each, which will create 4000 jobs, but technically the DAG system submits 1 large job and then goes downstream, so might be fine