Choice between fastq and fasterq-dump when using --force_sratools_download about fetchngs HOT 5 CLOSED

What if you set fasterq-dump to run single-threaded? Would that solve the issue? fasterq is basically deprecated so I wouldn't want to introduce it if it can be avoided.

from fetchngs.

didn't dare to try it yet as I did not want to break the cluster again and since IT very strongly suggested to not use it anymore. In anyway they encouraged us to test it at some point on a free node. However, using it single-threadedly is equivalent to just using fastq-dump no? But I see it is much less of a hassle to simply run it on a single thread than implement a new process. I will try to get around testing it to see if it breaks the cluster again

from fetchngs.

Hi @dmalzl ! Any updates on this issue?

from fetchngs.

Hi @drpatelh
After consulting with our IT and a bit digging it seems this is a problem related to multithreading on our filesystem which seems to not mix well due to incompatibilities between concurrent IO operations on a concurrent filesystem (I do not fully understand why this is). Anyway I resolved it by simply rewriting the module to use fastq-dump (as IT suggested) and this worked quite fine. However, I did not come around testing single threaded fasterq-dump yet but I guess it will be fine too

from fetchngs.

Thanks for the update! Ok, will close this for now but feel free to re-open if the issue persists and if you have any suggestions for how this can be fixed in the pipeline.

from fetchngs.