Flexbar crashes with high number of barcode combinations about flexbar HOT 9 CLOSED

Thank you Martin to report this. Which error message do you get and how do you know that the number of output files is causing the issue?

Currently, I do not see a reason why 5000 barcode combinations and resulting output files should be a problem. Could you provide a rather small test dataset for it?

Kind regards,
Johannes

from flexbar.

Thanks for the quick reply! The messages reads as follows:
ERROR: Could not open file flexbarOut_barcode_myfrwcode-myrevcode_1.fastq
Where myfrwcode and myrevcode are some sequences.
There are at this point 1018 flexbarOut*.fastq files.
I run a Debian 9 inside virtualbox.

Unfortunately I am not allowed to share my files. I can offer to test any development stage of new code.

from flexbar.

Ok, that makes it difficult for me to reproduce that issue. Actually, there is no restriction for the number of barcode combinations and files in Flexbar. So that should not be the problem in my eyes.

Are you sure that it is not a simple issue, like not enough space in your VM or non filename characters in the names of your barcodes? What are the names of myfrwcode and myrevcode barcodes in case of the affected file?

from flexbar.

Sorry for not being able to share the sequences.

At the time of the crash all files have 0 bytes (not 100% checked). It happens when I use my home folder in the Vm or a mounted folder from the windows host. There is some space available.
I tried with nano the create the file where flexbar fails and it it worked.
For the moment I used the barcode as a barcode name, so the name is just a combination of ACTG

from flexbar.

Is this an open file limit issue? What is your ulimit -Hn?

https://www.tecmint.com/increase-set-open-file-limits-in-linux/

from flexbar.

Good point, this is probably an issue with open file limits.

@martinclausen2 Does the above mentioned post solve your issue?

from flexbar.

Sorry for not responding faster: my open file limit is 1048576, so that should not be the point My issue is not solved so far.

from flexbar.

Thank you for reporting this. I recently tested the issue and Flexbar finished the file creation stage without errors for 5000 and also higher numbers of barcode combinations when the open file limit was high enough. However if the limit was too low I got the same error message that you reported.

Which numbers do you get for:
cat /proc/sys/fs/file-max
ulimit -Sn
ulimit -Hn

Kind regards, Johannes

from flexbar.

Hi Johannes, you are right, I did use ulimit -n unlimited, which was supposed to change both limits, but the softlimit remained unchanged (and unnoticed by me). Maybe I rebooted in between the tests. After an additional ulimit -Sn unlimited flexbar works. (well, it is still running ...) Thanks a lot!
Best regards,
Martin

from flexbar.