Operation Timed out on ROSE2 about pipeline HOT 13 OPEN

moved to Charles Lin site where they see the same issue
linlabbcm/rose2#6

from pipeline.

from pipeline.

No - normally I see files in that folder, but currently they are being left blank. in fact, there are no folders at all underneath the mappedGFF folder.

from pipeline.

from pipeline.

there is no liquidating style output:

but I am running this on a cluster node as a batch job, and thus it is possible that I'm not able to see everything I would if I ran it on an interactive node. Should I try that to see what other errors are thrown?

from pipeline.

ok so here are some more lines of error following the first "Operation Timed Out" error:
MAPPING TO THE FOLLOWING BAMS:
(edited out paths per Berkley's request)

OPERATION TIMED OUT. FILE NOT FOUND
bamliquidator_batch --sense . -e 200 --match_bamToGFF -r

OPERATION TIMED OUT. FILE /data/khanlab/projects/ChIP_seq/DATA/Sample_H3K27ac_024_C_HLFMLBGX3/MACS_Out_p_1e-14/ROSE_out_12500/mappedGFF/Sample_H3K27ac_024_C_HLFMLBGX3_peaks_0KB_STITCHED_TSS_DISTAL_Sample_H3K27ac_024_C_HLFMLBGX3.bam_MAPPED/matrix.txt NOT FOUND

ERROR: FAILED TO MAP

from pipeline.

from pipeline.

We found the 5 minute max time hardcoded into ROSE here:
if utils.checkOutput(mappedOut1File,0.2,5):
print("SUCCESSFULLY MAPPED TO %s FROM BAM: %s" % (stitchedGFFFile, bamFileName))
else:
print("ERROR: FAILED TO MAP %s FROM BAM: %s" % (stitchedGFFFile, bamFileName))

And tried to increase the max run time from 5 minutes to 30 minutes – turns out that didn’t fix it, and since bamliquidator can run at more than 11 million reads per second, and our BAM files don’t have more than 40 million reads each, it is not a time out of running bamliquidator.

I'll try to run bamliquidator on its own and continue from there. Got a few people working on it now with professional code skills, so we will figure it out soon I hope, and will report back here when we solve it.

from pipeline.

also, bamliqidator is running just fine from the command line on this sample. Just isn't working from ROSE2.

from pipeline.

from pipeline.

so we got a “segmentation fault” when running bamliquidator_batch.py on an interactive node; what does that mean?

from pipeline.

from pipeline.

So, while I don't understand it entirely, we got it working by correcting the version control of bamliquidator.
during the failures, it was calling up in this manner:
[+] Loading hdf5 1.8.15
[+] Loading bamliquidator version 1.3...

now, hdf5 doesn't even appear in the loading modules list. we are using something called "singularity", which loads a uniform version:
[+] Loading bamliquidator 1.3.4 on cn1654
[+] Loading singularity 2.4.5 on cn1654

The explanation I got from the folks managing our clusters:

Bamliquidator v.1.3.4 (the default now) is in a singularity container. By loading the module it gives you bamliquidator and bamliquidator_batch as executables/wrappers, which calls bamliquidator inside the container. Singularity is a type of container (just like docker). In the case of bamliquidator, it allows bamliquator to be installed using ubuntu OS (using apt-get) and used/run in the HPC centos OS. More details on: https://hpc.nih.gov/apps/singularity.html

from pipeline.