IOError: [Errno 71] Protocol error about methylpy HOT 26 CLOSED

Can you provide some test data to reproduce the error? Thanks.

from methylpy.

I first ran python run_test.py to test the package, 2 of the tests failed: DMRfind and reidentify-DMR

Tests start
Wed Sep 25 17:34:24 2019

Test importing methylpy module.

• importing call_mc_se: pass
• importing call_mc_pe: pass
• importing DMRfind: pass
• importing parser: pass
  Check methylpy executable:pass

methylpy is successfully installed!

Check whether dependencies are available in PATH

• find bowtie: found version 1.2.2
• find bowtie2: found version 2.3.5.1
• find minimap2: found version 2.17-r941
• find samtools: found version 1.9
• find cutadapt: found version 1.18
• find java: found version 2019-04-16
  Both bowtie and bowtie2 are available and will be tested.

Test DMRfind: failed
Test reidentify-DMR: failed

Test build-reference with bowtie: pass
Test build-reference with bowtie2: pass
Test build-reference with minimap2: pass

Test single-end-pipeline with bowtie: pass
Test single-end-pipeline with bowtie2: pass
Test single-end-pipeline with minimap2: pass

Test paired-end-pipeline with bowtie: pass
Test paired-end-pipeline with bowtie2: pass
Test paired-end-pipeline with minimap2: pass

Test quality filter for BAM file of single-end data: pass

Test call-methylation-state for single-end data: pass
Test call-methylation-state for paired-end data: pass

Test merge-allc: pass

Test filter-allc: pass

All tests are done!
Wed Sep 25 17:46:25 2019
The tests took 720.78 seconds!

then I ran
methylpy build-reference
--input-files wtMouseTGFBI.fa
--output-prefix wtMouseTGFBI
--aligner minimap2

Traceback (most recent call last):
File "/home/hilar/.local/bin/methylpy", line 5, in
parse_args()
File "/home/hilar/.local/lib/python2.7/site-packages/methylpy/parser.py", line 50, in parse_args
buffsize=args.buffsize)
File "/home/hilar/.local/lib/python2.7/site-packages/methylpy/call_mc_se.py", line 604, in build_ref
f.close()
IOError: [Errno 71] Protocol error

I used bowtie2 instead of minimap and it worked.
what can I do for failed tests for DMRfind and reidentify-DMR?

Thanks

from methylpy.

Thanks. What was the error message of the DMR failure? You can find it in the test_error_msg.txt and test_output_msg.txt files.

You may want to try to follow this procedure and see if the DMRfind issue can be fixed: https://github.com/yupenghe/methylpy#optional-step---compile-rmscpp

Are you fine with using bowtie2? Or you will need to use minimap2?

from methylpy.

Thanks for your response. below are the tests results I get for failed DMR and re-identify-DMR:

test_error_msg.txt
test_output_msg.txt

When I run the code below I get the following error:
methylpy DMRfind
--allc-files ResultsControlHighMe/allc_mCornea_H.tsv
ResultsControlLowMe/allc_mCornea_L.tsv
--samples HighMe_control LowMe_control
--mc-type "CGN"
--chroms wtMouseTGFBI
--num-procs 8
--min-num-dms 5
--output-prefix Results/CG_DMR_High_Low

output error:
Filtering allc files using 2 node(s).
Mon Sep 30 14:52:18 2019

Splitting allc files for chromosome wtMouseTGFBI
Mon Sep 30 14:52:19 2019

(<type 'exceptions.KeyError'>, 184)
'wtMouseTGFBI'
Running RMS tests failed.

from methylpy.

Also regarding your question for using minimap2 vs Bowtie2, when I use minimap2 it gives me an error saying that the format is not supported because it internally runs bowtie2 although I use --aligner minimap2 function. So when I changed the code to --aligner bowtie2 it works.

from methylpy.

The error you got in running DMRfind is due to two issues:

• The DMRfind module relies on gsl library. You may need to install the library and recompile. See https://github.com/yupenghe/methylpy#optional-step---compile-rmscpp
• --chroms option is used to specify the chromosomes you would like to run DMRfind on. Could you please double check wtMouseTGFBI is the name of a chromosome in your allc files (e.g. ResultsControlLowMe/allc_mCornea_L.tsv).

Regarding minimap2, this aligner is being actively updated. Unfortunately, I haven't got time to update methylpy to keep up with the changes. Please do not use minimap2 for now.

from methylpy.

I tried the executable but didnt work:

./run_rms_tests.out
./run_rms_tests.out: error while loading shared libraries: libgsl.so.0: cannot open shared object file: No such file or directory

I then tried below:
g++ -O3 -l gsl -l gslcblas -o run_rms_tests.out rms.cpp

rms.cpp:17:10: fatal error: gsl/gsl_rng.h: No such file or directory
#include <gsl/gsl_rng.h>
^~~~~~~~~~~~~~~
compilation terminated.

from methylpy.

Got it. I believe you would need to install gsl library (https://www.gnu.org/software/gsl/). If you already installed it, you may want to check if the environmental variable is set correctly:
see https://stackoverflow.com/questions/22222666/error-while-loading-shared-libraries-libgsl-so-0-cannot-open-shared-object-fil

from methylpy.

I installed the gsl library and tried methylpy DMRfind. I got the following error:

Filtering allc files using 2 node(s).
Tue Oct 8 19:45:32 2019

Splitting allc files for chromosome wtMouseTGFBI
Tue Oct 8 19:45:32 2019

(<type 'exceptions.KeyError'>, 184)
'wtMouseTGFBI'
Running RMS tests failed.

is this because of the chromosome name "wtMouseTGFBI"? my sample is a construct made of one chromosome and that is how I labeled it in all of the allc files.

Thanks

from methylpy.

Were you able to run ./run_rms_tests.out without error? Do you mind to share a subset of the data for me to reproduce the error?

from methylpy.

this is the error I get when running the test:
./run_rms_tests.out: error while loading shared libraries: libgsl.so.0: cannot open shared object file: No such file or directory

I am attaching my FASTA reference along with one sample for you to run the folloiwng script:

methylpy DMRfind
--allc-files ResultsControlHighMe/allc_mCornea_H.tsv
ResultsControlLowMe/allc_mCornea_L.tsv
--samples HighMe_control LowMe_control
--mc-type "CGN"
--chroms wtMouseTGFBI
--num-procs 8
--min-num-dms 5
--output-prefix Results/CG_DMR_High_Low

ResultsControlLowMe.xlsx
ResultsControlHighMe.xlsx

wtMouseTGFBI.docx

from methylpy.

from methylpy.

previously, to install gsl I did the following:

cd /mnt/udrive/My_Work/softwares
wget ftp://ftp.gnu.org/gnu/gsl/gsl-2.6.tar.gz
tar -zxvf gsl-2.6.tar.gz
cd /mnt/udrive/My_Work/softwares/gsl-2.6
./configure --prefix=/mnt/udrive/My_Work/softwares/gsl-2.6
make
make check
make install

and I got the following error for all the "make" steps:

m: cannot remove 'gsl-config': Operation not permitted
make[1]: *** [Makefile:1559: gsl-config] Error 1
make[1]: Leaving directory '/mnt/udrive/My_Work/softwares/gsl-2.6'
make: *** [Makefile:963: check-recursive] Error 1

so probably the libraries gsl is not installed properly. Is that right?

from methylpy.

Yes. It looks like that GSL library was not installed properly. Alternatively, you may want to try installing GSL using conda (https://anaconda.org/conda-forge/gsl).

from methylpy.

I installed gsl using conda this time, but running the DMR test failed again:

Package Plan

environment location: U:\My_Work\softwares\miniConda

added / updated specs:
- gsl

The following packages will be downloaded:

package                    |            build
---------------------------|-----------------
ca-certificates-2019.9.11  |       hecc5488_0         181 KB  conda-forge
certifi-2019.9.11          |           py27_0         147 KB  conda-forge
conda-4.7.12               |           py27_0         3.0 MB  conda-forge
gsl-2.5                    |       hebfefe3_1         1.4 MB  conda-forge
libblas-3.8.0              |           13_mkl         3.5 MB  conda-forge
libcblas-3.8.0             |           13_mkl         3.5 MB  conda-forge
openssl-1.1.1c             |       h0c8e037_0         4.8 MB  conda-forge
------------------------------------------------------------
                                       Total:        16.6 MB

The following NEW packages will be INSTALLED:

gsl conda-forge/win-64::gsl-2.5-hebfefe3_1
libblas conda-forge/win-64::libblas-3.8.0-13_mkl
libcblas conda-forge/win-64::libcblas-3.8.0-13_mkl

The following packages will be UPDATED:

ca-certificates pkgs/main::ca-certificates-2019.8.28-0 --> conda-forge::ca-certificates-2019.9.11-hecc5488_0

The following packages will be SUPERSEDED by a higher-priority channel:

certifi pkgs/main --> conda-forge
conda pkgs/main --> conda-forge
openssl pkgs/main::openssl-1.1.1d-h0c8e037_2 --> conda-forge::openssl-1.1.1c-h0c8e037_0

Proceed ([y]/n)? y

Downloading and Extracting Packages
gsl-2.5 | 1.4 MB | ######################################################### | 100%
openssl-1.1.1c | 4.8 MB | ######################################################### | 100%
libblas-3.8.0 | 3.5 MB | ######################################################### | 100%
conda-4.7.12 | 3.0 MB | ######################################################### | 100%
certifi-2019.9.11 | 147 KB | ######################################################### | 100%
ca-certificates-2019 | 181 KB | ######################################################### | 100%
libcblas-3.8.0 | 3.5 MB | ######################################################### | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: done

/mnt/udrive/My_Work/Bioinf/methyl-Seq/mouseProject$ methylpy DMRfind
--allc-files ResultsControlHighMe/allc_mCornea_H.tsv
ResultsControlLowMe/allc_mCornea_L.tsv
--samples HighMe_control LowMe_control
--mc-type "CGN"
--chroms wtMouseTGFBI
--num-procs 8
--min-num-dms 5
--output-prefix Results/CG_DMR_High_Low

Filtering allc files using 2 node(s).
Fri Oct 11 13:40:38 2019

Splitting allc files for chromosome wtMouseTGFBI
Fri Oct 11 13:40:38 2019

(<type 'exceptions.KeyError'>, 184)
'wtMouseTGFBI'
Running RMS tests failed.

/mnt/udrive/My_Work/Bioinf/methyl-Seq/methylpy/methylpy$ ./run_rms_tests.out
./run_rms_tests.out: error while loading shared libraries: libgsl.so.0: cannot open shared object file: No such file or directory

could that be the path? Now im manually trying to transfer the miniConda to where I have ./run_rms_tests.out saved (/mnt/udrive/My_Work/Bioinf/methyl-Seq/methylpy/methylpy), still doesnt work:

_mnt/udrive/My_Work/Bioinf/methyl-Seq/methylpy/methylpy$_ ls
call_mc_pe.py DMRfind.py miniConda rms.cpp utilities.py
call_mc_se.py init.py parser.py run_rms_tests.out
_/mnt/udrive/My_Work/Bioinf/methyl-Seq/methylpy/methylpy$_ ./run_rms_tests.out
./run_rms_tests.out: error while loading shared libraries: libgsl.so.0: cannot open shared object file: No such file or directory

from methylpy.

If GSL is installed successfully, the error is because the location of GSL library file is unknown to the system. The solution is to set the LD_LIBRARY_PATH variable to include the path to the library GSL library file.

Do you see libgsl.so.0 or libgsl.so.* files in folder like U:\My_Work\softwares\miniConda\lib\?

If that is the case, you should be able to fix the error for the current terminal by doing something like:

export LD_LIBRARY_PATH=U:\My_Work\softwares\miniConda\lib\:$LD_LIBRARY_PATH

Adding this line of command to your ~/.bashrc file (assuming you are using bash) and restarting the terminal should fix the problem.

You can check out this post:
https://stackoverflow.com/questions/22222666/error-while-loading-shared-libraries-libgsl-so-0-cannot-open-shared-object-fil

from methylpy.

Thanks, I opened a new terminal and here is what I did:

~/methylpy$ ls
call_mc_pe.py DMRfind.py miniConda rms.cpp utilities.py
call_mc_se.py init.py parser.py run_rms_tests.out

~/methylpy$ LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib
export LD_LIBRARY_PATH
./run_rms_tests.out
Error:
Usage: ./rms.out <min_cov> <num_sims> <num_sig_tests>

The DMR tests failed again, do you think dependencies such as "wigToBigWig" is an issue? because I am not able to install it,
test_error_msg.txt
test_output_msg.txt

"set up channel in miniConda"
conda config --add channels defaults
conda config --add channels bioconda
conda config --add channels conda-forge

conda install -c bioconda ucsc-wigtobigwig
conda install -c bioconda/label/cf201901 ucsc-wigtobigwig

Error:
PackagesNotFoundError: The following packages are not available from current channels:

• ucsc-wigtobigwig

Current channels:

• https://conda.anaconda.org/bioconda/label/cf201901/win-64
• https://conda.anaconda.org/bioconda/label/cf201901/noarch
• https://conda.anaconda.org/bioconda/win-64
• https://conda.anaconda.org/bioconda/noarch
• https://conda.anaconda.org/conda-forge/win-64
• https://conda.anaconda.org/conda-forge/noarch
• https://repo.anaconda.com/pkgs/main/win-64
• https://repo.anaconda.com/pkgs/main/noarch
• https://repo.anaconda.com/pkgs/r/win-64
• https://repo.anaconda.com/pkgs/r/noarch
• https://repo.anaconda.com/pkgs/msys2/win-64
• https://repo.anaconda.com/pkgs/msys2/noarch

To search for alternate channels that may provide the conda package you're
looking for, navigate to

https://anaconda.org

and use the search bar at the top of the page.

from methylpy.

I don't think wigtobigwig is the issue. The issue is still that the system is unable to find the gsl library file.

Did you test methylpy after running the command or before?

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib

If you add the above line to your ~/.bashrc file and open up a new terminal for testing, would it help?

What is the full output when you ran ./run_rms_tests.out after running the above command?

from methylpy.

to answer your questions:
Did you test methylpy after running the command or before?

• yes, I tried before and after and got the following error:
  Filtering allc files using 2 node(s).
  Tue Oct 15 14:42:35 2019

Splitting allc files for chromosome wtMouseTGFBI
Tue Oct 15 14:42:35 2019

(<type 'exceptions.KeyError'>, 184)
'wtMouseTGFBI'
Running RMS tests failed.

If you add the above line to your ~/.bashrc file and open up a new terminal for testing, would it help?
-I am new to linux and am not familiar with writing a .bashrc file.

What is the full output when you ran ./run_rms_tests.out after running the above command?
-Usage: ./rms.out <min_cov> <num_sims> <num_sig_tests>

from methylpy.

Thanks.

1. Do you know if you use bash, zsh or other version of shell? If you use bash, then the terminal will automatically set up environmental variables based on ~/.bashrc file where ~ is a short cut to your home directly. For adding the above line to your ~/.bashrc file, you can do

cd ~/
emacs .bashrc

You can use other text editor like vim too. If you are not familiar with command line, you can do

cd ~/
pwd

and then go to the printed out folder and create/modify the .bashrc file with GUI text editor like notepad.

Then add the code to the .bashrc file (if it does not exist, please create one).

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib

Last run

source .bashrc

2. From the test error file, the methylpy you installed is using

/home/hilar/.local/lib/python2.7/site-packages/methylpy/run_rms_tests.out

What message would you get if you run

/home/hilar/.local/lib/python2.7/site-packages/methylpy/run_rms_tests.out

before and after you run the "export xxx" command?

3. If you get an error like below from Step 2, can you try to recompile the file with

g++ -O3 -l gsl -l gslcblas -o /home/hilar/.local/lib/python2.7/site-packages/methylpy/run_rms_tests.out
 /home/hilar/.local/lib/python2.7/site-packages/methylpy/rms.cpp

from methylpy.

Thank you for taking time to help me with this, I do really appreciate your help.
I am using GNU bash, version 5.0.2(1)-release (x86_64-pc-linux-gnu).
I used emacs command, a new page opened up and I added/saved the export xxxx to the .bashrc. There were other export commands in that file (below), so I added your line at the end:

export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/usr/local/include
export PPH=/home/hilar/Programs/polyphen-2.2.2
export PATH=$PATH:$PPH/bin
export PATH=$PATH:/home/hilar/Programs/vcf2db:/home/hilar/Programs/vcf2maf:/home/hilar/Programs/vcfanno
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib

then in the terminal I typed source .bashrc (nothing happened). Then ran the test again and got the same error. the test gave me the same error before adding this line as well. please see below:

hilar@kali:$ /home/hilar/.local/lib/python2.7/site-packages/methylpy/run_rms_tests.out
Usage: ./rms.out <min_cov> <num_sims> <num_sig_tests>
hilar@kali:$ cd /
hilar@kali:$ emacs .bashrc
hilar@kali:$ source .bashrc
hilar@kali:$ /home/hilar/.local/lib/python2.7/site-packages/methylpy/run_rms_tests.out
Usage: ./rms.out <min_cov> <num_sims> <num_sig_tests>

3. recompiling the file failed as well:

hilar@kali:~$ g++ -O3 -l gsl -l gslcblas -o /home/hilar/.local/lib/python2.7/site-packages/methylpy/run_rms_tests.out
/usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/8/../../../x86_64-linux-gnu/Scrt1.o: in function _start': (.text+0x20): undefined reference to main'
collect2: error: ld returned 1 exit status

from methylpy.

Thanks for the update. I am happy to help. What OS are you using?

There are a few issues that are quite confusing.

1. The default printed message of "run_rms_tests.out" is weird. The full message should be

Usage: ./rms.out <chunk files> <output file> <samples> <min_cov> <num_sims> <num_sig_tests> <seed>

I checked the code and I cannot imagine how only part of the message was printed out.

Nonetheless, since you can get some help message printed out, we should be able to test this binary file separately. Can you try the below commands?

wget http://neomorph.salk.edu/yupeng/share/test_dmr.tar.gz
tar xf test_dmr.tar.gz
/home/hilar/.local/lib/python2.7/site-packages/methylpy/run_rms_tests.out allc_P0_FB_1.tsv,allc_P0_HT_1.tsv test_output.txt A,B 1 1000 100 -1

If the command can be run without error, we can be sure that the run_rms_tests.out is working well.

2. When you tried to recompile the run_rms_tests.out, the command missed a key input parameter. Can you try the below command?

g++ -O3 -l gsl -l gslcblas -o /home/hilar/.local/lib/python2.7/site-packages/methylpy/run_rms_tests.out METHYLPY_PATH/rms.cpp

where METHYLPY_PATH/rms.cpp is the path to the CPP file under methylpy/methylpy
https://github.com/yupenghe/methylpy/blob/methylpy/methylpy/rms.cpp

If you are using Ubuntu, please try the below command too if the above does not work

g++ -o /home/hilar/.local/lib/python2.7/site-packages/methylpy/run_rms_tests.out METHYLPY_PATH/rms.cpp `gsl-config --cflags —libs`

Sorry it has been a hassle to just have DMRfind working.

Yupeng

from methylpy.

OS is Windows 10 Pro, 64-bit and I am using Kali-Linux oracle VM to run these scripts

1. command aborted:

hilar@kali:/mnt/udrive/My_Work/Bioinf$ wget http://neomorph.salk.edu/yupeng/share/test_dmr.tar.gz
--2019-10-17 16:06:39-- http://neomorph.salk.edu/yupeng/share/test_dmr.tar.gz
Resolving neomorph.salk.edu (neomorph.salk.edu)... 198.202.69.49
Connecting to neomorph.salk.edu (neomorph.salk.edu)|198.202.69.49|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 499837 (488K) [application/x-gzip]
Saving to: ‘test_dmr.tar.gz’

test_dmr.tar.gz 100%[====================================>] 488.12K 2.49MB/s in 0.2s

2019-10-17 16:06:40 (2.49 MB/s) - ‘test_dmr.tar.gz’ saved [499837/499837]

hilar@kali:/mnt/udrive/My_Work/Bioinf$ tar xf test_dmr.tar.gz
hilar@kali:/mnt/udrive/My_Work/Bioinf$ /home/hilar/.local/lib/python2.7/site-packages/methylpy/run_rms_tests.out allc_P0_FB_1.tsv,allc_P0_HT_1.tsv test_output.txt A,B 1 1000 100 -1
free(): double free detected in tcache 2
Aborted

2. I will do this tomorrow, our system is down now.

from methylpy.

Ok. Please let me know if you have any luck with 2. Btw, do you have any experience working with docker? It could be much easier solution for this issue. It will take a few days but I can prepare a docker image with methylpy installed for you. Within the docker, you should be able to run DMRfind without any further setup.

from methylpy.

Thanks Yupeng. "docker" solution is great. I have not worked with it before, but I start learning now. It would be great to have all the methylpy pipeline in one place, would you please add all the alignment and other pre-processing steps in addition to DMRfind function?

this is what I got for question 2:

hilar@kali:/mnt/udrive/My_Work/softwares$ g++ -O3 -l gsl -l gslcblas -o /home/hilar/.local/lib/python2.7/site-packages/methylpy/run_rms_tests.out /mnt/udrive/My_Work/Bioinf/methyl-Seq/methylpy/methylpy/rms.cpp

/mnt/udrive/My_Work/Bioinf/methyl-Seq/methylpy/methylpy/rms.cpp: In function ‘int rms_test(std::vector<std::vector >, float*, std::vector<std::vector >, std::vector<std::vector >)’:
/mnt/udrive/My_Work/Bioinf/methyl-Seq/methylpy/methylpy/rms.cpp:88:1: warning: no return statement in function returning non-void [-Wreturn-type]
}
^

from methylpy.

Hey I add docker solution for running methylpy. Please check it out:
https://github.com/yupenghe/methylpy#use-methylpy-without-installation

This should resolve the DMRfind issue. I close this issue and please feel free to reopen it if there is still issue in running DMRfind.

from methylpy.