Git Product home page Git Product logo

Comments (14)

MikkelSchubert avatar MikkelSchubert commented on July 29, 2024

Hi,

I'll try to look into this tomorrow, but it would be helpful if you could answer a couple of questions:

  1. Is the conda environment active when you try to run python rmdup_collapsed.py?
  2. And what happens if you try to run the script the normal way, namely via paleomix rmdup_collapsed?

The scripts included in paleomix are generally not meant to be run directly, though the problem you're experiencing is not the kind I would have expected to see.

Cheers,
Mikkel

from paleomix.

sagitaninta avatar sagitaninta commented on July 29, 2024

Hi, thanks for your prompt reply. Yeah, the error message is different when it is because I ran it separately, just that it cannot find paleomix module.

  1. Yes the conda environment is active when all of this happened.
  2. Here is the result:
paleomix rmdup_collapsed
Traceback (most recent call last):
  File "/home/btx638/software/miniconda3/envs/paleomix/bin/paleomix", line 5, in <module>
    from paleomix.main import entry_point
  File "/home/btx638/software/miniconda3/envs/paleomix/lib/python3.6/site-packages/paleomix/main.py", line 25, in <module>
    import pysam
  File "/home/btx638/software/miniconda3/envs/paleomix/lib/python3.6/site-packages/pysam/__init__.py", line 5, in <module>
    from pysam.libchtslib import *
ImportError: libhts.so.2: cannot open shared object file: No such file or directory

I was looking also at my own dir and found that I have pysam and libhts.so.2 in multiple env but not sure how to solve it if this is the problem (still in conda activate paleomix):

find ~/ -name "libhts.so.2"
/home/btx638/software/miniconda3/envs/vep_env/lib/libhts.so.2
/home/btx638/software/miniconda3/pkgs/htslib-1.9-h4da6232_3/lib/libhts.so.2
/home/btx638/software/miniconda3/pkgs/htslib-1.9-ha228f0b_7/lib/libhts.so.2
/home/btx638/software/ensembl-vep/htslib/libhts.so.2
/home/btx638/software/freebayes/SeqLib/htslib/libhts.so.2
/home/btx638/smcpp/lib/libhts.so.2
/home/btx638/htslib/libhts.so.2

find ~/ -name "pysam"
/home/btx638/software/miniconda3/envs/paleomix/lib/python3.6/site-packages/pysam
/home/btx638/software/miniconda3/pkgs/pysam-0.15.3-py37hda2845c_1/lib/python3.7/site-packages/pysam
/home/btx638/software/miniconda3/pkgs/pysam-0.11.2.2-py36_1/lib/python3.6/site-packages/pysam
/home/btx638/software/miniconda3/lib/python3.7/site-packages/pysam
/home/btx638/smcpp/pkgs/pysam-0.11.2.2-py36_1/lib/python3.6/site-packages/pysam
/home/btx638/smcpp/lib/python3.6/site-packages/pysam

Let me know if you need anything else.

Best,
Sabhrina

from paleomix.

MikkelSchubert avatar MikkelSchubert commented on July 29, 2024

Hi and sorry for the delay.

The problem seems to be caused by breaking changes in some conda packages that aren't handled as such by the package manager, resulting in an non-functional combination of programs and libraries. You have the htslib installed, just not the version that pysam was looking for.

I've attached an updated conda environment file, which should produce a working paleomix environment:
paleomix_environment.yaml.gz

The file is zipped because github doesn't allow yaml files to be attached, so you'll have to unzip it first.
Also, remember to remove the old conda environment before installing the updated version:

gunzip paleomix_environment.yaml.gz
conda env remove -n paleomix
conda env create -n paleomix -f paleomix_environment.yaml

Please let me know if that also works for you.

Cheers,
Mikkel

from paleomix.

ztang040 avatar ztang040 commented on July 29, 2024

Hi Mikkel,
This ImportError is also an error that I had encountered during my study.
I had followed the comment from you above to de-install and re-install paleomix.
But the error still exists,
Here is the outcome of my script job, which is quite similar to sagitaninta's issue

(paleomix) [ztang@gruffalo project]$ cat slurm-544598.out
/var/spool/slurm/job544598/slurm_script: line 10: activate: No such file or directory
Traceback (most recent call last):
File "/home/ztang/miniconda3/envs/paleomix/bin/paleomix", line 5, in
from paleomix.main import entry_point
File "/home/ztang/miniconda3/envs/paleomix/lib/python3.6/site-packages/paleomix/main.py", line 25, in
import pysam
File "/home/ztang/miniconda3/envs/paleomix/lib/python3.6/site-packages/pysam/init.py", line 5, in
from pysam.libchtslib import *
ImportError: libhts.so.2: cannot open shared object file: No such file or directory

Could you provide any alternative solvent for this error please? Thanks in advance

from paleomix.

MikkelSchubert avatar MikkelSchubert commented on July 29, 2024

Hi,
Thank you for letting me know.

Can you try this conda environment file instead? paleomix_environment.yaml.gz

As before, you'll probably need to remove your existing conda environment first:

gunzip paleomix_environment.yaml.gz
conda env remove -n paleomix
conda env create -n paleomix -f paleomix_environment.yaml

Cheers

from paleomix.

ztang040 avatar ztang040 commented on July 29, 2024

Hi,
Yes I have already finished those before the last attempt for my script.
But the error is still the same.

By the way, the file inside this link "https://github.com/MikkelSchubert/paleomix/files/5965990/paleomix_environment.yaml.gz"
is already "paleomix_environment.yaml"
It can be installed smoothly, but it seems didn't make change for this error...
I have no idea where the problem is because everything seems well for the environment installation.
If you have any idea where is the problem, please give me some hints.

Cheers

from paleomix.

MikkelSchubert avatar MikkelSchubert commented on July 29, 2024

Hi,

Just to be clear:
The file I attached in my reply to you is different than the first file I attached in my reply to sagitaninta.
I should probably have given it a different filename, but unfortunately I did not think that far.

Please try to remove the paleomix_environment.yaml* files you already have, and follow the instructions above using this file:
https://github.com/MikkelSchubert/paleomix/files/5965990/paleomix_environment.yaml.gz

Cheers

from paleomix.

ztang040 avatar ztang040 commented on July 29, 2024

Hi Mikkel,

Thank you for your remind.
I would try to run this one now.

Cheers,
Zitian

from paleomix.

ztang040 avatar ztang040 commented on July 29, 2024

Hi Mikkel,
I am back. I have re-installed again using your last link, but regret to say the result is the same.

(paleomix) [ztang@gruffalo project]$ sbatch script_paleomix.sh
Submitted batch job 544847
(paleomix) [ztang@gruffalo project]$ sacct
JobID JobName Partition Account AllocCPUS State ExitCode

544442 paleomix gpu cropdiv-a+ 8 FAILED 1:0
544442.batch batch cropdiv-a+ 8 FAILED 1:0
544442.exte+ extern cropdiv-a+ 8 COMPLETED 0:0
544444 test job medium cropdiv-a+ 1 COMPLETED 0:0
544444.batch batch cropdiv-a+ 1 COMPLETED 0:0
544444.exte+ extern cropdiv-a+ 1 COMPLETED 0:0
544847 paleomix gpu cropdiv-a+ 8 FAILED 1:0
544847.batch batch cropdiv-a+ 8 FAILED 1:0
544847.exte+ extern cropdiv-a+ 8 COMPLETED 0:0

Test job could be completed easily, but when it comes to paleomix script "ImportError: libhts.so.2: cannot open shared object file: No such file or directory" would come again.

Sincerely,
Zitian

from paleomix.

MikkelSchubert avatar MikkelSchubert commented on July 29, 2024

Hi,

Can you try to run the following command and attach the output?

conda list -n paleomix

Can you also tell me the exactly what your script_paleomix.sh script does?

Cheers

from paleomix.

ztang040 avatar ztang040 commented on July 29, 2024

Sure

(paleomix) [ztang@gruffalo project]$ conda list -n paleomix

packages in environment at /home/ztang/miniconda3/envs/paleomix:

Name Version Build Channel

_libgcc_mutex 0.1 conda_forge conda-forge
_openmp_mutex 4.5 1_gnu conda-forge
_r-mutex 1.0.0 anacondar_1
adapterremoval 2.3.1 hb7ba0dd_1 bioconda
bcftools 1.11 h7c999a4_0 bioconda
binutils_impl_linux-64 2.35.1 h193b22a_2 conda-forge
binutils_linux-64 2.35 hc3fd857_29 conda-forge
bowtie2 2.4.2 py36h5202f60_1 bioconda
bwa 0.7.17 hed695b0_7 bioconda
bwidget 1.9.11 1
bzip2 1.0.8 h7b6447c_0
c-ares 1.17.1 h27cfd23_0
ca-certificates 2021.1.19 h06a4308_0
cairo 1.16.0 h7979940_1007 conda-forge
certifi 2020.12.5 py36h06a4308_0
coloredlogs 15.0 pypi_0 pypi
configargparse 1.2.3 pypi_0 pypi
curl 7.71.1 he644dc0_8 conda-forge
fontconfig 2.13.1 hba837de_1004 conda-forge
freetype 2.10.4 h5ab3b9f_0
fribidi 1.0.10 h7b6447c_0
gcc_impl_linux-64 9.3.0 h70c0ae5_18 conda-forge
gcc_linux-64 9.3.0 h7247604_29 conda-forge
gettext 0.19.8.1 h9b4dc7a_1
gfortran_impl_linux-64 9.3.0 hc4a2995_18 conda-forge
gfortran_linux-64 9.3.0 ha1c937c_29 conda-forge
graphite2 1.3.14 h23475e2_0
gsl 2.6 he838d99_2 conda-forge
gxx_impl_linux-64 9.3.0 hd87eabc_18 conda-forge
gxx_linux-64 9.3.0 h0d07fa4_29 conda-forge
harfbuzz 2.7.4 h5cf4720_0 conda-forge
htslib 1.11 hd3b49d5_2 bioconda
humanfriendly 9.1 pypi_0 pypi
icu 68.1 h2531618_0
jpeg 9d h36c2ea0_0 conda-forge
kernel-headers_linux-64 2.6.32 h77966d4_13 conda-forge
krb5 1.17.1 h173b8e3_0
ld_impl_linux-64 2.35.1 hea4e1c9_2 conda-forge
libblas 3.9.0 8_openblas conda-forge
libcblas 3.9.0 8_openblas conda-forge
libcurl 7.71.1 hcdd3856_8 conda-forge
libdeflate 1.7 h27cfd23_5
libedit 3.1.20191231 h14c3975_1
libev 4.33 h7b6447c_0
libffi 3.3 he6710b0_2
libgcc 7.2.0 h69d50b8_2
libgcc-devel_linux-64 9.3.0 h7864c58_18 conda-forge
libgcc-ng 9.3.0 h2828fa1_18 conda-forge
libgfortran-ng 9.3.0 hff62375_18 conda-forge
libgfortran5 9.3.0 hff62375_18 conda-forge
libglib 2.66.6 h1f3bc88_3 conda-forge
libgomp 9.3.0 h2828fa1_18 conda-forge
libiconv 1.16 h516909a_0 conda-forge
liblapack 3.9.0 8_openblas conda-forge
libnghttp2 1.41.0 hf8bcb03_2
libopenblas 0.3.12 pthreads_h4812303_1 conda-forge
libpng 1.6.37 hbc83047_0
libssh2 1.9.0 h1ba5d50_1
libstdcxx-devel_linux-64 9.3.0 hb016644_18 conda-forge
libstdcxx-ng 9.3.0 h6de172a_18 conda-forge
libtiff 4.1.0 h2733197_1
libuuid 2.32.1 h7f98852_1000 conda-forge
libxcb 1.14 h7b6447c_0
libxml2 2.9.10 h72842e0_3 conda-forge
lz4-c 1.9.3 h2531618_0
make 4.2.1 h1bed415_1
mapdamage2 2.2.1 pyr40_0 bioconda
ncurses 6.2 he6710b0_1
openjdk 8.0.152 h7b6447c_3
openssl 1.1.1i h27cfd23_0
paleomix 1.3.2 pypi_0 pypi
pango 1.42.4 h69149e4_5 conda-forge
pcre 8.44 he6710b0_0
pcre2 10.36 h032f7d1_1 conda-forge
perl 5.26.2 h14c3975_0
picard 2.25.0 0 bioconda
pip 20.3.3 py36h06a4308_0
pixman 0.40.0 h7b6447c_0
pysam 0.11.2.2 py36_1 bioconda
python 3.6.12 hcff3b4d_2
python_abi 3.6 1_cp36m conda-forge
r-assertthat 0.2.1 r40h6115d3f_2 conda-forge
r-backports 1.2.1 r40hcfec24a_0 conda-forge
r-base 4.0.3 hd23ff56_6 conda-forge
r-brio 1.1.1 r40hcfec24a_0 conda-forge
r-callr 3.5.1 r40h142f84f_0 conda-forge
r-cli 2.3.0 r40hc72bb7e_0 conda-forge
r-codetools 0.2_18 r40hc72bb7e_0 conda-forge
r-colorspace 2.0_0 r40h9e2df91_0 conda-forge
r-crayon 1.4.1 r40hc72bb7e_0 conda-forge
r-desc 1.2.0 r40h6115d3f_1003 conda-forge
r-diffobj 0.3.3 r40hcfec24a_0 conda-forge
r-digest 0.6.27 r40h1b71b39_0 conda-forge
r-ellipsis 0.3.1 r40hcdcec82_0 conda-forge
r-evaluate 0.14 r40h6115d3f_2 conda-forge
r-fansi 0.4.2 r40hcfec24a_0 conda-forge
r-farver 2.0.3 r40h0357c0b_1 conda-forge
r-foreach 1.5.1 r40h142f84f_0 conda-forge
r-gam 1.20 r40h580db52_1 conda-forge
r-ggplot2 3.3.3 r40hc72bb7e_0 conda-forge
r-glue 1.4.2 r40hcfec24a_0 conda-forge
r-gtable 0.3.0 r40h6115d3f_3 conda-forge
r-inline 0.3.17 r40hc72bb7e_0 conda-forge
r-isoband 0.2.3 r40h03ef668_0 conda-forge
r-iterators 1.0.13 r40h142f84f_0 conda-forge
r-jsonlite 1.7.2 r40hcfec24a_0 conda-forge
r-labeling 0.4.2 r40h142f84f_0 conda-forge
r-lattice 0.20_41 r40hcfec24a_3 conda-forge
r-lifecycle 0.2.0 r40h6115d3f_1 conda-forge
r-magrittr 2.0.1 r40hcfec24a_1 conda-forge
r-mass 7.3_53 r40hcfec24a_0 conda-forge
r-matrix 1.3_2 r40he454529_0 conda-forge
r-mgcv 1.8_33 r40he454529_0 conda-forge
r-munsell 0.5.0 r40h6115d3f_1003 conda-forge
r-nlme 3.1_152 r40h859d828_0 conda-forge
r-pillar 1.4.7 r40hc72bb7e_0 conda-forge
r-pkgbuild 1.2.0 r40hc72bb7e_0 conda-forge
r-pkgconfig 2.0.3 r40h6115d3f_1 conda-forge
r-pkgload 1.1.0 r40h0357c0b_0 conda-forge
r-praise 1.0.0 r40h6115d3f_1004 conda-forge
r-prettyunits 1.1.1 r40h6115d3f_1 conda-forge
r-processx 3.4.5 r40hcfec24a_0 conda-forge
r-ps 1.5.0 r40hcfec24a_0 conda-forge
r-r6 2.5.0 r40hc72bb7e_0 conda-forge
r-rcolorbrewer 1.1_2 r40h6115d3f_1003 conda-forge
r-rcpp 1.0.6 r40h03ef668_0 conda-forge
r-rcppgsl 0.3.8 r40hc8faad4_0 conda-forge
r-rematch2 2.1.2 r40h6115d3f_1 conda-forge
r-rlang 0.4.10 r40hcfec24a_0 conda-forge
r-rprojroot 2.0.2 r40hc72bb7e_0 conda-forge
r-rstudioapi 0.13 r40hc72bb7e_0 conda-forge
r-scales 1.1.1 r40h6115d3f_0 conda-forge
r-testthat 3.0.1 r40h03ef668_0 conda-forge
r-tibble 3.0.6 r40hcfec24a_0 conda-forge
r-utf8 1.1.4 r40hcdcec82_1003 conda-forge
r-vctrs 0.3.6 r40hcfec24a_0 conda-forge
r-viridislite 0.3.0 r40h6115d3f_1003 conda-forge
r-waldo 0.2.3 r40hc72bb7e_0 conda-forge
r-withr 2.4.1 r40hc72bb7e_0 conda-forge
r-zeallot 0.1.0 r40h6115d3f_1002 conda-forge
readline 8.1 h27cfd23_0
ruamel-yaml 0.16.12 pypi_0 pypi
ruamel-yaml-clib 0.2.2 pypi_0 pypi
samtools 1.11 h6270b1f_0 bioconda
sed 4.8 h7b6447c_0
seqtk 1.3 hed695b0_2 bioconda
setproctitle 1.2.2 pypi_0 pypi
setuptools 52.0.0 py36h06a4308_0
sqlite 3.33.0 h62c20be_0
sysroot_linux-64 2.12 h77966d4_13 conda-forge
tbb 2020.3 hfd86e86_0
tk 8.6.10 hbc83047_0
tktable 2.10 h14c3975_0
wheel 0.36.2 pyhd3eb1b0_0
xorg-kbproto 1.0.7 h7f98852_1002 conda-forge
xorg-libice 1.0.10 h516909a_0 conda-forge
xorg-libsm 1.2.3 h84519dc_1000 conda-forge
xorg-libx11 1.6.12 h516909a_0 conda-forge
xorg-libxext 1.3.4 h516909a_0 conda-forge
xorg-libxrender 0.9.10 h516909a_1002 conda-forge
xorg-libxt 1.2.1 h7f98852_0 conda-forge
xorg-renderproto 0.11.1 h14c3975_1002 conda-forge
xorg-xextproto 7.3.0 h7f98852_1002 conda-forge
xorg-xproto 7.0.31 h27cfd23_1007
xz 5.2.5 h7b6447c_0
zlib 1.2.11 h7b6447c_3
zstd 1.4.5 h9ceee32_0

The script_paleomix.sh script:

#!/bin/bash

#SBATCH --job-name="paleomix"
#SBATCH --export=ALL
#SBATCH --partition=gpu --mem=16G --cpus-per-task=8

Do some work:

source activate paleomix
paleomix bam run Thibauld_makefile.yaml --jre-option=-Xmx50g

Additionally, just to confirm it, are the updated .yaml document inside like this:

channels:

  • defaults
  • bioconda
  • conda-forge
    dependencies:
  • pip
  • adapterremoval==2.3.1
  • samtools==1.6
  • picard==2.24.2
  • bowtie2==2.3.0
  • bwa==0.7.17
  • mapdamage2==2.2.1
  • pysam==0.16.0
  • r-base
  • r-rcpp
  • r-rcppgsl
  • r-gam
  • r-inline
  • pip:
    • paleomix==1.3.2

from paleomix.

MikkelSchubert avatar MikkelSchubert commented on July 29, 2024

Hi again,

I am not sure what is going on, but your conda environment appears to be that produced by the original paleomix_environment.yaml file linked in the the paleomix documentation. For example, you have bowtie 2.4.2, but the version specified in the updated paleomix_environment.yaml is 2.3.0. More importantly, you have pysam 0.12.2.2, but you should be having 0.16.0. That is what is causing the problem you have.

I've tested with multiple versions of conda/minicona, but I am not able to reproduce this using the latest paleomix_environment.yaml file.

Can you try running the following and attach the output?

rm -fv paleomix_environment.yaml*
curl -L https://github.com/MikkelSchubert/paleomix/files/5965990/paleomix_environment.yaml.gz | gunzip > paleomix_environment.yaml
cat paleomix_environment.yaml
conda env remove -n paleomix
conda env create -n paleomix -f paleomix_environment.yaml
conda list -n paleomix

Cheers

from paleomix.

ztang040 avatar ztang040 commented on July 29, 2024

from paleomix.

ztang040 avatar ztang040 commented on July 29, 2024

from paleomix.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.