nanoporetech / katuali Goto Github PK

View Code? Open in Web Editor NEW

50.0 50.0 7.0 34.19 MB

Analysis pipelines from Oxford Nanopore Technologies' Research Division

License: Other

Makefile 19.02% Python 71.04% Shell 9.94%

katuali's People

Contributors

Stargazers

Watchers

Forkers

skerker pythseq vikash84 julianzaugg tclin422 sunnyev gloriamhl

katuali's Issues

Training medaka variant calling model

Hello,

I'd like to train some new medaka variant calling models.
For example, i'd like to train on for human. 2 questions.

How do I go about this, and what data do I need?
can I train on just a region of a human genome?

Any further information on how the model is created would be awesome.

Thanks!

Error on Installation of Katuali

Hi,

I am trying to install Katuali. I already have snakemake installed but on running the command "make install" to run the makefile I get the following error below in the last 10 lines on the installation script.
May you please assist me?

ERROR RECEIVED IN LAST 10 LINES OF INSTALLATION SCRIPT:

Writing /tmp/easy_install-0uhyrzri/datrie-0.7.1/setup.cfg
Running datrie-0.7.1/setup.py -q bdist_egg --dist-dir /tmp/easy_install-0uhyrzri/datrie-0.7.1/egg-dist-tmp-jo41x002
warning: no files found matching 'tests/words100k.txt.zip'
src/datrie.c:4:10: fatal error: Python.h: No such file or directory
#include "Python.h"
compilation terminated.
error: Setup script exited with error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
Makefile:22: recipe for target 'install' failed
make: *** [install] Error 1

barcoding

Dear nanoporetech team,
thank you very much for an inspiring tool which seems to work fine. Not sure if this is an issue or rather feature request: What is the best way to handle barcoded data with katuali? For technical reasons, we currently do base calling after sequencing not during. I might add guppy params to the config to do de-barcoding (eg. --barcode_kits EXP-NBD104) but how do I tell katuali to run assembly and polishing for each barcode separately?

Thanks

virtualenv requirement not explicit

We should explicitly add virtualenv to the dependencies.

Starting from existing basecalls docu not sufficient

Hi,

I just ran into an issue, that I can't create a custom pipeline from existing basecalls. There is a documentation about what to edit, but I have no clue where to edit it. Would be nice if someone can help me out here.

Thanks!

Issue with combining fastq files

Hello, so from what I can tell I have all of the preinstallation done correctly, but when I run
sudo make test

It ends up failing in the guppy step, however the weird part is it does successfully basecall. I will include a log below. The important part to look at are the bottom 8 lines, where basecalling is completed successfully but it fails after that (however im including the full log it to show that guppy and pomoxis are found by the check_file portion).

[22:43:19 - check_file] Checking file on kgallman: /usr/bin/guppy_basecaller
[22:43:19 - check_file] File is symlink, following link. File: /usr/bin/guppy_basecaller
[22:43:19 - check_file] Checking file on kgallman: /opt/ont/guppy/bin/guppy_basecaller
[22:43:19 - check_file] File exists! File: /opt/ont/guppy/bin/guppy_basecaller Size: 306759232
[22:43:19 - check_file] Checking file on kgallman: /home/kazengallman/git/pomoxis/venv/bin/activate
[22:43:19 - check_file] File exists! File: /home/kazengallman/git/pomoxis/venv/bin/activate Size: 2227
[22:43:19 - check_file] Checking file on kgallman: MinIonRun1/reads
[22:43:19 - check_file] File exists! File: MinIonRun1/reads Size: 12288
[22:43:19 - check_files] Finished checking that input files exist.
GPU status before
kgallman                Sun May 16 22:43:20 2021  455.45.01
[0] GeForce GTX 1050 Ti | 49°C,   0 % |   232 /  4036 MB | gdm(35M) kazengallman(145M) kazengallman(32M) kazengallman(1M) kazengallman(1M) kazengallman(1M) kazengallman(1M) kazengallman(1M) kazengallman(1M)
[22:43:39 - pick_gpu] SGE_HGR_gpu was not set, setting GPU to 0 based on memory and utilization
Runnning on host kgallman GPU 0
ONT Guppy basecalling software version 4.5.4+66c1a7753
config file:        /opt/ont/guppy/data/dna_r9.4.1_450bps_hac.cfg
model file:         /opt/ont/guppy/data/template_r9.4.1_450bps_hac.jsn
input path:         MinIonRun1/reads
save path:          MinIonRun1/guppy
chunk size:         2000
chunks per runner:  512
minimum qscore:     9
records per file:   4000
num basecallers:    4
gpu device:         cuda:0
kernel path:        
runners per device: 4
Found 162 fast5 files to process.
Init time: 528 ms

0%   10   20   30   40   50   60   70   80   90   100%
|----|----|----|----|----|----|----|----|----|----|
***************************************************
Caller time: 37127 ms, Samples called: 14237088, samples/s: 383470
Finishing up any open output files.
Basecalling completed successfully.
gpustat after
kgallman                Sun May 16 22:44:20 2021  455.45.01
[0] GeForce GTX 1050 Ti | 56°C,  56 % |   227 /  4036 MB | gdm(35M) kazengallman(133M) kazengallman(38M) kazengallman(1M) kazengallman(1M) kazengallman(1M) kazengallman(1M) kazengallman(1M) kazengallman(1M)
Combining the following fastq files into MinIonRun1/guppy/basecalls.fastq.gz

Here is the snakemake log before failing

sleep 5
set +u; source /home/kazengallman/git/pomoxis/venv/bin/activate; set -u;
echo "Combining the following fastq files into MinIonRun1/guppy/basecalls.fastq.gz" >> MinIonRun1/guppy.log
ls MinIonRun1/guppy/*.fastq >> MinIonRun1/guppy.log
cat MinIonRun1/guppy/*.fastq | bgzip -@ 4 -c >MinIonRun1/guppy/basecalls.fastq.gz 2>> MinIonRun1/guppy.log
rm MinIonRun1/guppy/*.fastq

        (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

Removing output files of failed job basecall_guppy since they might be corrupted:
MinIonRun1/guppy/sequencing_summary.txt
Job failed, going on with independent jobs.
Exiting because a job execution failed. Look above for error message
Complete log: /home/kazengallman/Desktop/katuali-test/katuali/test/.snakemake/log/2021-05-16T224319.536460.snakemake.log

So overall I am just confused what is causing it to fail, since unless im mistaken it does not exactly show the command that it failed on (I am assuming the command that it broke on is after the rm MinIonRun1/guppy/*.fastq)

command not found error even if it is in $PATH

Hi,

I just ran into an issue where the logfile said that the command mini_align can't be found. But if I type it in the terminal as a normal command it can be executed. The Error happens while running the test with "sudo make test". The scrappie basecaller works fine so far, but when the katuali pipeline starts the pomoxis part it just hits this error every time.

Thank you for any advice!

make test problem with python

Hi there!

I recently installed katulai and tried running 'make test'. After 'make test', I got this error:

(omit info of installation of many packages...)
Using /share/apps/miniconda3/lib/python3.8/site-packages/wcwidth-0.2.5-py3.8.egg
Searching for smmap==3.0.5
Best match: smmap 3.0.5
Processing smmap-3.0.5-py3.8.egg
smmap 3.0.5 is already the active version in easy-install.pth

Using /share/apps/miniconda3/lib/python3.8/site-packages/smmap-3.0.5-py3.8.egg
Finished processing dependencies for katuali==0.3.2
mkdir -p test/MinIonRun1
. ./venv/bin/activate && cd test/MinIonRun1 && katuali_datafile test/reads.tgz | xargs -I {} tar -xf {}
/bin/sh: /share/apps/katuali/venv/bin/katuali_datafile: /share/apps/katuali/venv/bin/python: bad interpreter: No such file or directory
mkdir -p test/MinIonRun2
. ./venv/bin/activate && cd test/MinIonRun2 && katuali_datafile test/reads.tgz | xargs -I {} tar -xf {}
/bin/sh: /share/apps/katuali/venv/bin/katuali_datafile: /share/apps/katuali/venv/bin/python: bad interpreter: No such file or directory
mkdir -p test/GridIonRun1
. ./venv/bin/activate && cd test/GridIonRun1 && katuali_datafile test/reads.tgz | xargs -I {} tar -xf {}
/bin/sh: /share/apps/katuali/venv/bin/katuali_datafile: /share/apps/katuali/venv/bin/python: bad interpreter: No such file or directory
mkdir -p test/GridIonRun2
. ./venv/bin/activate && cd test/GridIonRun2 && katuali_datafile test/reads.tgz | xargs -I {} tar -xf {}
/bin/sh: /share/apps/katuali/venv/bin/katuali_datafile: /share/apps/katuali/venv/bin/python: bad interpreter: No such file or directory
. ./venv/bin/activate && cd test && katuali -s Snakefile --configfile config.yaml --printshellcmds -j 48 --config THREADS_PER_JOB=48 MinIonRun1/guppy/basecalls.fastq.gz
/share/apps/katuali/venv/bin/katuali: /share/apps/katuali/venv/bin/katuali_datafile: /share/apps/katuali/venv/bin/python: bad interpreter: No such file or directory
/share/apps/katuali/venv/bin/katuali: /share/apps/katuali/venv/bin/katuali_datafile: /share/apps/katuali/venv/bin/python: bad interpreter: No such file or directory
/share/apps/katuali/venv/bin/katuali: /share/apps/katuali/venv/bin/katuali_datafile: /share/apps/katuali/venv/bin/python: bad interpreter: No such file or directory

/share/apps/katuali/venv/bin/katuali: /share/apps/katuali/venv/bin/process_katuali_config: /share/apps/katuali/venv/bin/python: bad interpreter: No such file or directory
Running snakemake -s Snakefile --configfile configs/katuali_2021-03-15_15:33:05+00:00.yaml --restart-times 0 --keep-going --printshellcmds -j 48 MinIonRun1/guppy/basecalls.fastq.gz
Config saved to configs/katuali_2021-03-15_15:33:05+00:00.yaml
bash: /share/apps/katuali/venv/bin/snakemake: /share/apps/katuali/venv/bin/python: bad interpreter: No such file or directory
make: *** [test_basecall] Error 126

It seems that python causes the problem. But I have my python in /share/apps/miniconda3/bin/python, and the version is 3.8.5. I tried to copy the executable to /share/apps/katuali/venv/bin/, as suggested in other posts, but it got me an error: "cp: not writing through dangling symlink ‘/share/apps/katuali/venv/bin/python’ "

Could you please tell me how I can copy the python executable to the right folder, if this is the cause of the problem? If not, could you please help me solve this problem?

Thanks a lot for your time!

Issue installing katuali

Hello,
I've been trying to install katuali on my university server (running debian) and when I do "make install" I get the following error :

OSError: Command /data/home/fsassolas/katuali/venv/bin/python3 - setuptools pkg_resources pip wheel failed with error code 1
make: *** [Makefile:17: venv/bin/activate] Error 1

How may I solve the issue ?

make test problem

Hey!

I've installed the Katuali pipeline and installed the requiered packages as well, mostly with conda. I needed to install pomoxis into a virtual env. becouse the server has python 3.7 and 3.6 was requierd. When i do the make test following problem comes up:

Using /home/hflatau/katuali/venv/lib/python3.6/site-packages Finished processing dependencies for katuali==0.3.2

15x Wildcard constraints in inputs are ignored.
Building DAG of jobs...
MissingInputException in line 30 of /home/hflatau/katuali/katuali/data/basecalling.snake:
Missing input files for rule basecall_guppy:
/home/hflatau/git/pomoxis/venv/bin/activate
Makefile:56: recipe for target 'test_basecall' failed
make: *** [test_basecall] Error 1

Pomoxis is in the /home/hflatau directory, but it had no venv folder, /bin/activate were just in the /home/hflatau/pomoxis. I figuerd that creating the venv folder and putting the bin folder into it would help resolve the problem but it did not.

In the config.yaml file I changed the directory for pomoxis from line 119 IN_POMOXIS: "~/git/pomoxis/venv/bin/activate" to IN_POMOXIS: "~/hflatau/git/pomoxis/venv/bin/activate". It did not resolve the problem as well.

Thanks for taking your time :)

Problem with installing Katuali

Hi,
I tried to install Katuali program in ubuntu (https://releases.ubuntu.com/16.04/?_ga=2.229415391.231427745.1601872308-250255130.1599689398). After running the following commands, I still get the following Error 1 message when I try to “make install”. Please see the commands lines I have used bellow. Is this because I am using a different version of python? It would be great if someone can help.

Sudo apt install git
git clone https://github.com/nanoporetech/katuali.git
cd katuali
sudo apt install python3-venv python3-pip
python3 -m pip install –user virtualenv
make install
The message at the end
Makefile:21: recipe for target ‘install’ failed
Make: *** [install] Error 1

mini_assemble (racon) returning empty polished sequence

I'm writing a custom pipeline that closely resembles all_fast_assm_polish. I have a proper assembly using mini_assemble for de novo assembly from my guppy generated basecalls. However, when I try to polish the assembly, it returns an empty polish sequence. Code below:

# -m 2 for simplicity as anything greater than 2 produces similar errors
mini_assemble -i guppy_basecalls.fastq -r consensus_draft.fasta -o racon -t 8 -p assm

Gives me the following error:

Copying FASTX input to workspace: test/guppy/test.fastq > racon/assm.fa.gz
Skipped adapter trimming.
Skipped pre-assembly correction.
Using supplied reference to perform reference-guided consensus.
Warning: Parts of this reference might not be polished by racon and such parts will be present in the final consensus unmodified.
Running racon read shuffle 1...
Running round 1 consensus...
[M::mm_idx_gen::0.003*5.27] collected minimizers
[M::mm_idx_gen::0.004*5.48] sorted minimizers
[M::main::0.004*5.46] loaded/built the index for 1 target sequence(s)
[M::mm_mapopt_update::0.005*5.01] mid_occ = 7
[M::mm_idx_stat] kmer size: 15; skip: 10; is_hpc: 0; #seq: 1
[M::mm_idx_stat::0.005*4.73] distinct minimizers: 11031 (99.27% are singletons); average occurrences: 1.010; average spacing: 5.365
[M::worker_pipeline::0.038*4.41] mapped 162 sequences
[M::main] Version: 2.17-r941
[M::main] CMD: minimap2 -K 500M -t 8 /home/robertlink/Non-thesis-work/HIV_nanopore_pipeline/test/guppy/assembly/test.fa assm.fa.gz
[M::main] Real time: 0.039 sec; CPU: 0.170 sec; Peak RSS: 0.010 GB
[racon::Polisher::initialize] loaded target sequences 0.000369 s
[racon::Polisher::initialize] loaded sequences 0.016006 s
[racon::Polisher::initialize] loaded overlaps 0.000370 s
[racon::Polisher::initialize] aligning overlaps [====================] 0.181798 s
[racon::Polisher::initialize] transformed data into windows 0.000289 s
Running round 2 consensus...
[M::mm_idx_gen::0.000*5.18] collected minimizers
[M::mm_idx_gen::0.001*4.71] sorted minimizers
[M::main::0.001*4.66] loaded/built the index for 0 target sequence(s)
[M::mm_mapopt_update::0.001*4.52] mid_occ = 1
[M::mm_idx_stat] kmer size: 15; skip: 10; is_hpc: 0; #seq: 0
[M::mm_idx_stat::0.001*4.35] distinct minimizers: 0 (-nan% are singletons); average occurrences: -nan; average spacing: -nan
[M::worker_pipeline::0.021*2.88] mapped 162 sequences
[M::main] Version: 2.17-r941
[M::main] CMD: minimap2 -K 500M -t 8 racon_1_1.fa.gz assm.fa.gz
[M::main] Real time: 0.021 sec; CPU: 0.060 sec; Peak RSS: 0.007 GB
[racon::Polisher::initialize] error: empty target sequences set!
Waiting for cleanup.
Final assembly written to racon/assm_final.fa. Have a nice day.

Even when I use -m 1 it still returns a blank output file. Is there something that I'm misunderstanding? What are your thoughts on the manner?

what is porpita?

How to install porpita in rule porpita？I did not find relevant information。

venv/usr/local/bin instead of venv/bin ?

I'm trying to install using the instructions here, and I see an error when activate is called:

git clone [etc]
cd katuali
make install
test -d venv || virtualenv venv --prompt '(katuali) ' --python=python3 . ./venv/bin/activate && pip install pip --upgrade /bin/sh: ./venv/bin/activate: No such file or directory make: *** [venv/bin/activate] Error 1

ls -lha venv/bin
ls: venv/bin: No such file or directory
ls -lha venv/usr/local/bin
activate activate.fish activate.xsh easy_install easy_install3 pip-3.7 python python3.7 wheel-3.7 activate.csh activate.ps1 activate_this.py easy_install-3.7 pip pip3 python3 wheel wheel3

python --version
Python 2.7.16
python3 --version
Python 3.7.3
pip --version
pip 20.0.2 from /Library/Python/2.7/site-packages/pip (python 2.7)
pip3 --version
pip 19.0.3 from /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/site-packages/pip (python 3.7)

I'm on macOS:
sw_vers
ProductName: Mac OS X
ProductVersion: 10.15.2
BuildVersion: 19C57

I'd like to use this software, so any help debugging would be appreciated. I've tried a couple of the solutions for similar issues but they haven't worked.

Problems settting up Katuali

HI,
I have installed katuali following the instructions given in Docs. However, when I try to run the test data I encounter the following error:

Dynamic output is deprecated in favor of checkpoints (see docs). It will be removed in Snakemake 6.0.
Dynamic output is deprecated in favor of checkpoints (see docs). It will be removed in Snakemake 6.0.
Dynamic output is deprecated in favor of checkpoints (see docs). It will be removed in Snakemake 6.0.
Building DAG of jobs...
MissingInputException in line 154 of /home/duncan/katuali/venv/lib/python3.6/site-packages/katuali-0.2.0-py3.6.egg/katuali/data/Snakefile:
Missing input files for rule basecall_guppy:
/home/duncan/home/duncan/pomoxis/venv/bin/activate
home/duncan/katuali/test/MinIonRun1/reads

when I try to follow up on the error and check the snakefile, I cant see any problem in the script on line 154

input:
guppy = ancient(GUPPY_EXEC),
fast5 = ancient("{runid}/reads"),
venv = ancient(IN_POMOXIS),

Please find attached my config file and hope you can assist me and help me identify where I am getting it wrong with the setup of Katuali.

I am running Ubuntu 18.04. on an HP core i7 laptop, 16GB ram
I have all dependecies installed for Katuali and downloaded guppy from the ONT website.

Make test issue

Hi,

I am a newby and I was trying to install katuali following the instruction. I think I got it right until the setup of the environment. I was then running the make test but I get lots of "warning" and "error" messages.
The problem might be a dependencies installation issue? I have tried to install what I need however, for some tools the installation did not work: for example, pomoxis installation did not work with bioconda nor pip.
Any help will be appreciated.
Thanks

(nano) (katuali) linnaeus@Carls-Mac-Pro katuali % make test
. ./venv/bin/activate && python setup.py develop
running develop
running egg_info
writing katuali.egg-info/PKG-INFO
writing dependency_links to katuali.egg-info/dependency_links.txt
writing entry points to katuali.egg-info/entry_points.txt
writing requirements to katuali.egg-info/requires.txt
writing top-level names to katuali.egg-info/top_level.txt
reading manifest file 'katuali.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
no previously-included directories found matching 'build'
no previously-included directories found matching 'docs'
writing manifest file 'katuali.egg-info/SOURCES.txt'
running build_ext
Creating /Users/linnaeus/katuali/venv/lib/python3.6/site-packages/katuali.egg-link (link to .)
katuali 0.3.2 is already the active version in easy-install.pth
Installing check_files_exist script to /Users/linnaeus/katuali/venv/bin
Installing katuali_config script to /Users/linnaeus/katuali/venv/bin
Installing katuali_datafile script to /Users/linnaeus/katuali/venv/bin
Installing pick_gpu script to /Users/linnaeus/katuali/venv/bin
Installing process_katuali_config script to /Users/linnaeus/katuali/venv/bin
Installing katuali script to /Users/linnaeus/katuali/venv/bin

Installed /Users/linnaeus/katuali
Processing dependencies for katuali==0.3.2
Searching for pysam==0.15.2
Reading https://pypi.org/simple/pysam/
Downloading https://files.pythonhosted.org/packages/15/f6/ce0611aaa1865a616f7dc164fbf046eaf38f2b17c6d404403c56250beb93/pysam-0.15.2.tar.gz#sha256=d049efd91ed5b1af515aa30280bc9cb46a92ddd15d546c9b21ee68a6ed4055d9
Best match: pysam 0.15.2
Processing pysam-0.15.2.tar.gz
Writing /var/folders/vq/mc2zrs352hl8pykqzn0nt25r0000gn/T/easy_install-oi3q60rr/pysam-0.15.2/setup.cfg
Running pysam-0.15.2/setup.py -q bdist_egg --dist-dir /var/folders/vq/mc2zrs352hl8pykqzn0nt25r0000gn/T/easy_install-oi3q60rr/pysam-0.15.2/egg-dist-tmp-a4yfet1v
# pysam: no cython available - using pre-compiled C
# pysam: htslib mode is shared
# pysam: HTSLIB_CONFIGURE_OPTIONS=None
checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables... 
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking for ranlib... ranlib
checking for grep that handles long lines and -e... /usr/bin/grep
checking for C compiler warning flags... -Wall
checking for special C compiler options needed for large files... no
checking for _FILE_OFFSET_BITS value needed for large files... no
checking for _LARGEFILE_SOURCE value needed for large files... no
checking shared library type for unknown-Darwin... Darwin dylib
checking how to run the C preprocessor... gcc -E
checking for egrep... /usr/bin/grep -E
checking for ANSI C header files... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking for stdlib.h... (cached) yes
checking for unistd.h... (cached) yes
checking for sys/param.h... yes
checking for getpagesize... yes
checking for working mmap... yes
checking for gmtime_r... yes
checking for fsync... yes
checking for drand48... yes
checking whether fdatasync is declared... no
checking for library containing log... none required
checking for zlib.h... yes
checking for inflate in -lz... yes
checking for library containing recv... none required
checking for bzlib.h... yes
checking for BZ2_bzBuffToBuffCompress in -lbz2... yes
checking for lzma.h... no
checking for lzma_easy_buffer_encode in -llzma... yes
checking for libdeflate.h... no
checking for libdeflate_deflate_compress in -ldeflate... no
checking for curl_easy_pause in -lcurl... yes
checking for CCHmac... yes
checking whether PTHREAD_MUTEX_RECURSIVE is declared... yes
configure: creating ./config.status
config.status: creating config.mk
config.status: creating htslib.pc.tmp
config.status: creating config.h
# pysam: htslib configure options: --enable-libcurl
make[1]: ./version.sh: Command not found
make[1]: ./version.sh: Command not found
# pysam: htslib_config LDFLAGS=
# pysam: htslib_config LIBHTS_OBJS=kfunc.o knetfile.o kstring.o bcf_sr_sort.o bgzf.o errmod.o faidx.o hfile.o hfile_net.o hts.o hts_os.o md5.o multipart.o probaln.o realn.o regidx.o sam.o synced_bcf_reader.o vcf_sweep.o tbx.o textutils.o thread_pool.o vcf.o vcfutils.o cram/cram_codecs.o cram/cram_decode.o cram/cram_encode.o cram/cram_external.o cram/cram_index.o cram/cram_io.o cram/cram_samtools.o cram/cram_stats.o cram/files.o cram/mFILE.o cram/open_trace_file.o cram/pooled_alloc.o cram/rANS_static.o cram/sam_header.o cram/string_alloc.o hfile_libcurl.o hfile_gcs.o hfile_s3.o
# pysam: htslib_config LIBS=-llzma -lbz2 -lz -lcurl
# pysam: htslib_config PLATFORM=Darwin
# pysam: config_option: ENABLE_PLUGINS=0
# pysam: config_option: HAVE_COMMONCRYPTO=1
# pysam: config_option: HAVE_GMTIME_R=1
# pysam: config_option: HAVE_HMAC=0
# pysam: config_option: HAVE_IRODS=0
# pysam: config_option: HAVE_LIBCURL=1
# pysam: config_option: HAVE_MMAP=1
package init file 'samtools/__init__.py' not found (or not a regular file)
package init file 'bcftools/__init__.py' not found (or not a regular file)
package init file 'samtools/win32/__init__.py' not found (or not a regular file)
package init file 'htslib/__init__.py' not found (or not a regular file)
package init file 'htslib/htslib/__init__.py' not found (or not a regular file)
warning: no files found matching 'KNOWN_BUGS'
warning: no files found matching 'THANKS'
no previously-included directories found matching 'tests/'
warning: no files found matching 'samtools/configure'
warning: no files found matching 'samtools/config.mk.in'
warning: no files found matching 'samtools/config.h.in'
/Users/linnaeus/katuali/venv/lib/python3.6/site-packages/setuptools/lib2to3_ex.py:41: SetuptoolsDeprecationWarning: 2to3 support is deprecated. If the project still requires Python 2 support, please migrate to a single-codebase solution or employ an independent conversion process.
  SetuptoolsDeprecationWarning)
htslib/thread_pool.c:642:9: warning: code will never be executed [-Wunreachable-code]
        printf("%d waiting, %d running, %d output, %d, arun %d => %d\t", p->njobs,
        ^~~~~~
htslib/thread_pool.c:641:9: note: silence by adding parentheses to mark code as explicitly dead
    if (0) {
        ^
        /* DISABLES CODE */ ( )
1 warning generated.
htslib/cram/cram_encode.c:1605:13: warning: code will never be executed [-Wunreachable-code]
            cram_byte_array_len_encoder e;
            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
htslib/cram/cram_encode.c:1595:13: note: silence by adding parentheses to mark code as explicitly dead
        if (1) {
            ^
            /* DISABLES CODE */ ( )
htslib/cram/cram_encode.c:2684:40: warning: code will never be executed [-Wunreachable-code]
                                int ol=l;
                                       ^
htslib/cram/cram_encode.c:2681:33: note: silence by adding parentheses to mark code as explicitly dead
                            if (0 && CRAM_MAJOR_VERS(fd->version) >= 3) {
                                ^
                                /* DISABLES CODE */ ( )
htslib/cram/cram_encode.c:2656:17: warning: code will never be executed [-Wunreachable-code]
                int l;
                ^~~~~~
3 warnings generated.
bcftools/convert.c.pysam.c:446:66: warning: code will never be executed [-Wunreachable-code]
        default: error("Unexpected type: %d\n", fmt->fmt->type); exit(1); break;
                                                                 ^~~~
1 warning generated.
bcftools/tabix.c.pysam.c:57:22: warning: code will never be executed [-Wunreachable-code]
            detect = 0;
                     ^
1 warning generated.
bcftools/vcfroh.c.pysam.c:764:32: warning: result of comparison of constant 256 with expression of type 'int8_t'
      (aka 'signed char') is always true [-Wtautological-constant-out-of-range-compare]
            case BCF_BT_INT8:  BRANCH(int8_t); break;
                               ^~~~~~~~~~~~~~
bcftools/vcfroh.c.pysam.c:754:34: note: expanded from macro 'BRANCH'
                prob[0] = p[irr] < 256 ? args->pl2p[ p[irr] ] : args->pl2p[255]; \
                          ~~~~~~ ^ ~~~
bcftools/vcfroh.c.pysam.c:764:32: warning: result of comparison of constant 256 with expression of type 'int8_t'
      (aka 'signed char') is always true [-Wtautological-constant-out-of-range-compare]
            case BCF_BT_INT8:  BRANCH(int8_t); break;
                               ^~~~~~~~~~~~~~
bcftools/vcfroh.c.pysam.c:755:34: note: expanded from macro 'BRANCH'
                prob[1] = p[ira] < 256 ? args->pl2p[ p[ira] ] : args->pl2p[255]; \
                          ~~~~~~ ^ ~~~
bcftools/vcfroh.c.pysam.c:764:32: warning: result of comparison of constant 256 with expression of type 'int8_t'
      (aka 'signed char') is always true [-Wtautological-constant-out-of-range-compare]
            case BCF_BT_INT8:  BRANCH(int8_t); break;
                               ^~~~~~~~~~~~~~
bcftools/vcfroh.c.pysam.c:756:34: note: expanded from macro 'BRANCH'
                prob[2] = p[iaa] < 256 ? args->pl2p[ p[iaa] ] : args->pl2p[255]; \
                          ~~~~~~ ^ ~~~
bcftools/vcfroh.c.pysam.c:794:32: warning: result of comparison of constant 256 with expression of type 'int8_t'
      (aka 'signed char') is always true [-Wtautological-constant-out-of-range-compare]
            case BCF_BT_INT8:  BRANCH(int8_t); break;
                               ^~~~~~~~~~~~~~
bcftools/vcfroh.c.pysam.c:784:34: note: expanded from macro 'BRANCH'
                prob[0] = p[irr] < 256 ? args->pl2p[ p[irr] ] : args->pl2p[255]; \
                          ~~~~~~ ^ ~~~
bcftools/vcfroh.c.pysam.c:794:32: warning: result of comparison of constant 256 with expression of type 'int8_t'
      (aka 'signed char') is always true [-Wtautological-constant-out-of-range-compare]
            case BCF_BT_INT8:  BRANCH(int8_t); break;
                               ^~~~~~~~~~~~~~
bcftools/vcfroh.c.pysam.c:785:34: note: expanded from macro 'BRANCH'
                prob[1] = p[ira] < 256 ? args->pl2p[ p[ira] ] : args->pl2p[255]; \
                          ~~~~~~ ^ ~~~
bcftools/vcfroh.c.pysam.c:794:32: warning: result of comparison of constant 256 with expression of type 'int8_t'
      (aka 'signed char') is always true [-Wtautological-constant-out-of-range-compare]
            case BCF_BT_INT8:  BRANCH(int8_t); break;
                               ^~~~~~~~~~~~~~
bcftools/vcfroh.c.pysam.c:786:34: note: expanded from macro 'BRANCH'
                prob[2] = p[iaa] < 256 ? args->pl2p[ p[iaa] ] : args->pl2p[255]; \
                          ~~~~~~ ^ ~~~
bcftools/vcfroh.c.pysam.c:936:36: warning: result of comparison of constant 256 with expression of type 'int8_t'
      (aka 'signed char') is always true [-Wtautological-constant-out-of-range-compare]
                case BCF_BT_INT8:  BRANCH(int8_t); break;
                                   ^~~~~~~~~~~~~~
bcftools/vcfroh.c.pysam.c:931:33: note: expanded from macro 'BRANCH'
                pdg[0] = p[irr] < 256 ? args->pl2p[ p[irr] ] : args->pl2p[255]; \
                         ~~~~~~ ^ ~~~
bcftools/vcfroh.c.pysam.c:936:36: warning: result of comparison of constant 256 with expression of type 'int8_t'
      (aka 'signed char') is always true [-Wtautological-constant-out-of-range-compare]
                case BCF_BT_INT8:  BRANCH(int8_t); break;
                                   ^~~~~~~~~~~~~~
bcftools/vcfroh.c.pysam.c:932:33: note: expanded from macro 'BRANCH'
                pdg[1] = p[ira] < 256 ? args->pl2p[ p[ira] ] : args->pl2p[255]; \
                         ~~~~~~ ^ ~~~
bcftools/vcfroh.c.pysam.c:936:36: warning: result of comparison of constant 256 with expression of type 'int8_t'
      (aka 'signed char') is always true [-Wtautological-constant-out-of-range-compare]
                case BCF_BT_INT8:  BRANCH(int8_t); break;
                                   ^~~~~~~~~~~~~~
bcftools/vcfroh.c.pysam.c:933:33: note: expanded from macro 'BRANCH'
                pdg[2] = p[iaa] < 256 ? args->pl2p[ p[iaa] ] : args->pl2p[255]; \
                         ~~~~~~ ^ ~~~
9 warnings generated.
bcftools/filter.c.pysam.c:2755:12: warning: code will never be executed [-Wunreachable-code]
    if (0) filter_debug_print(out, NULL, nout);
           ^~~~~~~~~~~~~~~~~~
bcftools/filter.c.pysam.c:2755:9: note: silence by adding parentheses to mark code as explicitly dead
    if (0) filter_debug_print(out, NULL, nout);
        ^
        /* DISABLES CODE */ ( )
1 warning generated.
pysam/libcutils.c:6272:3: error: implicit declaration of function 'bcftools_set_stderr' is invalid in C99
      [-Werror,-Wimplicit-function-declaration]
  bcftools_set_stderr(__pyx_t_10);
  ^
pysam/libcutils.c:6272:3: note: did you mean 'samtools_set_stderr'?
/private/var/folders/vq/mc2zrs352hl8pykqzn0nt25r0000gn/T/easy_install-oi3q60rr/pysam-0.15.2/samtools/samtools.pysam.h:16:8: note: 
      'samtools_set_stderr' declared here
FILE * samtools_set_stderr(int fd);
       ^
pysam/libcutils.c:6444:5: error: implicit declaration of function 'bcftools_set_stdout_fn' is invalid in C99
      [-Werror,-Wimplicit-function-declaration]
    bcftools_set_stdout_fn(__pyx_t_13);
    ^
pysam/libcutils.c:6444:5: note: did you mean 'samtools_set_stdout_fn'?
/private/var/folders/vq/mc2zrs352hl8pykqzn0nt25r0000gn/T/easy_install-oi3q60rr/pysam-0.15.2/samtools/samtools.pysam.h:27:6: note: 
      'samtools_set_stdout_fn' declared here
void samtools_set_stdout_fn(const char * fn);
     ^
pysam/libcutils.c:6455:5: error: implicit declaration of function 'bcftools_set_stdout' is invalid in C99
      [-Werror,-Wimplicit-function-declaration]
    bcftools_set_stdout(__pyx_t_10);
    ^
pysam/libcutils.c:6851:7: error: implicit declaration of function 'bcftools_set_stdout_fn' is invalid in C99
      [-Werror,-Wimplicit-function-declaration]
      bcftools_set_stdout_fn(__pyx_t_13);
      ^
pysam/libcutils.c:6991:7: error: implicit declaration of function 'bcftools_set_stdout' is invalid in C99
      [-Werror,-Wimplicit-function-declaration]
      bcftools_set_stdout(__pyx_t_10);
      ^
pysam/libcutils.c:7022:5: error: implicit declaration of function 'bcftools_set_stdout_fn' is invalid in C99
      [-Werror,-Wimplicit-function-declaration]
    bcftools_set_stdout_fn(((char const *)"-"));
    ^
pysam/libcutils.c:7287:5: error: implicit declaration of function 'bcftools_set_optind' is invalid in C99
      [-Werror,-Wimplicit-function-declaration]
    bcftools_set_optind(1);
    ^
pysam/libcutils.c:7287:5: note: did you mean 'samtools_set_optind'?
/private/var/folders/vq/mc2zrs352hl8pykqzn0nt25r0000gn/T/easy_install-oi3q60rr/pysam-0.15.2/samtools/samtools.pysam.h:45:6: note: 
      'samtools_set_optind' declared here
void samtools_set_optind(int);
     ^
pysam/libcutils.c:7316:5: error: implicit declaration of function 'bcftools_set_optind' is invalid in C99
      [-Werror,-Wimplicit-function-declaration]
    bcftools_set_optind(0);
    ^
pysam/libcutils.c:7366:22: error: implicit declaration of function 'bcftools_main' is invalid in C99
      [-Werror,-Wimplicit-function-declaration]
    __pyx_v_retval = bcftools_main((__pyx_v_n + 2), __pyx_v_cargs);
                     ^
pysam/libcutils.c:7366:22: note: did you mean 'samtools_main'?
/private/var/folders/vq/mc2zrs352hl8pykqzn0nt25r0000gn/T/easy_install-oi3q60rr/pysam-0.15.2/samtools/samtools.pysam.h:47:12: note: 
      'samtools_main' declared here
extern int samtools_main(int argc, char *argv[]);
           ^
pysam/libcutils.c:7435:3: error: implicit declaration of function 'bcftools_unset_stderr' is invalid in C99
      [-Werror,-Wimplicit-function-declaration]
  bcftools_unset_stderr();
  ^
pysam/libcutils.c:7435:3: note: did you mean 'bcftools_set_stderr'?
pysam/libcutils.c:6272:3: note: 'bcftools_set_stderr' declared here
  bcftools_set_stderr(__pyx_t_10);
  ^
pysam/libcutils.c:7471:5: error: implicit declaration of function 'bcftools_unset_stdout' is invalid in C99
      [-Werror,-Wimplicit-function-declaration]
    bcftools_unset_stdout();
    ^
11 errors generated.
error: Setup script exited with error: command 'gcc' failed with exit status 1
make: *** [install] Error 1

error at test after installation of required dependencies

I have installed katuali as per instructions and I now trying to get the pipeline running.

I have installed the required dependencies (except for guppy, which was installed via the .deb package) as follows:

created a conda environment
installed the dependencies pomoxis, canu, flye and medaka with conda
copied the content from the conda installation to the directories suggested by katuali (see below)

(katuali) bhinckel@Brightcore-testsrv:~/downloads/katuali$ ll ~/git/pomoxis/venv/bin/activate
total 12
drwxr-xr-x 3 bhinckel 17930 4096 Jul 16 15:09 ../
drwxr-xr-x 5 bhinckel 17930 4096 Jul 16 15:15 pomoxis-0.3.4-py_0/
drwxr-xr-x 3 bhinckel 17930 4096 Jul 16 15:15 ./
(katuali) bhinckel@Brightcore-testsrv:~/downloads/katuali$ ll /usr/bin/guppy_basecaller
lrwxrwxrwx 1 root root 35 Jul 16 13:49 /usr/bin/guppy_basecaller -> /opt/ont/guppy/bin/guppy_basecaller*
(katuali) bhinckel@Brightcore-testsrv:~/downloads/katuali$ ll ~/git/canu-1.8/Linux-amd64/bin/canu
lrwxrwxrwx 1 bhinckel 17930 47 Jul 16 17:17 /home/bhinckel/git/canu-1.8/Linux-amd64/bin/canu -> /home/bhinckel/miniconda3/envs/pomoxis/bin/canu*
(katuali) bhinckel@Brightcore-testsrv:~/downloads/katuali$ ll ~/git/Flye/bin/flye
lrwxrwxrwx 1 bhinckel 17930 47 Jul 16 17:21 /home/bhinckel/git/Flye/bin/flye -> /home/bhinckel/miniconda3/envs/pomoxis/bin/flye*
(katuali) bhinckel@Brightcore-testsrv:~/downloads/katuali$ ll ~/git/medaka/venv/bin/activate
total 12
drwxr-xr-x 3 bhinckel 17930 4096 Jul 16 17:32 ../
drwxr-xr-x 3 bhinckel 17930 4096 Jul 16 17:33 ./
drwxr-xr-x 5 bhinckel 17930 4096 Jul 16 17:33 medaka-0.11.5-py36h148d290_0/
(katuali) bhinckel@Brightcore-testsrv:~/downloads/katuali$

When I run (katuali) bhinckel@Brightcore-testsrv:~/downloads/katuali$ make test, the snakemake is launched, and I see that guppy was running, though I get an error (printed to the screen) after some time:

/bin/bash: line 24: source: /home/bhinckel/git/pomoxis/venv/bin/activate: is a directory
[Fri Jul 17 11:26:19 2020]
Error in rule basecall_guppy:
    jobid: 0
    output: MinIonRun1/guppy/basecalls.fasta, MinIonRun1/guppy/sequencing_summary.txt
    log: MinIonRun1/guppy.log (check log file(s) for error message)
    shell:
        
        check_files_exist --quiet /usr/bin/guppy_basecaller MinIonRun1/reads /home/bhinckel/git/pomoxis/venv/bin/activate &> MinIonRun1/guppy.log
        # snakemake will create the output dir, guppy will fail if it exists..
        rm -r MinIonRun1/guppy

        echo "GPU status before" >> MinIonRun1/guppy.log
        gpustat >> MinIonRun1/guppy.log

        sleep $(((RANDOM % 30)  + 1 ))

        GPU=$(pick_gpu 2>> MinIonRun1/guppy.log)

        echo "Runnning on host $HOSTNAME GPU $GPU" >> MinIonRun1/guppy.log

        /usr/bin/guppy_basecaller -s MinIonRun1/guppy -r -i MinIonRun1/reads -c dna_r9.4.1_450bps_hac.cfg &>> MinIonRun1/guppy.log

        echo "gpustat after" >> MinIonRun1/guppy.log
        gpustat >> MinIonRun1/guppy.log

        # convert fastq to fasta
        sleep 5
        echo "Combining the following fastq files into MinIonRun1/guppy/basecalls.fasta" >> MinIonRun1/guppy.log
        ls MinIonRun1/guppy/*.fastq >> MinIonRun1/guppy.log
        set +u; source /home/bhinckel/git/pomoxis/venv/bin/activate; set -u;
        seqkit fq2fa MinIonRun1/guppy/*.fastq > MinIonRun1/guppy/basecalls.fasta
        rm MinIonRun1/guppy/*.fastq
        
        (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

Removing output files of failed job basecall_guppy since they might be corrupted:
MinIonRun1/guppy/sequencing_summary.txt
Job failed, going on with independent jobs.
Exiting because a job execution failed. Look above for error message
Complete log: /home/bhinckel/downloads/katuali/test/.snakemake/log/2020-07-17T111026.492659.snakemake.log
Makefile:56: recipe for target 'test_basecall' failed
make: *** [test_basecall] Error 1

What is strange is that the basecall appears to have been finished successfully (the message below is the last lines of the file /home/bhinckel/downloads/katuali/test/MinIonRun1/guppy/guppy_basecaller_log-2020-07-17_11-10-37.log

2020-07-17 11:26:12.948457 [guppy/message] Caller time: 929580 ms, Samples called: 14237088, samples/s: 15315.6
2020-07-17 11:26:12.948602 [guppy/message] Finishing up any open output files.
2020-07-17 11:26:14.467721 [guppy/message] Basecalling completed successfully.

In fact the fastq was generated (/home/bhinckel/downloads/katuali/test/MinIonRun1/guppy/fastq_runid_f1d7aa40eb01e7882b06a486c721890952f0f34a_0_0.fastq), so I guess the error occurred at the generation of the .fasta from the .fastq.

From the error message it appears that this should be accomplished by the command seqkit fq2fa MinIonRun1/guppy/*.fastq > MinIonRun1/guppy/basecalls.fasta. In the master config file (/home/bhinckel/downloads/katuali/build/lib/katuali/data/config.yaml) I do not see where seqkit should be installed, and in fact I noticed that it is not installed on my system.

Is this error indeed being caused by the absence of seqkit? if so, where should I install it? If not, what could be the cause of the problem?

Also, is there any plan to launch a conda package for katuali?

Make install failed

Hello!

I am working on Ubuntu 16.04.6 and I get an error when I run make install:

test -d venv || virtualenv venv --prompt '(katuali) ' --python=python3
/bin/sh: 1: virtualenv: not found
Makefile:17: recipe for target 'venv/bin/activate' failed
make: *** [venv/bin/activate] Error 127

Any hint what is wrong?

Thank you!
/Katerina

guppy still checking gpustat when CPU-only guppy is installed

I installed a CPU-only version of guppy, ont_guppy_cpu_5.0.7-1_xenial_amd64.deb, for the katuali pipeline. When I run katuali --configfile my_config.yaml all_fast_assm_polish, I see that rule basecall_guppy: is still resources: gpu=1. I get error Error on querying NVIDIA devices. Use --debug flag for details, at the following step: gpustat >> run1/guppy.log. I think if I set configure rule basecall_guppy to resources: gpu=0 then it would skip the gpustat step, but where do I configure this? There is no hint in the config.yaml that comes with katuali. Thank you.