iLASH - IBD Estimation Using Locality Sensitive Hashing
License: Other
My biggest project is iLASH, our open-source IBD estimation tool which scales to large biobank. It is written in C++ and uses multi-threading to speed up the process of IBD estimation. This tool has been in use by multiple labs since 2018.
Updates:
Have this issue for some ped/map files generated by PLINK.
terminate called after throwing an instance of 'DimensionException'
what(): The number of SNPs in at least one of the samples does not match the number of SNPs in the map file
Hello,
We have a multisample bcf which was generated from a heterogenous exome data pool.
To generate the PED and MAP file, we used the command -
time plink --bcf all_samples.bcf --recode tab --out outfile --double-id --allow-extra-chr
demo_idb was created using the test configuration, with the following modifications made -
map outfile.map
ped outfile.ped
But when we ran ilash demo_ibd we got the following error-
terminate called after throwing an instance of 'DimensionException'
what(): The number of SNPs in at least one of the samples does not match the number of SNPs in the map file
However, we have not messed around with the output of plink! What could the issue be?
It'd be easier to use iLash with other tooling if it could directly accept VCF input. At a glance, it looks like it might be possible to do this with only a new filereader implementation.
Running Mint 21 (Ubuntu 22.04). cmake ran fine, make gives some c errors.
user@computer:/iLASH/build$ cmake ..
-- The C compiler identification is GNU 11.3.0
-- The CXX compiler identification is GNU 11.3.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE
-- Found Boost: /usr/lib/x86_64-linux-gnu/cmake/Boost-1.74.0/BoostConfig.cmake (found version "1.74.0")
-- Configuring done
-- Generating done
-- Build files have been written to: /iLASH/build
user@computer:/iLASH/build$ make
[ 11%] Building CXX object CMakeFiles/ilash.dir/src/main.cpp.o
[ 22%] Building CXX object CMakeFiles/ilash.dir/src/context.cpp.o
[ 33%] Building CXX object CMakeFiles/ilash.dir/src/filereader.cpp.o
[ 44%] Building CXX object CMakeFiles/ilash.dir/src/minhasher.cpp.o
[ 55%] Building CXX object CMakeFiles/ilash.dir/src/experiment.cpp.o
In file included from /iLASH/src/experiment.cpp:7:
/iLASH/src/headers/lsh_slave.h:26:10: error: ‘shared_ptr’ in namespace ‘std’ does not name a template type
26 | std::shared_ptr<std::queue<std::unique_ptr< std::string>>> linesQ;
| ^~~~~~~~~~
/iLASH/src/headers/lsh_slave.h:19:1: note: ‘std::shared_ptr’ is defined in header ‘<memory>’; did you forget to ‘#include <memory>’?
18 | #include <fstream>
+++ |+#include <memory>
19 | #include <unordered_map>
/iLASH/src/headers/lsh_slave.h:27:10: error: ‘atomic’ in namespace ‘std’ does not name a template type
27 | std::atomic<bool> * runFlag; //This flag tells the program when to stop listening.
| ^~~~~~
/iLASH/src/headers/lsh_slave.h:19:1: note: ‘std::atomic’ is defined in header ‘<atomic>’; did you forget to ‘#include <atomic>’?
18 | #include <fstream>
+++ |+#include <atomic>
19 | #include <unordered_map>
/iLASH/src/headers/lsh_slave.h:29:59: error: ‘std::shared_ptr’ has not been declared
29 | LSH_Slave(Corpus *corpus, std::mutex *linesLock, std::shared_ptr<std::queue<std::unique_ptr<std::string>>>, std::atomic<bool> *);
| ^~~~~~~~~~
/iLASH/src/headers/lsh_slave.h:29:69: error: expected ‘,’ or ‘...’ before ‘<’ token
29 | Slave(Corpus *corpus, std::mutex *linesLock, std::shared_ptr<std::queue<std::unique_ptr<std::string>>>, std::atomic<bool> *);
| ^
/iLASH/src/experiment.cpp:22:57: error: ‘shared_ptr’ has not been declared
22 | void lsh_thread(Corpus *corpus, std::mutex * linesLock, shared_ptr<queue<unique_ptr<std::string>>> linesQ, atomic<bool> *runFlag){
| ^~~~~~~~~~
/iLASH/src/experiment.cpp:22:67: error: expected ‘,’ or ‘...’ before ‘<’ token
22 | sh_thread(Corpus *corpus, std::mutex * linesLock, shared_ptr<queue<unique_ptr<std::string>>> linesQ, atomic<bool> *runFlag){
| ^
/iLASH/src/experiment.cpp: In function ‘void lsh_thread(Corpus*, std::mutex*, int)’:
/iLASH/src/experiment.cpp:23:43: error: ‘linesQ’ was not declared in this scope
23 | LSH_Slave slave(corpus,linesLock,move(linesQ),runFlag);
| ^~~~~~
/iLASH/src/experiment.cpp:23:51: error: ‘runFlag’ was not declared in this scope
23 | LSH_Slave slave(corpus,linesLock,move(linesQ),runFlag);
| ^~~~~~~
/iLASH/src/experiment.cpp: In member function ‘void Experiment::read_bulk(const char*, const char*)’:
/iLASH/src/experiment.cpp:45:19: error: ‘make_shared’ was not declared in this scope
45 | auto linesQ = make_shared<queue<unique_ptr<string>>>(); //Samples will be loaded in the this queue
| ^~~~~~~~~~~
/iLASH/src/experiment.cpp:16:1: note: ‘std::make_shared’ is defined in header ‘<memory>’; did you forget to ‘#include <memory>’?
15 | #include <atomic>
+++ |+#include <memory>
16 |
/iLASH/src/experiment.cpp:45:56: error: expected primary-expression before ‘>’ token
45 | auto linesQ = make_shared<queue<unique_ptr<string>>>(); //Samples will be loaded in the this queue
| ^
/iLASH/src/experiment.cpp:45:58: error: expected primary-expression before ‘)’ token
45 | auto linesQ = make_shared<queue<unique_ptr<string>>>(); //Samples will be loaded in the this queue
| ^
make[2]: *** [CMakeFiles/ilash.dir/build.make:132: CMakeFiles/ilash.dir/src/experiment.cpp.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:83: CMakeFiles/ilash.dir/all] Error 2
make: *** [Makefile:91: all] Error 2
Instructions say cmake of >= 3.5, but there's no such thing as far as I can tell: https://cmake.org/download/. Newest stable is 3.25. Installed is 3.22:
user@computer:/iLASH/build$ cmake --version
cmake version 3.22.1
CMake suite maintained and supported by Kitware (kitware.com/cmake).
Hi ,
I was running through my data and I got the following stdout but I have got empty results
Map file address: ALL-phased_chr1_maf0nly.map
Loaded 56872 lines of SNP data.
Number of slices:1
Done with slicing.
The seed used for pseudo-random number generation is:1697441505944131589
Approximating for 0.7 for interest and 0.1 for Match
1/5: 0.447214
2/5: 0.632456
3/5: 0.774597
4/5: 0.894427
5/5: 1
Interest T:3
Match T:1
The number of cores on the host: 64
Genotype file:ALL-phased_chr1_maf0nly.ped.
Threads started
Read everything from the file.
Waiting for threads for finish their jobs
Writing
---1653034
Starting Writer Threads...
All Threads Working
Everything done in the writer part
My Config file
map ALL-phased_chr1_maf0nly.map
ped ALL-phased_chr1_maf0nly.ped
output ALL-phased_chr1.match
slice_size 30
step_size 30
perm_count 10
shingle_size 10
shingle_overlap 0
bucket_count 5
max_thread 2
match_threshold 0.10
interest_threshold 0.70
min_length 0.6
auto_slice 1
slice_length 2.9
cm_overlap 0
minhash_threshold 5
Map File : I have kept the centimorgans in plink as 0 for every position
1 1_784860 0 784860
1 1_785739 0 785739
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.