cibiv / iq-tree Goto Github PK
View Code? Open in Web Editor NEWEfficient phylogenomic software by maximum likelihood
Home Page: http://www.iqtree.org
License: GNU General Public License v2.0
Efficient phylogenomic software by maximum likelihood
Home Page: http://www.iqtree.org
License: GNU General Public License v2.0
Substitution of unacceptable characters in sequence IDs never happens if names are imported from NEXUS (as opposed to IDs in a FASTA file).
I have a bunch of complex names, something like this:
Asterionellopsis_glacialis,_Strain_CCMP134|CAMPEP_0199907168
Phatr2|1719
They have commas and pipes and whatnot. IQ-TREE itself accepts them just fine by substituting all offending chars with the underscore, but when I try to pass them through NEXUS file as cluster members for the likelihood mapping, it leaves a lengthy log like this:
Cluster 2 "diatom" lists 341 sequences:
WARNING: sequence name "Asterionellopsis_glacialis"! Will be ignored.
WARNING: sequence name ","! Will be ignored.
WARNING: sequence name "_Strain_CCMP134|CAMPEP_0199907168"! Will be ignored.
WARNING: sequence name "Phatr2|1719"! Will be ignored.
And so on for the rest of sequences. Of course, analysis either doesn't happen or proceeds with incorrect clustering. Running something like `s/[,|-]/_/g' on NEXUS file lets it proceed correctly. So the source is obvious: a NEXUS parser makes different assumptions about what a seq ID may and may not be.
Currently the IQ-TREE build system will pass --gc-sections
to the linker on macOS if the compiler is GCC. This causes build failure.
Specifically the problem in CMakeLists.txt is here:
Line 129 in 9aa8628
For Homebrew, we build IQ-TREE with openmp, so the build uses GCC, and I've worked around the build failure with the following string replacement:
if OS.mac?
inreplace "CMakeLists.txt",
"${CMAKE_EXE_LINKER_FLAGS_RELEASE} -Wl,--gc-sections",
"${CMAKE_EXE_LINKER_FLAGS_RELEASE}"
end
Allowing users to supply a constraint tree topology (-g
option) to guide tree search, such that the resulting tree obeys the constraint. A naive implementation will be released with version 1.5.0.
iqtree -s alignment -g constraint_tree ...
TODOs (as of Sept 8, 2016):
% iqtree -nt AUTO
<chooses 6 threads>
WARNING: Number of threads seems too high for short alignments. Use -nt AUTO to determine best number of threads.
Happens in 1.6 beta and 1.5
There is other software can do this (like Archaeopteryx) but hard to use. That's why I ask help here how to do these things in iqtree.
Thanks a lot.
Ping Wu
Iker Irisarri:
I was wondering whether it is possible to conduct a negative constrained tree search in IQTREE, similar to using the -g option. By negative constraint I mean the best ML tree that does not contain a given bipartition (e.g. best ML where A and B are not monophyletic). If not, do you plan to implement it in the future?
See discussion: https://groups.google.com/forum/#!topic/iqtree/8VZdfnKTwXk
Computing the ancestral sequences as well as ancestral state probabilities of all internal nodes for a given tree. Although this was long implemented in several programs, including very good Tal Pupko's FastML, they do not provide the variety of models available in IQ-TREE. A marginal state probabilities computation is straight-forward and was already implemented.
iqtree -s alignment -te user_tree -asr
TODOs (as of Sept 8, 2016):
Implementation of the (non-reversible) Lie Markov models. Right now, it is in the liemarkov branch of the git repository. The code is functioning by e.g.:
iqtree -s example.phy -m LM5.6b
iqtree -s example.phy -m TEST -mset liemarkov #model testing
DONE items:
TODOs (as of Mar 19, 2018):
Hello,
following documentation http://www.iqtree.org/doc/Compilation-Guide/#compiling-mpi-version
I issued the following
OMPI_CC=gcc OMPI_CXX=g++ cmake -DCMAKE_C_COMPILER=mpicc -DCMAKE_CXX_COMPILER=mpicxx /local/gensoft2/src/IQ-TREE/IQ-TREE-1.5.0a
generates following binary: iqtree
not iqtree-mpi as expected.
best regards
Eric
Hi Bui, congrats for the great work and the recognition by the SMBE!!!
$ iqtree -s regions.fst -st DNA -m HKY+ASC+G -nt AUTO -redo
IQ-TREE multicore version 1.6.1 for Linux 64-bit built Dec 26 2017
Developed by Bui Quang Minh, Nguyen Lam Tung, Olga Chernomor,
Heiko Schmidt, Dominik Schrempf, Michael Woodhams.
Host: Tenerife (SSE4.1, 7 GB RAM)
Command: iqtree -s regions.fst -st DNA -m HKY+ASC+G -nt AUTO -redo
Seed: 714051 (Using SPRNG - Scalable Parallel Random Number Generator)
Time: Wed Dec 27 19:21:21 2017
Kernel: SSE2 - auto-detect threads (4 CPU cores detected)
Reading alignment file regions.fst ... Fasta format detected
Alignment most likely contains DNA/RNA sequences
Alignment has 118 sequences with 1195 columns, 1097 distinct patterns
1037 parsimony-informative, 158 singleton sites, 0 constant sites
Gap/Ambiguity Composition p-value
1 1 0.00% passed 94.87%
...
NOTE: 111 is identical to 101 but kept for subsequent analysis
NOTE: 115 is identical to 113 but kept for subsequent analysis
Create initial parsimony tree by phylogenetic likelihood library (PLL)... 0.036 seconds
Ascertainment bias correction: 4 unobservable constant patterns
NOTE: 16 MB RAM (0 GB) is required!
Measuring multi-threading efficiency up to 4 CPU cores
Increase to 10 rounds for branch lengths
5 trees examined
Threads: 1 / Time: 4.043 sec / Speedup: 1.000 / Efficiency: 100% / LogL: -65164
*** Error in `iqtree': munmap_chunk(): invalid pointer: 0x00000000015c5110 ***
======= Backtrace: =========
[0x94c911]
[0x958fd2]
[0x5e0ba8]
[0x5a8d16]
[0x439f4c]
[0x445713]
[0x41a9d0]
[0x915f26]
[0x91611a]
[0x402b99]
======= Memory map: ========
00400000-00b57000 r-xp 00000000 08:07 1180657 /usr/local/biotools/bin/iqtree
00d56000-00d6e000 rw-p 00756000 08:07 1180657 /usr/local/biotools/bin/iqtree
00d6e000-00d86000 rw-p 00000000 00:00 0
01211000-02713000 rw-p 00000000 00:00 0 [heap]
7fd1b0000000-7fd1b0021000 rw-p 00000000 00:00 0
7fd1b0021000-7fd1b4000000 ---p 00000000 00:00 0
7fd1b7bff000-7fd1b7c00000 ---p 00000000 00:00 0
7fd1b7c00000-7fd1b8000000 rw-p 00000000 00:00 0
7fd1b8000000-7fd1b8021000 rw-p 00000000 00:00 0
7fd1b8021000-7fd1bc000000 ---p 00000000 00:00 0
7fd1bd2ef000-7fd1bd2f0000 rw-p 00000000 00:00 0
7fd1bd2f0000-7fd1bd2f1000 ---p 00000000 00:00 0
7fd1bd2f1000-7fd1bd6f1000 rw-p 00000000 00:00 0
7ffc2577e000-7ffc257a0000 rw-p 00000000 00:00 0 [stack]
7ffc257bf000-7ffc257c1000 r--p 00000000 00:00 0 [vvar]
7ffc257c1000-7ffc257c3000 r-xp 00000000 00:00 0 [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]
ERROR: STACK TRACE FOR DEBUGGING:
ERROR:
ERROR: *** IQ-TREE CRASHES WITH SIGNAL ABORTED
ERROR: *** For bug report please send to developers:
ERROR: *** Log file: regions.fst.log
ERROR: *** Alignment files (if possible)
Aborted (core dumped)
Host: Tenerife (SSE4.1, 7 GB RAM)
Command: iqtree -s regions.fst -st DNA -m HKY+G -nt 1 -redo
Seed: 505627 (Using SPRNG - Scalable Parallel Random Number Generator)
Time: Wed Dec 27 19:23:34 2017
Kernel: SSE2 - 1 threads (4 CPU cores detected)
HINT: Use -nt option to specify number of threads because your CPU has 4 cores!
HINT: -nt AUTO will automatically determine the best number of threads to use.
Reading alignment file regions.fst ... Fasta format detected
...
IQ-TREE multicore version 1.5.6 for Linux 64-bit built Dec 4 2017
Copyright (c) 2011-2017 by Bui Quang Minh, Nguyen Lam Tung,
Olga Chernomor, Heiko Schmidt, and Arndt von Haeseler.
Host: Tenerife (SSE4.1, 7 GB RAM)
Command: /home/vinuesa/Software_downloads/iqtree-omp-1.5.6-Linux/bin/iqtree-omp -s regions.fst -st DNA -m HKY+G -nt AUTO -redo
Seed: 977243 (Using SPRNG - Scalable Parallel Random Number Generator)
Time: Wed Dec 27 19:28:02 2017
Kernel: SSE2 - auto-detect threads (4 CPU cores detected)
Reading alignment file regions.fst ... Fasta format detected
Alignment most likely contains DNA/RNA sequences
Alignment has 118 sequences with 1195 columns and 1097 patterns (1037 informative sites, 0 constant sites)
...
END
Would be great to get your feedback on this issue!
Keep up with the great work!
Cheers,
Pablo
The files and line numbers refer to branch pomo_latest
, commit 23aa0965
Amend gitignore. Change output, max branch length.
Testing. I changed quite a bit of core code in PoMo which has to be
thoroughly tested. We will do this when we run simulations for the
application note that we are planning.
Output. The output needs to be improved.
./model/modelpomo.cpp:611: // TODO DS: Output separation (rvbl and flux).
./model/modelpomo.cpp:837: // TODO DS: Separation (rvbl and flux).
Decomposition of the rate matrix. The function performing the
eigendecomposition requests the rate matrix but also recomputes the rate
matrix. This leaves room for speed improvements.
EigenDecomposition::eigensystem_sym()
expects a matrix[][]
object with
two indices. However, it is not used, because
ModelPoMo::computeRateMatrix()
is called anyways from within
eigensystem_sym()
.
./model/modelpomo.cpp:940: // TODO DS: This leaves room for speed improvements.
ModelFinder. Implement model finder. I must admit I did not look into this
much until now.
./main/phylotesting.cpp:1554: // TODO DS: Implement model finder.
Mixture models and gamma rate heterogeneity. A mixture model is used to
handle Gamma rate heterogeneity at the moment. Hence, it is not possible to
combine mixture models and gamma rate heterogeneity at the moment, a
restriction that could be removed.
Maximum genetic distance. The distance measure of PoMo includes not only
mutations (which can be compared to substitutions) but also frequency shifts.
Hence, the branch lengths of PoMo are expected to be longer by a factor of
N*N
. This clashed with constants defined, e.g., in tools.h
./utils/tools.h:436:/* TODO DS: For PoMo, this setting does not make sense.
./utils/tools.cpp:835: // TODO DS: This seems inappropriate for PoMo. It is handled in
I had to manually recompute MAX_GENETIC_DIST
whenever it is used, because
during maximization of the likelihood, branch lengths were limited. This
should be working now, but it is important to keep in mind when changing
code in these areas. Maybe, a model dependent maximum distance may make
sense.
Do not temper with Params. The problem is the following: when running
IQ-TREE with PoMo, certain parameters need to be known already when reading
in the alignment file (counts file at the moment). This is, the virtual
population size N
which affects the data structure and the sampling method
(sampled or weighted). At the moment, those flags are stored in ModelPoMo
upon model creation and also in the Params
class. It may be advantageous
to refrain from using the Params
class to store model parameters.
./alignment/alignment.cpp:2001: // TODO DS: Do not temper with params;
Verbosity. The output with increased verbosity is too long.
./utils/eigendecomposition.cpp:255: // TODO DS: This seems a little too verbose for PoMo.
No polymorphic data. At the moment, it is not possible or at least it leads
to undefined behavior when PoMo is run on data without polymorphisms. Either,
the level of polymorphism should be fixed, or an clear error message should
be emitted.
./model/modelpomo.cpp:229: // TODO DS: Check if this works.
./model/modelpomo.cpp:230: // TODO DS: Move this where it belongs.
./model/modelpomo.cpp:234: // TODO DS: Obsolete because mutation rates are scaled
R and Phi matrices. IQ-TREE and PoMo is now aware of the symmetric mutation
rate matrix R
and the skew-symmetric mutation matrix Phi
(refer to our
most recent publication, Schrempf, Hobolth 2017). These matrices are
unnecessarily dragged along during the maximization of the likelihood but
really only need to be computed at the end when printing output.
./model/modelpomo.cpp:457: // TODO DS: R and PHI are actually not needed during the
Theta and sampling method sampled. The calculation of theta
(Watterson's
theta or level of polymorphism) is necessarily faulty, especially when N
is
low. I am not sure how to tackle this because the problem is of statistical
nature.
./model/modelpomo.cpp:791: // TODO DS: This is wrong because Watterson's estimator is
Support alignments with differences only. It may be good to support
alignments that only contain sites with differences but omit sites that are
the same in all species. I am not sure how many people will have data like that.
./model/modelfactory.cpp:215: // TODO DS: This is an important feature, because then,
Codon models have 61 states or so, thus it is quite sometimes the case that some codons are absent in the alignment, making likelihood computation unstable. There are two solutions for this:
However, we observed numerical problem with approach 1 and also a number of users reported it. We can switch to approach 2, which may avoid this problem. Here is a reply from @ziheng-yang about codon models in PAML:
this may be a problem for the F61 or similar models, with 60 frequency parameters, and less so for the F3x4 or similar models. if too many codons are missing in the F61 model, the states will not be commutative, which will cause problems. is this what you are talking about. if the chain is still commutative, perhaps there should be no problem.
my 2014 book has a section , 2.6, about the calculation of P(t), which also discusses how to deal with 0 frequencies. this is used in paml/codeml. i guess this sounds like your approach 2.
The above algorithm assumes that the frequency ;i > 0 for every i. If m states aremissing
(with ;i = 0), the Markov chain has in effect c – m states. We can rearrange the Q matrix
so that the submatrix for the existing states Q0, of size (c – m) ¡¿ (c – m), has the spectral
decomposition
but you should somehow discourage people from fitting big models to small datasets.
Thus, iqtree will switch to Approach 2 in a later 1.6.X release.
Hi,
I was running iqtree for ModelFinder when it crashed with an "out of memory" error. I found this surprising because before starting to run the models, it states
NOTE: ModelFinder requires 653 MB RAM!
Is that the base memory needed to start running MF or is it the memory it expects to use base on the input options?
This is the command I ran
iqtree-omp -st AA -s concat.archaea.209plus.2017-06-26.faa -m TESTONLY -mset LG,C60,PMSF -nt 4 -pre test_models
Below the complete MF log
NOTE: ModelFinder requires 653 MB RAM!
ModelFinder will test 24 protein models (sample size: 4200) ...
No. Model -LnL df AIC AICc BIC
1 LG 1049188.283 499 2099374.566 2099509.431 2102539.643
2 LG+I 1046542.281 500 2094084.563 2094220.005 2097255.983
3 LG+G4 1003257.220 500 2007514.441 2007649.883 2010685.861
4 LG+I+G4 1002863.271 501 2006728.542 2006864.562 2009906.304
5 LG+F 1050273.488 518 2101582.975 2101729.045 2104868.566
6 LG+F+I 1047711.775 519 2096461.551 2096608.224 2099753.484
7 LG+F+G4 1003295.430 519 2007628.860 2007775.534 2010920.794
8 LG+F+I+G4 1002926.773 520 2006893.546 2007040.826 2010191.823
Model C60 is alias for POISSON+G+FMIX{C60pi1:1:0.0169698865,C60pi2:1:0.0211683374,C60pi3:1:0.0276589079,C60pi4:1:0.0065675964,C60pi5:1:0.0141221416,C60pi6:1:0.0068774834,C60pi7:1:0.0146909701,C60pi8:1:0.0067225777,C60pi9:1:0.0018396660,C60pi10:1:0.0102547197,C60pi11:1:0.0230896163,C60pi12:1:0.0057941033,C60pi13:1:0.0125394534,C60pi14:1:0.0204526478,C60pi15:1:0.0070629602,C60pi16:1:0.0117982741,C60pi17:1:0.0068334668,C60pi18:1:0.0433775839,C60pi19:1:0.0318278731,C60pi20:1:0.0222546108,C60pi21:1:0.0102264969,C60pi22:1:0.0150545891,C60pi23:1:0.0134159878,C60pi24:1:0.0148552065,C60pi25:1:0.0239111516,C60pi26:1:0.0128776278,C60pi27:1:0.0222318842,C60pi28:1:0.0247444742,C60pi29:1:0.0214274810,C60pi30:1:0.0115001882,C60pi31:1:0.0076017389,C60pi32:1:0.0130258568,C60pi33:1:0.0093701965,C60pi34:1:0.0467194264,C60pi35:1:0.0441940314,C60pi36:1:0.0322263154,C60pi37:1:0.0402999891,C60pi38:1:0.0150234227,C60pi39:1:0.0104589903,C60pi40:1:0.0214742395,C60pi41:1:0.0154957836,C60pi42:1:0.0101789953,C60pi43:1:0.0227980379,C60pi44:1:0.0123204539,C60pi45:1:0.0066777583,C60pi46:1:0.0004150083,C60pi47:1:0.0344385130,C60pi48:1:0.0113663379,C60pi49:1:0.0127143049,C60pi50:1:0.0124323741,C60pi51:1:0.0262124415,C60pi52:1:0.0064994957,C60pi53:1:0.0103203293,C60pi54:1:0.0142463512,C60pi55:1:0.0215600067,C60pi56:1:0.0199150700,C60pi57:1:0.0038964200,C60pi58:1:0.0113448855,C60pi59:1:0.0128595846,C60pi60:1:0.0117656776}
STACK TRACE FOR DEBUGGING:
1 double* aligned_alloc<double>(unsigned long)
2 PhyloTree::initializeAllPartialLh(int&, int&, PhyloNode*, PhyloNode*)
3 PhyloTree::initializeAllPartialLh()
4 testModel(Params&, PhyloTree*, std::vector<ModelInfo, std::allocator<ModelInfo> >&, std::ostream&, ModelsBlock*, int, std::string, bool, std::string)
5 initializeParams(Params&, IQTree&, std::vector<ModelInfo, std::allocator<ModelInfo> >&, ModelsBlock*)
6 runTreeReconstruction(Params&, std::string&, IQTree&, std::vector<ModelInfo, std::allocator<ModelInfo> >&)
7 runPhyloAnalysis(Params&, Checkpoint*)
8 main()
9 __libc_start_main()
ERROR: Not enough memory, allocation of 40273689632 bytes failed (bad_alloc)
Thanks,
Xabi
Improving UFBoot implementation to e.g. address model violations.
Option -blfix is not available for the heterotachy model. This means it is not possible to calculate the likelihood of the data on a specific tree. The error message received when trying to do this is:
ERROR: Fixing branch lengths is not supported under specified site rate model
Hi,
there was bug reported for the Debian package since the test suite seems to assume when running the test script I've created from the information I obtained from the documentation.
Do you see any source for the trouble when running on a single core machine?
Kind regards, Andreas.
iqtree-omp -s input.fa -spp parts.nex -m GTR+G -bo 1 -wbt -seed $RANDOM -pre boot_1 -nt 16
builds without errors, but consistently crashes at the same stage. Prebuilt binary segfaults immediately with 'FATAL: kernel too old' on RHEL 6 - 2.6.32-504.30.3.el6.x86_64 kernel.
===> START BOOTSTRAP REPLICATE NUMBER 1
Creating bootstrap alignment (seed: 14945)...
Create initial parsimony tree by phylogenetic likelihood library (PLL)... 13.961 seconds
NOTE: 11387 MB RAM is required!
STACK TRACE FOR DEBUGGING:
1 double* aligned_alloc(unsigned long)
2 PhyloSuperTreePlen::initializeAllPartialLh()
3 runTreeReconstruction(Params&, std::__cxx11::basic_string<char, std::char_traits, std::allocator >&, IQTree&, std::vector<ModelInfo, std::allocator >&)
4 runStandardBootstrap(Params&, std::cxx11::basic_string<char, std::char_traits, std::allocator >&, Alignment, IQTree)
5 runPhyloAnalysis(Params&, Checkpoint*)
6 main()
7 __libc_start_main()
ERROR: Not enough memory, allocation of 881796096 bytes failed (bad_alloc)
Allowing users to convert character states to a new state space. For example, the existing -st NT2AA
option allows to convert codon into AA sequences. Something like -st NT2RY
to convert DNA into RY (purine pyrimidine). A more general syntax might look like:
-st ABC:0,DE:1
to convert states A,B,C into state 0 and D,E into state 1.
As an application to reduce amino-acid state space to Dayhoff group (https://doi.org/10.1093/molbev/msm144):
-st AGPST:0,C:1,FWY:2,HRK:3,MILV:4,NDEQ:5
This library allows to detect trees with exactly the same likelihood as the ML tree but with different topologies. Usefully to report multiple equally good ML trees.
@upsj, @stamatak
Code: https://github.com/upsj/terraphast-one
bioRxiv preprint: https://doi.org/10.1101/211276
Reported by Remi Denise
https://groups.google.com/forum/#!topic/iqtree/ux9bvgnNBCk
I launch IQtree 1.5.4 on the cluster of my institute and sometime it crashed with a segmentation fault during the optimizing step.
I launch with the command line :
iqtree-mpi -s FirmicutesSmall.aa.grp.aln -st AA -pre FirmicutesSmall.aa.grp -bb 1000 -wbtl -nm 6000 -m LG+R10
And I backtrack the error from the core file :
#0 0x00000000004e2d54 in errstreambuf::overflow(int) ()
#1 0x00007f8cdf8c048a in std::basic_streambuf<char, std::char_traits >::sputc (__c=10 '\n', this=) at /tmp/build/x86_64-unknown-linux-gnu/libstdc++-v3/include/streambuf:434
#2 std::ostream::put (this=this@entry=0xa44c00 std::cerr, __c=) at /tmp/build/x86_64-unknown-linux-gnu/libstdc++-v3/include/bits/ostream.tcc:163
#3 0x00007f8cdf8c06bf in std::endl<char, std::char_traits > (__os=...) at /tmp/build/x86_64-unknown-linux-gnu/libstdc++-v3/include/ostream:565
#4 0x00000000004e3771 in funcAbort ()
#5
#6 0x00000000004965b1 in IQTree::saveUFBoot(Checkpoint*) ()
#7 0x000000000049c04b in IQTree::syncCurrentTree() ()
#8 0x00000000004a0ac4 in IQTree::doTreeSearch() ()
#9 0x0000000000513448 in runTreeReconstruction(Params&, std::string&, IQTree&, std::vector<ModelInfo, std::allocator >&) ()
#10 0x000000000051dfd8 in runPhyloAnalysis(Params&, Checkpoint*) (
#11 0x00000000004f3e0d in main ()
This would be useful for some users, e.g.:
TODO: Output format?
Reported by Mark Miller from the iqtree forum
https://groups.google.com/forum/#!topic/iqtree/QhPxocbZtzM
Note that v1.5 does not suffer from this crash.
This feature was requested by Steven.
There are two types of polymorphic coding in the nexus format. (0 1) indicates the character is observed as both states 0 AND 1, while {0 1} indicates the character is observed as either state 0 OR 1. Three or more states for a single character are also possible. I have attached a screenshot of how these are entered into the software Mesquite during character scoring as well as the "export > simplified nexus" file that is output by Mesquite.
MrBayes, PAUP and TNT all accept polymorphic coded characters but during analyses treat them as ambiguous/missing. RAxML requires that polymorphic characters are re-coded as missing before loading the alignment. This recoding is no small task with morpholgical matricies that often contain several hundred or even thousands of characters. Also, then there becomes two matrices to keep track of - one with polmorphic scores and one that has been recoded.
If IQ-TREE could also accept polymorphic coding of morphological characters it would be a huge benefit! (which will add to the many other advantages IQ-TREE has over similar software) My initial thought is that when reading in a sequence file, if the character "(" is observed, the sequence would continue to be read until the character ")" is found and then "(...)" is simply replaced by a question mark. Also, perhaps a notice could be printed to the screen that says something like "polymorphic scoring of morphological data has been detected and these characters will be treated as ambiguous/missing". The same would be done with the “{“ and “}” characters.
I hope this suggestion is helpful. You have certainly already made my research more flexible and efficient… thanks so much!
(This is low priority as there is an obvious workaround.)
I downloaded an alignment and tree from TreeBase. Annoyingly, some of
the taxon names contain spaces, and use quotes around them. IQtree reads
the alignment OK, doing some auto-substitutions to deal with space and
quote characters. The problem is then when it reads the tree file, it
doesn't do the same auto-substitutions, so it can't reconcile the
alignment taxa with the tree taxa.
./iqtree -s M39346.fasta -te T100530.treefile -redo -m 012345
Allowing the possibility to have mixture models (or PMSF) as well as partitioned data analysis. One could imagine allowing the weight parameters of the mixture either being linked across partitions or unlinked).
Message from Andrew Roger:
One thing that might be really useful is for people to be able to use partition models at the same time as these PMSF (or mixture) models. For instance they may wish to have different branchlengths and alpha shape parameters for different partitions, but at the same time use the mixture models (or PMSF) with the same weights for all the partitions (or partition-specific weights maybe).
It is relevant to recent debates between Nicolas Lartillot (phylobayes CAT model) and Ken Halanych over animal phylogeny. Halanych suggests its more important to partition data than accommodate site-heterogeneity (i.e. through mixture models like CAT) for accurate phylogenetic inference. Nicolas Lartillot argues the opposite.I think Nicolas is mostly correct — that site-heterogeneity is more important usually to accommodate than partitioning…but ultimately having both partitions and the ability to have site-heterogeneity would lead to the most model ‘realism’ in my view.
Just to follow up on the rationale for a model that allows partitions AND mixture models (where the mixture models are ‘linked’ across partitions).
I think the issue of gene-specific ‘heterotachy’ — i.e. different genes having different branchlengths — can cause problems if ignored and branchlengths are linked across partitions. This is the general problem of heterotachy (see a paper we wrote on this in 2005). However, as I mentioned in my phyloseminar talk, I think the site-specific ‘constraints’ on evolution are probably and even more important issue in phylogenetics. Hence the need for the site profile mixture models and the PMSF models we’ve developed.
Ideally however, it would be nice to be able to have both at the same time. In this case I really don’t think it is necessary for the partitions to have different ‘weights’ for different mixture classes (or even different gamma distribution shape parameters) — so I think linnking the mixture models so that the weights are the same for all sites in all partitions is fine. The main rationale for partitioning, in my view, is to allow for separate branchlengths for different partitions. In my view, for concatenated protein alignments, allowing for different exchangeabilities or different mixture weigths per partition is not really important. I realize I haven’t given you a lot of literature references to back up my assertions, these are more based on my own intuition.
From discussions with @roblanf several improvements in PartitionFinder2 can help to substantially speed up:
Hi,
I think the 1.6.1 release was redone. This causes packaging to fail due to hash and/or size mismatches. If there are no downsides, in such situations, could a new release such as 1.6.1.1 be made to prevent these problems?
=> Attempting to fetch https://codeload.github.com/Cibiv/IQ-TREE/tar.gz/v1.6.1?dummy=/Cibiv-IQ-TREE-v1.6.1_GH0.tar.gz
fetch: https://codeload.github.com/Cibiv/IQ-TREE/tar.gz/v1.6.1?dummy=/Cibiv-IQ-TREE-v1.6.1_GH0.tar.gz: size unknown
fetch: https://codeload.github.com/Cibiv/IQ-TREE/tar.gz/v1.6.1?dummy=/Cibiv-IQ-TREE-v1.6.1_GH0.tar.gz: size of remote file is not known
Cibiv-IQ-TREE-v1.6.1_GH0.tar.gz 0 B 0 Bps
=> Fetched file size mismatch (expected 4481806, actual 4481835)
=> Couldn't fetch it - please try to retrieve this
=> port manually into /portdistfiles/ and try again.
*** Error code 1
Stop.
make: stopped in /usr/ports/biology/iqtree
I am trying to launch an analyses using the beta4 with the following options:
iqtree -nt 16 -s Concat-Tc0410st-rp3.phy -m LG+C20+R4+FO -bb 1000
However, the run stops with the following error message: ERROR: PoMo does not yet support free rate models (+R).
which implies that the protein model was confused for the PoMo model. Am I doing something wrong, is there a way to fix that? Thanks!
Hi,
I found an edge-case which makes IQTree crash. If a constraint tree is provided, but taxa in the tree are identical in the alignment, IQTree crashes unless you specify not to remove identical taxa. This could be caught and handled better. I assume the same would happen if you provided a user starting tree or fix topology tree.
Thanks,
Simon Harris
When optimizing a +R model, the optimization reaches a hard limit of 99 iterations without having converged. For example:
./iqtree -s humanMito.fasta -te humanMito.tree -m K3P+FQ+R -pre temp -redo -v
(For this particular data set, this occurs on nearly all DNA models.)
Here is the tail end of the output. Note how the +R parameters are slowly migrating in one direction.
I've e-mailed the sample files to Minh.
Total number of iterations: 0
CPU time used for tree search: 0.0000070000 sec (0h:0m:0s)
Wall-clock time used for tree search: 0.0000067140 sec (0h:0m:0s)
Total CPU time used: 200.5198740000 sec (0h:3m:20s)
Total wall-clock time used: 200.4912709580 sec (0h:3m:20s)
Best tree printed to temp.treefile
Analysis results written to:
IQ-TREE report: temp.iqtree
Maximum-likelihood tree: temp.treefile
Screen log file: temp.log
Date and Time: Mon Aug 21 14:44:10 2017
Hi,
Im trying to construct a partitioned phylogeny using loci locations. I input the partitions into the file part.nex and use the following command to determine the model of best fit for each
iqtree -nt AUTO -s alignment.fasta -spp part.nex -m TESTMERGE --rcluster-max 100
My dataset has 140 tax and 2530 gene Im currently using iq-tree beta 1.6.4. To my understanding once the models have been selected it should progress into tree construction. I tried re-starting the analysis but it again ends after model selection.
Hi,
I am looking to run iqtree on a cluster and would like to be able to take advantage of -nt AUTO
. However, by default, iqtree is detecting the total number of cores on the given compute node, which may or may not match the resource allocation requested. For example, if a node has 64 cores but my job only requested 32, I would like to be able to use AUTO while also specifying "no more than 32". Is this possible with the current version?
Thanks!
-Stephanie
cormorants.nex.txt
cormorants.tree.txt
I have several cases where optimization fails for a time reversible
model. In all cases, the model is 011110+FQ (plus some
rates-across-sites option.) Here is an example:
./iqtree -s cormorants.nex -te cormorants.tree -redo -pre temp -m
011110+FQ+I
[...]
(Sorry, I tried to attach the input files for the example but couldn't make it work. I've e-mailed them to Minh, or you can request them from me.)
Note that this model is like K2P, except with a different pair of rates
being allowed to differ. (K2P would be 010010.)
The problem appears to be that the variable rate parameter leaps to near
100 on the very first iteration, and then only very slowly declines on
further iterations, until iqtree reaches a hard coded maximum of 99
iterations and gives up before converging on the optimum. I suspect the
fix is to look at that very first iteration and place some limit on how
large a leap it can make. I think it starts with rate A-C: 1.000, rate
A-G:1.000 ('-v' doesn't show this) and next step has A-G: 99.9191431513
(this is the first value shown by '-v'.) If you said "from 1 to 100 in
one go is too far, I'll never change by more than a factor of 3 in a
single step" I think the problem would be solved.
Having said all this, it is a pretty rare problem. I've been doing tests
with 72 time reversible models on about 15 data sets, and I've seen this
I think just three times, all on the 011110+FQ model.
By comparison:
./iqtree -s cormorants.nex -te cormorants.tree -redo -pre temp -m
'011110{20}+FQ+I' -optfromgiven
[...]
This is also a Lie-Markov model, so the following is equivalent (but
uses a different parameterization of the model):
./iqtree -s cormorants.nex -te cormorants.tree -redo -pre temp -m MK2.2b+I
[...]
StateFreqType is an enum. There are in tools.cpp the following methods directly applying to it:
StateFreqType parseStateFreqFromPlusF(string model_name)
StateFreqType parseStateFreqDigits(string digits)
bool freqsFromParams(double *freq_vec, double *params, StateFreqType freq_type)
void paramsFromFreqs(double *params, double *freq_vec, StateFreqType freq_type)
void forceFreqsConform(double *base_freq, StateFreqType freq_type)
int nFreqParams(StateFreqType freq_type)
void setBoundsForFreqType(double *lower_bound,
double *upper_bound,
bool *bound_check,
double min_freq,
StateFreqType freq_type) {
In addition, I'm about to add some new code to correctly checkpoint some new StateFreqTypes.
I suggest that this is now too much functionality to hang off an enum, and it should be a class.
Hi,
Just wanted to compile latest IQ-TREE on NetBSD but it failed to link the iqtree binary
[...]
[100%] Linking CXX executable iqtree
CMakeFiles/iqtree.dir/tools.cpp.o: In function print_stacktrace(std::ostream&, unsigned int)': /home/njoly/temp/IQ-TREE/tools.cpp:4054: undefined reference to
backtrace'
/home/njoly/temp/IQ-TREE/tools.cpp:4063: undefined reference to `backtrace_symbols'
--- iqtree ---
*** [iqtree] Error code 1
On some systems the backtrace functions are provided by libC, but on NetBSD they are located in their own library "libexecinfo.so"
Adding a check for the "Backtrace" package in CMakeList.txt, fixes the problem.
find_package(Backtrace)
target_link_libraries(iqtree ${Backtrace_LIBRARY})
Thanks in advance.
Given an alignment and an associated PDB structure of one sequence in the alignment, compute site-specific rates and map these onto the PDB structure file by altering the occupancy and bfactor fields.
This is useful to see how protein structure and rates are correlated. This can be then displayed by a 3D structure viewer (see demo here: http://www.iqtree.org/ModelFinder/).
This feature has currently low priority but can be changed depending on user demand.
Hello,
compiling IQ-TREE-1.5.0a with -DIQTREE_FLAGS="omp" leads to linking to openmpi libraries.
see
test -d /tmp/IQ-TREE || mkdir -m 2775 /tmp/IQ-TREE/
cd /tmp/IQ-TREE && source /local/gensoft2/adm/etc/profile.d/modules.sh && module purge && module load eigen/3.1.2 gcc/4.9.0 cmake/2.8.12.2 && cmake -DIQTREE_FLAGS="omp" /local/gensoft2/src/IQ-TREE/IQ-TREE-1.5.0a
IQ-TREE flags : omp
Builde mode : Release
Target OS : Unix
Compiler : GNU Compiler (gcc)
Target binary : 64-bit
Parallel : OpenMP/PThreads
Vectorization : SSE3
C flags : -Wall -g -pthread -O3 -g
CXX flags : -fopenmp -O3 -g
Using system zlib
-- Configuring done
-- Generating done
-- Build files have been written to: /tmp/IQ-TREE
cd /tmp/IQ-TREE && source /local/gensoft2/adm/etc/profile.d/modules.sh && module purge && module load eigen/3.1.2 gcc/4.9.0 && make -j "$(grep -c ^processor /proc/cpuinfo)"
make[1]: Entering directory `/tmp/IQ-TREE'
make[2]: Entering directory `/tmp/IQ-TREE'
make[3]: Entering directory `/tmp/IQ-TREE'
make[3]: Entering directory `/tmp/IQ-TREE'
make[3]: Entering directory `/tmp/IQ-TREE'
make[3]: Entering directory `/tmp/IQ-TREE'
make[3]: Leaving directory `/tmp/IQ-TREE'
make[3]: Leaving directory `/tmp/IQ-TREE'
[ 1%] Built target pllavx
make[3]: Leaving directory `/tmp/IQ-TREE'
make[3]: Entering directory `/tmp/IQ-TREE'
[ 1%] make[3]: Leaving directory `/tmp/IQ-TREE'
Built target avxkernel
make[3]: Entering directory `/tmp/IQ-TREE'
make[3]: Leaving directory `/tmp/IQ-TREE'
[ 19%] [ 30%] Built target pll
Built target ncl
make[3]: Entering directory `/tmp/IQ-TREE'
make[3]: Leaving directory `/tmp/IQ-TREE'
make[3]: Entering directory `/tmp/IQ-TREE'
[ 31%] Built target lbfgsb
[ 36%] Built target whtest
make[3]: Entering directory `/tmp/IQ-TREE'
make[3]: Leaving directory `/tmp/IQ-TREE'
make[3]: Leaving directory `/tmp/IQ-TREE'
make[3]: Entering directory `/tmp/IQ-TREE'
[ 40%] [ 41%] Built target sprng
Built target vectorclass
make[3]: Leaving directory `/tmp/IQ-TREE'
[ 45%] Built target gsl
make[3]: Leaving directory `/tmp/IQ-TREE'
[ 59%] Built target model
make[3]: Entering directory `/tmp/IQ-TREE'
make[3]: Leaving directory `/tmp/IQ-TREE'
[100%] Built target iqtree
make[2]: Leaving directory `/tmp/IQ-TREE'
make[1]: Leaving directory `/tmp/IQ-TREE'
this generate the iqtree-omp binarie as expected, but hen running ldd on the produced binarie one can note that it tries to link to some mpi libraries.
ldd /tmp/IQ-TREE/iqtree-omp
linux-vdso.so.1 => (0x00007ffdf43fe000)
libz.so.1 => /lib64/libz.so.1 (0x0000003cb8e00000)
libmpi_cxx.so.1 => not found
libmpi.so.1 => not found
libstdc++.so.6 => /local/gensoft2/exe/gcc/4.9.0/lib64/libstdc++.so.6 (0x00007fe7cebbf000)
libm.so.6 => /lib64/libm.so.6 (0x0000003cb8600000)
libgomp.so.1 => /local/gensoft2/exe/gcc/4.9.0/lib64/libgomp.so.1 (0x00007fe7ce9a9000)
libgcc_s.so.1 => /local/gensoft2/exe/gcc/4.9.0/lib64/libgcc_s.so.1 (0x00007fe7ce793000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x0000003cb7e00000)
libc.so.6 => /lib64/libc.so.6 (0x0000003cb7a00000)
/lib64/ld-linux-x86-64.so.2 (0x0000003cb7600000)
librt.so.1 => /lib64/librt.so.1 (0x0000003cb8a00000)
regards
Eric
This is related to #38, as I found it when attempting to work around that problem.
The command:
$ ./iqtree -s humanMito.fasta -te humanMito.tree -m 'K3P+R{0.35,0.00001,0.35,0.0001,0.25,1.2,0.05,27}' -pre temp -redo -v -optfromgiven
should optimize the +R parameters, starting at the values specified on the command line. However, it optimizes the rates but not the proportions, in this case finishing at:
Rate parameters: A-C: 1.0000000000 A-G: 27.6510651518 A-T: 0.9658284715 C-G: 0.9658284715 C-T: 27.6510651518 G-T: 1.0000000000
Base frequencies: A: 0.2500000000 C: 0.2500000000 G: 0.2500000000 T: 0.2500000000
Site proportion and rates: (0.3500000000,0.0000060605) (0.3500000000,0.0000606046) (0.2500000000,0.7272557580) (0.0500000000,16.3632545544)
Optimal log-likelihood: -29971.6783480812
As I'm the person who added the -optfromgiven command line option, if I find the time I'll look into this one myself, but I'm logging it here so it doesn't get lost if I forget or don't get around to it.
Polymorphism-aware models allow to infer species tree in the presence of incomplete lineage sorting (http://www.iqtree.org/doc/Polymorphism-Aware-Models/). Right now it is in the PoMo
branch of the git repository, and will be merged into the master code in a near future version 1.5.X.
TODOs (as of Sept 8, 2016):
https://doi.org/10.1101/154542
Pers. comm. with Frederic Lemoine and Olivier Gascuel:
we have been working on a new way of computing branch supports, given a reference tree and a set of bootstrap trees.
The idea is to find, for each branch of the reference tree, the branch of each bootstrap tree that minimizes the taxa "transfer distance". The support of a ref branch will then be computed as 1-(the average distance over bootstrap trees)/(p-1). p being the number of taxa on the light side of this ref branch.
If you are interested and if you want more information, a preprint is available here:
https://doi.org/10.1101/154542
The paper is otherwise under revision and should hopefully be out in the near future.
We were wondering if you think it could be interesting to integrate this new support computation in IQ-TREE?
As it is a different way of comparing reference branches to bootstrap trees, it is compatible with all methods that generate a set of bootstrap trees (classical bootstrap as well as Ultra-Fast bootstrap).
We have a C implementation of this computation here:
https://github.com/evolbioinfo/booster
MPI parallelization allows to run IQ-TREE on distributed (multi-node) computing system. It can be used in conjunction with multicore (OpenMP) parallelization to fully utilize the system. The code is in the mpi
branch of the git repository, and will be merged into the master code in a near future version 1.5.X. It has been tested will several data sets showing near linear speedups.
mpirun -np 4 iqtree-mpi -s alignment...
mpirun -np 4 iqtree-omp-mpi -nt 2 -s alignment .... #4 processes, each with 2 cores
TODOs (as of Sept 8, 2016):
On some rare occasions IQ-TREE crashes when running the heterotachy model. Somehow the base frequency vector seems to go to a boundary condition, whereby one base has frequency equal to 1. I suspect the reason is that the upper and lower bounds (1 and 0.0001) on base frequencies are such that it is possible for the sum of fA+fC+fG > 1 as in the example below. The frequency of T is constrained to 1 - (fA+fC+fG) and therefore we have fT < 0 which results in the crash. If this is the cause then a possible solution would be to reduce the upper bound on the base frequencies from 1 down to 0.9997.
The heterotachy model is a mixture model allowing to have separate sets of branch lengths per mixture class, or branch-unlinked mixture models, which generalizes the Gamma as well as FreeRate heterogeneity across sites. It naturally accounts for heterotachous evolution (for more details see http://www.iqtree.org/doc/Heterotachy-Models/). The models will be integrated into version 1.6.0.
Right now the beta-testing implementation is available in the heterotachy
branch of the git repository. The run can be invoked by e.g.:
iqtree -s alignment -m GTR+FO+H3
where H
for Heterotachy and H3
means 3 heterotachous mixture classes.
Items done (as of Nov 16 2016):
/
so that it is readable by common tree viewers).CAT
model), so that user can specify something like -m C10+H
(CAT profile mixture model, where +H
will override the default Gamma model [+G
]).-m GTR+I+H3
.TODOs:
+H
.This is a small cosmetic enhancement request.
My command line:
./iqtree -s peabacteria.nex -te peabacteria.tree -sp partition.nex -redo -pre temp -m 010010+FO -v
Contents of partition.nex:
#nexus
begin sets;
charset pos1 = 2-1100\3;
charset pos2 = 3-1100\3;
charset pos3 = 1-1100\3;
end;
(This alignment did not start on codon partition 1.)
In the output:
Empirical state frequencies: 0.2290836658 0.2372652251 0.4254410892 0.1082100200
Empirical state frequencies: 0.2776395303 0.2408105227 0.1847849282 0.2967650188
Empirical state frequencies: 0.0844789551 0.4743215279 0.2938955756 0.1473039413
Please could you label which partition each of these state frequency vectors belongs to? (As it turns out, they are in order (pos1, pos2, pos3) which makes sense, but later when model fitting it does things in order (pos3, pos1, pos2). )
Hi there,
It looks as though the beta 1.6.4 release of IQ-tree is having some trouble restarting from its checkpoint:
CHECKPOINT: Candidate tree set restored, best LogL: -2991061.651
Finish initializing candidate tree set (18)
Current best tree score: -2991061.651 / CPU time: 0.000
Number of iterations: 124
Refining ufboot trees with NNI...
ERROR: Tree file does not start with an opening-bracket '(' (line 2 column 0)
This is after a crash due to hitting a memory limit set by LSF on our cluster, while running the CAT-Poisson model.
I'd appreciate any advice. The tree file does definitely begin with an opening bracket. (although, indeed, not on line 2). I'd love to be able to re-start and not have to recompute everything, if there's a straightforward fix -- or if it's a real bug, happy to provide more details to help debug!
Best,
Chris L
Hi All,
Im currently trying to construct a tree with data from two files. I have run the files individually with no issue. My partition file is as follows:
begin sets;
charset part1 = file.aln: *;
charset part2 = file2.fa: *;
charpartition mine = model:part1, model:part2;
end;
The command is
iqtree -spp test.nex
I then get the following error:
Reading partition model file test.nex ...
Reading partition part1 (model=model, aln=file.aln, seq=DNA, pos= *) ...
Reading alignment file file.aln ... ERROR: Cannot read file file.aln
I have used this file in a single data analysis (not partitioned) with no issue so I dont know what this is happening.
Thanks
Got a consent from @simonwhelan to implement ModelOMatic into IQ-TREE.
ModelOMatic: Fast and Automated Model Selection between RY, Nucleotide, Amino Acid, and Codon Substitution Models.
Simon Whelan James E. Allen Benjamin P. Blackburne David Talavera
Syst Biol (2015) 64 (1): 42-55. DOI: https://doi.org/10.1093/sysbio/syu062
https://github.com/simonwhelan/modelomatic
Further discussions:
@simonwhelan: I’m glad you like the method and I’m very happy for you to implement it into IQ-TREE. If you need any secondary verification of numbers for particular data sets let me know, but I think the standard output provides you with all the values you need.
I’m not sure whether IQ-TREE has RNA models, but our paper in GBE (https://academic.oup.com/gbe/article/6/1/65/664843/Assessing-the-State-of-Substitution-Models) does the same thing for the various different state-spaces in RNA models.
It should extended to the Dayhoff4 grouping easily providing you have the original data, preferably as codons so you can do the whole Codon/AA/Dayhoff4 comparison.
The way I think about the state-space projection is from a generative perspective, so we have data generated from the aggregated process (e.g. Dayhoff4) out to the tips of the tree. We then need a projection from those aggregated states to our general space (e.g. codons). The likelihood that this model generated the original data is exactly what we’re calculating. So that means in the inference framework the two parts are nicely orthogonal since the normal pruning algorithm can be applied on the aggregated alignment for the first part of the likelihood function, whereas the projection provides the second part in terms of the probability of getting the original alignment from our aggregated alignment.
This download appears to have been altered recently:
https://github.com/Cibiv/IQ-TREE/archive/v1.6.1.tar.gz
Error: SHA256 mismatch
Previos: 32ed901ec0d9b908218df9ac899378668abd1a0dd34f9c46cc5fcd6ff895d02c
Now: cdba4b68e15ef62b1717c5e5d951a925f4242de5f668cb267c2011bd7ea70541
Did you remove it, change it, and re-release it with the same version by any chance?
Currently testing +R is too slow. Some ideas to speed it up:
First perform standard model selection without +R. Then take only the best 2 matrices and all matrices with IC score of at most 10 points higher. Only these matrices will then be evaluated with +R.
Look at Smart model selection (SMS).
Jackknife = subsampling without replacement
With a new option like "-j 0.4" to jackknife 40% of the alignment sites.
Less priority: taxon jackknife, as jackknife trees do not have the same taxon set -> consensus tree construction does not work.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.