rnajena / viralclust Goto Github PK
View Code? Open in Web Editor NEWSmall pipeline to cluster viral genomes based on their k-mer content. WiP
License: GNU General Public License v3.0
Small pipeline to cluster viral genomes based on their k-mer content. WiP
License: GNU General Public License v3.0
If several NCBI IDs are found in the header, store all and check later, whether one of them exists in the ncbi_metainfo.pkl
file.
grep -v 'cluster_weight=1 in sumaclust.nf reicht nicht aus --> grep -v 'cluster_weight=1;'
otherwise cluster with size 10, 11, 12, .... 105.... or in general 1\d+ are filtered as well.
based on ORF density, sequences can be (roughly, heuristically) grouped into sense and antisense strand. Sometimes, both strands are interesting, thus it should be provided.
currently, the conda environments in viralclust are throwing errors, due to some numpy/scipy update. This should be resolved with better version management in the yml files
to be consistent with the other tools, those should be divided properly.
Using kmer frequencies is really demanding for the memory in the HDBSCAN module.
I am thinking of re-implementing this via a (maybe pickled) data dump on the harddrive, that is read into the memory on demand, line-by-line. More I/O, less RAM...
It seems like wget
sometimes fails to download some tar-balls correctly from the FTP server.
Re-downloading those might fix the issue, however, so far I am only able to detect invalid tar.gz
in the gunzip
step.
At the moment, the database has many different formats and resolutions of the collection date.
Sorting via YYYY/MM/DD and maybe removing the day
-field should be an improvement in terms of overview.
Implement multi-processing.
Most likely it has to be restricted by Memory and/or chunking the sequences for writing.
If no ORF is found at all, the "sorted" sequence file contains empty fasta records.
This might be changed to report the "original" sequence instead of an empty sequence.
sigh
apparently, the taxonomic lineage of viruses isn't consistent among viruses. sometimes, the species is the last entry, sometimes it is second to last and everything else gets shifted as well.
a regex that looks for the family/genus suffix (-viridae and -virus) should help to avoid mis-annotations in my results
The process reverseComp
is labeled with revcomp
, but the config files (conda and local) use the label revComp
(notice the upper case C). This leads to the process not using the python3 conda environment as intended.
When applied to short sequences (length ~200nt, roughly 2.500 sequences) and using k=5 for UMAP and hdbscan, the centroid sequences are empty.
Got this error message while running ViralClust with SARS-CoV-2 alpha genomes (152,307 non-redundant seqs).
Traceback (most recent call last): File "/home/nu76fet/programs/viralclust/bin/hdbscan_virus.py", line 663, in <module> perform_clustering() File "/home/nu76fet/programs/viralclust/bin/hdbscan_virus.py", line 604, in perform_clustering virusClusterer.determine_profile(multiPool) File "/home/nu76fet/programs/viralclust/bin/hdbscan_virus.py", line 267, in determine_profile allProfiles = p.map(self.profile, self.d_sequences.items()) File "/home/nu76fet/programs/viralclust/conda/hdbscan-9fec0a1dfe235db7d7c78f1a0bba3ac9/lib/python3.7/multiprocessing/pool.py", line 268, in map return self._map_async(func, iterable, mapstar, chunksize).get() File "/home/nu76fet/programs/viralclust/conda/hdbscan-9fec0a1dfe235db7d7c78f1a0bba3ac9/lib/python3.7/multiprocessing/pool.py", line 657, in get raise self._value File "/home/nu76fet/programs/viralclust/conda/hdbscan-9fec0a1dfe235db7d7c78f1a0bba3ac9/lib/python3.7/multiprocessing/pool.py", line 431, in _handle_tasks put(task) File "/home/nu76fet/programs/viralclust/conda/hdbscan-9fec0a1dfe235db7d7c78f1a0bba3ac9/lib/python3.7/multiprocessing/connection.py", line 206, in send self._send_bytes(_ForkingPickler.dumps(obj)) File "/home/nu76fet/programs/viralclust/conda/hdbscan-9fec0a1dfe235db7d7c78f1a0bba3ac9/lib/python3.7/multiprocessing/connection.py", line 393, in _send_bytes header = struct.pack("!i", n) struct.error: 'i' format requires -2147483648 <= number <= 2147483647
One could create UpSetR plots for the sets of representative genomes and also for the cluster that contain an optional GOI.
extend the NF pipeline with two parameter config profiles (--local
and --global
?); this should set the parameters of all clustering tools such that the corresponding structures are found, i.e. resolution of cluster structures are modified here.
Even though we'd like to keep all input sequences as they are without filtering out too much, having 40% N's within a genome is of no help for anyone.
the mv
command fails if HDBSCAN is able to cluster all sequences.
creating an empty file for the -1 cluster should work for now.
In the latest version (dev), the evaluation module is over-eager and wants to evaluate everything 5 times. While the extra effort is appreciated, it causes errors further downstream.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.