rnajena / viralclust Goto Github PK

If several NCBI IDs are found in the header, store all and check later, whether one of them exists in the ncbi_metainfo.pkl file.

grep -v 'cluster_weight=1 in sumaclust.nf reicht nicht aus --> grep -v 'cluster_weight=1;'

otherwise cluster with size 10, 11, 12, .... 105.... or in general 1\d+ are filtered as well.

based on ORF density, sequences can be (roughly, heuristically) grouped into sense and antisense strand. Sometimes, both strands are interesting, thus it should be provided.

currently, the conda environments in viralclust are throwing errors, due to some numpy/scipy update. This should be resolved with better version management in the yml files

to be consistent with the other tools, those should be divided properly.

Using kmer frequencies is really demanding for the memory in the HDBSCAN module.
I am thinking of re-implementing this via a (maybe pickled) data dump on the harddrive, that is read into the memory on demand, line-by-line. More I/O, less RAM...

It seems like wget sometimes fails to download some tar-balls correctly from the FTP server.
Re-downloading those might fix the issue, however, so far I am only able to detect invalid tar.gz in the gunzip step.

At the moment, the database has many different formats and resolutions of the collection date.
Sorting via YYYY/MM/DD and maybe removing the day-field should be an improvement in terms of overview.

Implement multi-processing.
Most likely it has to be restricted by Memory and/or chunking the sequences for writing.

If no ORF is found at all, the "sorted" sequence file contains empty fasta records.
This might be changed to report the "original" sequence instead of an empty sequence.

sigh
apparently, the taxonomic lineage of viruses isn't consistent among viruses. sometimes, the species is the last entry, sometimes it is second to last and everything else gets shifted as well.

a regex that looks for the family/genus suffix (-viridae and -virus) should help to avoid mis-annotations in my results

The process reverseComp is labeled with revcomp, but the config files (conda and local) use the label revComp (notice the upper case C). This leads to the process not using the python3 conda environment as intended.

When applied to short sequences (length ~200nt, roughly 2.500 sequences) and using k=5 for UMAP and hdbscan, the centroid sequences are empty.

Got this error message while running ViralClust with SARS-CoV-2 alpha genomes (152,307 non-redundant seqs).

Traceback (most recent call last): File "/home/nu76fet/programs/viralclust/bin/hdbscan_virus.py", line 663, in <module> perform_clustering() File "/home/nu76fet/programs/viralclust/bin/hdbscan_virus.py", line 604, in perform_clustering virusClusterer.determine_profile(multiPool) File "/home/nu76fet/programs/viralclust/bin/hdbscan_virus.py", line 267, in determine_profile allProfiles = p.map(self.profile, self.d_sequences.items()) File "/home/nu76fet/programs/viralclust/conda/hdbscan-9fec0a1dfe235db7d7c78f1a0bba3ac9/lib/python3.7/multiprocessing/pool.py", line 268, in map return self._map_async(func, iterable, mapstar, chunksize).get() File "/home/nu76fet/programs/viralclust/conda/hdbscan-9fec0a1dfe235db7d7c78f1a0bba3ac9/lib/python3.7/multiprocessing/pool.py", line 657, in get raise self._value File "/home/nu76fet/programs/viralclust/conda/hdbscan-9fec0a1dfe235db7d7c78f1a0bba3ac9/lib/python3.7/multiprocessing/pool.py", line 431, in _handle_tasks put(task) File "/home/nu76fet/programs/viralclust/conda/hdbscan-9fec0a1dfe235db7d7c78f1a0bba3ac9/lib/python3.7/multiprocessing/connection.py", line 206, in send self._send_bytes(_ForkingPickler.dumps(obj)) File "/home/nu76fet/programs/viralclust/conda/hdbscan-9fec0a1dfe235db7d7c78f1a0bba3ac9/lib/python3.7/multiprocessing/connection.py", line 393, in _send_bytes header = struct.pack("!i", n) struct.error: 'i' format requires -2147483648 <= number <= 2147483647

One could create UpSetR plots for the sets of representative genomes and also for the cluster that contain an optional GOI.

extend the NF pipeline with two parameter config profiles (--local and --global ?); this should set the parameters of all clustering tools such that the corresponding structures are found, i.e. resolution of cluster structures are modified here.

Even though we'd like to keep all input sequences as they are without filtering out too much, having 40% N's within a genome is of no help for anyone.

the mv command fails if HDBSCAN is able to cluster all sequences.
creating an empty file for the -1 cluster should work for now.

In the latest version (dev), the evaluation module is over-eager and wants to evaluate everything 5 times. While the extra effort is appreciated, it causes errors further downstream.