Pheniqs on very large barcode spaces about pheniqs HOT 1 OPEN

Hi Derek

PAMLD computes full Bayesian probabilities, that means it will compute the conditional probability of each possible barcode sequence for each observation, and then compute the posterior using all those and the priors. With a white list of 2-3 biilion possible barcode sequences this will be both slow and pointless. The conditional probabilities for the vast majority of possibilities will be extremely small and contribute almost nothing to the posterior. To be honest, even creating a JSON configuration file with such a long list might be challenging and require a significant amount of memory and load time.

The most realistic approach would be to try and narrow down your list. You can try and extract the relevant 18 bases from your data and do a basic sort|uniq -c to see the possible combinations. You can also try and run an initial MDD run and use the results from that run to compute priors for PAMLD. With a list this long, MDD will likely not allow any errors since almost every sequence is possible. That means MDD will stop scanning once it hits the first match. That said, it might not be that different from a basic sort|uniq. since almost every sequence is allowed to be a barcode, there is not much room for "error correcting" and I am not sure a bayesian approach even makes sense. If what you want is speed with no tolerance for errors that would be best achieved with a tool that uses suffix trees (trie). I think this one is mentioned in the pheniqs manuscript: https://academic.oup.com/bioinformatics/article/34/22/3924/5026649

from pheniqs.