mbhall88 / drprg-paper Goto Github PK
View Code? Open in Web Editor NEWWorkflow and files associated with the paper for DrPRG
Workflow and files associated with the paper for DrPRG
For the main assessment we will need a collection of Illumina and Nanopore data with phenotypes.
So far, I have collated accessions for 44,880 Illumina isolates with the following phenotype information
In terms of gathering Nanopore data, a main question is whether we want to only used Nanopore data basecalled with the same version of guppy or not? This is an overarching question for the field too I guess. I am inclined to think the version doesn't matter for this kind of benchmark - provided it was basecalled with guppy. Thoughts?
One last question is whether we want to use all of the 43K Illumina isolates or just a subset based on phenotypes for certain drugs?
I think it might be wise for use to use some kind of phenotype filtering strategy.
mbhall88/drprg#21 (comment) is a prime example of why.
In the NEJM Cryptic paper I know they ignored any samples where
Laboratory error was assumed in instances in which three or more phenotypes were discordant with the genotype of an isolate or in which susceptible phenotypes were recorded despite the presence of the high-level resistance mutation katG S315T for isoniazid or rpoB S450L for rifampin.14 Such isolates were excluded from further analysis.
I wonder if we do something similar?
This issue will document the rolling results.
The first sneak peak is all 437 Nanopore isolates and 400 Illumina isolates (selected at random).
Drug | Tool | FN(R) | FP(S) | Sensitivity (95% CI) | Specificity (95% CI) | MCC |
---|---|---|---|---|---|---|
Amikacin | Drprg | 1(15) | 3(157) | 93.3% (70.2-98.8%) | 98.1% (94.5-99.3%) | 0.8642875644329534 |
Amikacin | Mykrobe | 1(15) | 3(157) | 93.3% (70.2-98.8%) | 98.1% (94.5-99.3%) | 0.8642875644329534 |
Amikacin | Tbprofiler | 1(15) | 3(157) | 93.3% (70.2-98.8%) | 98.1% (94.5-99.3%) | 0.8642875644329534 |
Capreomycin | Drprg | 1(3) | 1(64) | 66.7% (20.8-93.9%) | 98.4% (91.7-99.7%) | 0.6510416666666666 |
Capreomycin | Mykrobe | 1(3) | 1(64) | 66.7% (20.8-93.9%) | 98.4% (91.7-99.7%) | 0.6510416666666666 |
Capreomycin | Tbprofiler | 1(3) | 1(64) | 66.7% (20.8-93.9%) | 98.4% (91.7-99.7%) | 0.6510416666666666 |
Ethambutol | Drprg | 7(29) | 21(328) | 75.9% (57.9-87.8%) | 93.6% (90.4-95.8%) | 0.5830010462966467 |
Ethambutol | Mykrobe | 6(29) | 22(328) | 79.3% (61.6-90.2%) | 93.3% (90.1-95.5%) | 0.5975951983859127 |
Ethambutol | Tbprofiler | 7(29) | 22(328) | 75.9% (57.9-87.8%) | 93.3% (90.1-95.5%) | 0.5747241429661474 |
Ethionamide | Drprg | 26(30) | 1(86) | 13.3% (5.3-29.7%) | 98.8% (93.7-99.8%) | 0.2624057235284411 |
Ethionamide | Mykrobe | 5(30) | 57(86) | 83.3% (66.4-92.7%) | 33.7% (24.6-44.2%) | 0.16405763204424978 |
Ethionamide | Tbprofiler | 14(30) | 4(86) | 53.3% (36.1-69.8%) | 95.3% (88.6-98.2%) | 0.5643248464313276 |
Isoniazid | Drprg | 35(117) | 3(265) | 70.1% (61.3-77.6%) | 98.9% (96.7-99.6%) | 0.7641591750285314 |
Isoniazid | Mykrobe | 18(117) | 45(265) | 84.6% (77.0-90.0%) | 83.0% (78.0-87.1%) | 0.6432989433455073 |
Isoniazid | Tbprofiler | 21(117) | 4(265) | 82.1% (74.1-88.0%) | 98.5% (96.2-99.4%) | 0.8445257661394283 |
Kanamycin | Drprg | 0(3) | 2(118) | 100.0% (43.9-100.0%) | 98.3% (94.0-99.5%) | 0.7680042372764464 |
Kanamycin | Mykrobe | 0(3) | 2(118) | 100.0% (43.9-100.0%) | 98.3% (94.0-99.5%) | 0.7680042372764464 |
Kanamycin | Tbprofiler | 0(3) | 2(118) | 100.0% (43.9-100.0%) | 98.3% (94.0-99.5%) | 0.7680042372764464 |
Moxifloxacin | Drprg | 0(0) | 1(1) | - | 0.0% (0.0-79.3%) | - |
Moxifloxacin | Mykrobe | 0(0) | 1(1) | - | 0.0% (0.0-79.3%) | - |
Moxifloxacin | Tbprofiler | 0(0) | 1(1) | - | 0.0% (0.0-79.3%) | - |
Ofloxacin | Drprg | 13(15) | 1(158) | 13.3% (3.7-37.9%) | 99.4% (96.5-99.9%) | 0.27378347692948213 |
Ofloxacin | Mykrobe | 3(15) | 4(158) | 80.0% (54.8-93.0%) | 97.5% (93.7-99.0%) | 0.7524691275756947 |
Ofloxacin | Tbprofiler | 3(15) | 3(158) | 80.0% (54.8-93.0%) | 98.1% (94.6-99.4%) | 0.7810126582278482 |
Pyrazinamide | Drprg | 16(28) | 3(243) | 42.9% (26.5-60.9%) | 98.8% (96.4-99.6%) | 0.5540455669886942 |
Pyrazinamide | Mykrobe | 10(28) | 5(243) | 64.3% (45.8-79.3%) | 97.9% (95.3-99.1%) | 0.6796400181154288 |
Pyrazinamide | Tbprofiler | 12(28) | 6(243) | 57.1% (39.1-73.5%) | 97.5% (94.7-98.9%) | 0.6093260891549527 |
Rifampicin | Drprg | 8(77) | 6(287) | 89.6% (80.8-94.6%) | 97.9% (95.5-99.0%) | 0.8837166974977075 |
Rifampicin | Mykrobe | 6(77) | 6(287) | 92.2% (84.0-96.4%) | 97.9% (95.5-99.0%) | 0.9011719987329744 |
Rifampicin | Tbprofiler | 75(77) | 1(287) | 2.6% (0.7-9.0%) | 99.7% (98.1-99.9%) | 0.10159114367294636 |
Streptomycin | Drprg | 9(55) | 16(126) | 83.6% (71.7-91.1%) | 87.3% (80.4-92.0%) | 0.6875051115519748 |
Streptomycin | Mykrobe | 7(55) | 38(126) | 87.3% (76.0-93.7%) | 69.8% (61.3-77.2%) | 0.526015018770466 |
Streptomycin | Tbprofiler | 15(55) | 19(126) | 72.7% (59.8-82.7%) | 84.9% (77.6-90.1%) | 0.5656453811464112 |
Next avenues of investigation:
Positives:
Drug | Tool | FN(R) | FP(S) | Sensitivity (95% CI) | Specificity (95% CI) | MCC |
---|---|---|---|---|---|---|
Amikacin | Drprg | 0(15) | 1(166) | 100.0% (79.6-100.0%) | 99.4% (96.7-99.9%) | 0.9653250279768749 |
Amikacin | Mykrobe | 1(15) | 1(166) | 93.3% (70.2-98.8%) | 99.4% (96.7-99.9%) | 0.9273092369477912 |
Amikacin | Tbprofiler | 0(15) | 1(166) | 100.0% (79.6-100.0%) | 99.4% (96.7-99.9%) | 0.9653250279768749 |
Capreomycin | Drprg | 2(13) | 4(93) | 84.6% (57.8-95.7%) | 95.7% (89.5-98.3%) | 0.7558571990231637 |
Capreomycin | Mykrobe | 3(13) | 4(93) | 76.9% (49.7-91.8%) | 95.7% (89.5-98.3%) | 0.70359611692274 |
Capreomycin | Tbprofiler | 2(13) | 4(93) | 84.6% (57.8-95.7%) | 95.7% (89.5-98.3%) | 0.7558571990231637 |
Delamanid | Drprg | 1(1) | 0(93) | 0.0% (0.0-79.3%) | 100.0% (96.0-100.0%) | - |
Delamanid | Mykrobe | 1(1) | 0(93) | 0.0% (0.0-79.3%) | 100.0% (96.0-100.0%) | - |
Delamanid | Tbprofiler | 1(1) | 0(93) | 0.0% (0.0-79.3%) | 100.0% (96.0-100.0%) | - |
Ethambutol | Drprg | 5(53) | 16(229) | 90.6% (79.7-95.9%) | 93.0% (89.0-95.7%) | 0.7795344823612268 |
Ethambutol | Mykrobe | 5(53) | 17(229) | 90.6% (79.7-95.9%) | 92.6% (88.4-95.3%) | 0.7712443312992736 |
Ethambutol | Tbprofiler | 5(53) | 18(229) | 90.6% (79.7-95.9%) | 92.1% (87.9-95.0%) | 0.7631197122668696 |
Ethionamide | Drprg | 20(37) | 11(127) | 45.9% (31.0-61.6%) | 91.3% (85.2-95.1%) | 0.4141740735523773 |
Ethionamide | Mykrobe | 11(37) | 15(127) | 70.3% (54.2-82.5%) | 88.2% (81.4-92.7%) | 0.5643018224168724 |
Ethionamide | Tbprofiler | 10(37) | 16(127) | 73.0% (57.0-84.6%) | 87.4% (80.5-92.1%) | 0.5737592501981046 |
Isoniazid | Drprg | 17(135) | 5(218) | 87.4% (80.8-92.0%) | 97.7% (94.7-99.0%) | 0.868118053524601 |
Isoniazid | Mykrobe | 14(135) | 7(218) | 89.6% (83.3-93.7%) | 96.8% (93.5-98.4%) | 0.8735871080954598 |
Isoniazid | Tbprofiler | 10(135) | 7(218) | 92.6% (86.9-95.9%) | 96.8% (93.5-98.4%) | 0.8977596305525896 |
Kanamycin | Drprg | 4(25) | 2(168) | 84.0% (65.3-93.6%) | 98.8% (95.8-99.7%) | 0.8582554180919595 |
Kanamycin | Mykrobe | 6(25) | 2(168) | 76.0% (56.6-88.5%) | 98.8% (95.8-99.7%) | 0.8066918414409607 |
Kanamycin | Tbprofiler | 5(25) | 2(168) | 80.0% (60.9-91.1%) | 98.8% (95.8-99.7%) | 0.8327103314106743 |
Levofloxacin | Drprg | 18(28) | 5(145) | 35.7% (20.7-54.2%) | 96.6% (92.2-98.5%) | 0.42231266491410374 |
Levofloxacin | Mykrobe | 2(28) | 10(145) | 92.9% (77.4-98.0%) | 93.1% (87.8-96.2%) | 0.7799214704762708 |
Levofloxacin | Tbprofiler | 1(28) | 11(145) | 96.4% (82.3-99.4%) | 92.4% (86.9-95.7%) | 0.7903590726235468 |
Linezolid | Drprg | 0(0) | 0(128) | - | 100.0% (97.1-100.0%) | - |
Linezolid | Mykrobe | 0(0) | 0(128) | - | 100.0% (97.1-100.0%) | - |
Linezolid | Tbprofiler | 0(0) | 0(128) | - | 100.0% (97.1-100.0%) | - |
Moxifloxacin | Drprg | 11(15) | 8(126) | 26.7% (10.9-52.0%) | 93.7% (88.0-96.7%) | 0.2244992239690201 |
Moxifloxacin | Mykrobe | 2(15) | 18(126) | 86.7% (62.1-96.3%) | 85.7% (78.5-90.8%) | 0.5388625547887866 |
Moxifloxacin | Tbprofiler | 2(15) | 20(126) | 86.7% (62.1-96.3%) | 84.1% (76.8-89.5%) | 0.5155328733959855 |
Ofloxacin | Drprg | 3(3) | 0(5) | 0.0% (0.0-56.1%) | 100.0% (56.6-100.0%) | - |
Ofloxacin | Mykrobe | 0(3) | 0(5) | 100.0% (43.9-100.0%) | 100.0% (56.6-100.0%) | 1.0 |
Ofloxacin | Tbprofiler | 0(3) | 0(5) | 100.0% (43.9-100.0%) | 100.0% (56.6-100.0%) | 1.0 |
Pyrazinamide | Drprg | 9(23) | 0(132) | 60.9% (40.8-77.8%) | 100.0% (97.2-100.0%) | 0.7548792871746883 |
Pyrazinamide | Mykrobe | 2(23) | 2(132) | 91.3% (73.2-97.6%) | 98.5% (94.6-99.6%) | 0.8978919631093544 |
Pyrazinamide | Tbprofiler | 2(23) | 2(132) | 91.3% (73.2-97.6%) | 98.5% (94.6-99.6%) | 0.8978919631093544 |
Rifampicin | Drprg | 8(118) | 4(240) | 93.2% (87.2-96.5%) | 98.3% (95.8-99.4%) | 0.9237938244724586 |
Rifampicin | Mykrobe | 6(118) | 4(240) | 94.9% (89.3-97.6%) | 98.3% (95.8-99.4%) | 0.936595807066951 |
Rifampicin | Tbprofiler | 117(118) | 1(240) | 0.8% (0.1-4.6%) | 99.6% (97.7-99.9%) | 0.0271689721964718 |
Streptomycin | Drprg | 12(42) | 1(64) | 71.4% (56.4-82.8%) | 98.4% (91.7-99.7%) | 0.7512240395539047 |
Streptomycin | Mykrobe | 8(42) | 5(64) | 81.0% (66.7-90.0%) | 92.2% (83.0-96.6%) | 0.7418210904541338 |
Streptomycin | Tbprofiler | 8(42) | 2(64) | 81.0% (66.7-90.0%) | 96.9% (89.3-99.1%) | 0.8037977341533646 |
Next avenues of investigation:
I'll probably wait until the Nanopore stuff is debugged and then run on a larger sample of data before debugging Illumina
One piece of analysis that would be nice to investigate is how results scale as read depth is reduced.
My thoughts we randomly subsample each sample to discrete depth thresholds (y-axis) and plot the sensitivity/specificity? FN/FP? some other metric? on the x-axis
If everyone is happy with this analysis, I thin start with depth thresholds
If 5 still looks okay, we could go lower still.
It could be interesting for the discussion/results to see if there are any common novel mutations in false negative samples. These would either be gaps in the catalogue, which we could link back to the number stated in the WHO catalogue, or it could be mutations we find that aren't previously described
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.