mbhall88 / drprg-paper Goto Github PK
View Code? Open in Web Editor NEWWorkflow and files associated with the paper for DrPRG
drprg-paper's People
Contributors
Watchers
drprg-paper's Issues
Collate samplesheets
For the main assessment we will need a collection of Illumina and Nanopore data with phenotypes.
So far, I have collated accessions for 44,880 Illumina isolates with the following phenotype information
In terms of gathering Nanopore data, a main question is whether we want to only used Nanopore data basecalled with the same version of guppy or not? This is an overarching question for the field too I guess. I am inclined to think the version doesn't matter for this kind of benchmark - provided it was basecalled with guppy. Thoughts?
One last question is whether we want to use all of the 43K Illumina isolates or just a subset based on phenotypes for certain drugs?
Filtering strategies for discrepant phenotypes
I think it might be wise for use to use some kind of phenotype filtering strategy.
mbhall88/drprg#21 (comment) is a prime example of why.
In the NEJM Cryptic paper I know they ignored any samples where
Laboratory error was assumed in instances in which three or more phenotypes were discordant with the genotype of an isolate or in which susceptible phenotypes were recorded despite the presence of the high-level resistance mutation katG S315T for isoniazid or rpoB S450L for rifampin.14 Such isolates were excluded from further analysis.
I wonder if we do something similar?
Rolling results
This issue will document the rolling results.
The first sneak peak is all 437 Nanopore isolates and 400 Illumina isolates (selected at random).
Nanopore
| Drug | Tool | FN(R) | FP(S) | Sensitivity (95% CI) | Specificity (95% CI) | MCC |
|---|---|---|---|---|---|---|
| Amikacin | Drprg | 1(15) | 3(157) | 93.3% (70.2-98.8%) | 98.1% (94.5-99.3%) | 0.8642875644329534 |
| Amikacin | Mykrobe | 1(15) | 3(157) | 93.3% (70.2-98.8%) | 98.1% (94.5-99.3%) | 0.8642875644329534 |
| Amikacin | Tbprofiler | 1(15) | 3(157) | 93.3% (70.2-98.8%) | 98.1% (94.5-99.3%) | 0.8642875644329534 |
| Capreomycin | Drprg | 1(3) | 1(64) | 66.7% (20.8-93.9%) | 98.4% (91.7-99.7%) | 0.6510416666666666 |
| Capreomycin | Mykrobe | 1(3) | 1(64) | 66.7% (20.8-93.9%) | 98.4% (91.7-99.7%) | 0.6510416666666666 |
| Capreomycin | Tbprofiler | 1(3) | 1(64) | 66.7% (20.8-93.9%) | 98.4% (91.7-99.7%) | 0.6510416666666666 |
| Ethambutol | Drprg | 7(29) | 21(328) | 75.9% (57.9-87.8%) | 93.6% (90.4-95.8%) | 0.5830010462966467 |
| Ethambutol | Mykrobe | 6(29) | 22(328) | 79.3% (61.6-90.2%) | 93.3% (90.1-95.5%) | 0.5975951983859127 |
| Ethambutol | Tbprofiler | 7(29) | 22(328) | 75.9% (57.9-87.8%) | 93.3% (90.1-95.5%) | 0.5747241429661474 |
| Ethionamide | Drprg | 26(30) | 1(86) | 13.3% (5.3-29.7%) | 98.8% (93.7-99.8%) | 0.2624057235284411 |
| Ethionamide | Mykrobe | 5(30) | 57(86) | 83.3% (66.4-92.7%) | 33.7% (24.6-44.2%) | 0.16405763204424978 |
| Ethionamide | Tbprofiler | 14(30) | 4(86) | 53.3% (36.1-69.8%) | 95.3% (88.6-98.2%) | 0.5643248464313276 |
| Isoniazid | Drprg | 35(117) | 3(265) | 70.1% (61.3-77.6%) | 98.9% (96.7-99.6%) | 0.7641591750285314 |
| Isoniazid | Mykrobe | 18(117) | 45(265) | 84.6% (77.0-90.0%) | 83.0% (78.0-87.1%) | 0.6432989433455073 |
| Isoniazid | Tbprofiler | 21(117) | 4(265) | 82.1% (74.1-88.0%) | 98.5% (96.2-99.4%) | 0.8445257661394283 |
| Kanamycin | Drprg | 0(3) | 2(118) | 100.0% (43.9-100.0%) | 98.3% (94.0-99.5%) | 0.7680042372764464 |
| Kanamycin | Mykrobe | 0(3) | 2(118) | 100.0% (43.9-100.0%) | 98.3% (94.0-99.5%) | 0.7680042372764464 |
| Kanamycin | Tbprofiler | 0(3) | 2(118) | 100.0% (43.9-100.0%) | 98.3% (94.0-99.5%) | 0.7680042372764464 |
| Moxifloxacin | Drprg | 0(0) | 1(1) | - | 0.0% (0.0-79.3%) | - |
| Moxifloxacin | Mykrobe | 0(0) | 1(1) | - | 0.0% (0.0-79.3%) | - |
| Moxifloxacin | Tbprofiler | 0(0) | 1(1) | - | 0.0% (0.0-79.3%) | - |
| Ofloxacin | Drprg | 13(15) | 1(158) | 13.3% (3.7-37.9%) | 99.4% (96.5-99.9%) | 0.27378347692948213 |
| Ofloxacin | Mykrobe | 3(15) | 4(158) | 80.0% (54.8-93.0%) | 97.5% (93.7-99.0%) | 0.7524691275756947 |
| Ofloxacin | Tbprofiler | 3(15) | 3(158) | 80.0% (54.8-93.0%) | 98.1% (94.6-99.4%) | 0.7810126582278482 |
| Pyrazinamide | Drprg | 16(28) | 3(243) | 42.9% (26.5-60.9%) | 98.8% (96.4-99.6%) | 0.5540455669886942 |
| Pyrazinamide | Mykrobe | 10(28) | 5(243) | 64.3% (45.8-79.3%) | 97.9% (95.3-99.1%) | 0.6796400181154288 |
| Pyrazinamide | Tbprofiler | 12(28) | 6(243) | 57.1% (39.1-73.5%) | 97.5% (94.7-98.9%) | 0.6093260891549527 |
| Rifampicin | Drprg | 8(77) | 6(287) | 89.6% (80.8-94.6%) | 97.9% (95.5-99.0%) | 0.8837166974977075 |
| Rifampicin | Mykrobe | 6(77) | 6(287) | 92.2% (84.0-96.4%) | 97.9% (95.5-99.0%) | 0.9011719987329744 |
| Rifampicin | Tbprofiler | 75(77) | 1(287) | 2.6% (0.7-9.0%) | 99.7% (98.1-99.9%) | 0.10159114367294636 |
| Streptomycin | Drprg | 9(55) | 16(126) | 83.6% (71.7-91.1%) | 87.3% (80.4-92.0%) | 0.6875051115519748 |
| Streptomycin | Mykrobe | 7(55) | 38(126) | 87.3% (76.0-93.7%) | 69.8% (61.3-77.2%) | 0.526015018770466 |
| Streptomycin | Tbprofiler | 15(55) | 19(126) | 72.7% (59.8-82.7%) | 84.9% (77.6-90.1%) | 0.5656453811464112 |
Next avenues of investigation:
- Figure out what has happened to cause the crazy dropout in tbprofiler RIF sensitivity - there must be a problem with the panel I created for it
- There is something weird going on with the PZA, FLQ and ETO sensitivities for drprg. Will also look into INH sensitivity
Positives:
- STM
Illumina
| Drug | Tool | FN(R) | FP(S) | Sensitivity (95% CI) | Specificity (95% CI) | MCC |
|---|---|---|---|---|---|---|
| Amikacin | Drprg | 0(15) | 1(166) | 100.0% (79.6-100.0%) | 99.4% (96.7-99.9%) | 0.9653250279768749 |
| Amikacin | Mykrobe | 1(15) | 1(166) | 93.3% (70.2-98.8%) | 99.4% (96.7-99.9%) | 0.9273092369477912 |
| Amikacin | Tbprofiler | 0(15) | 1(166) | 100.0% (79.6-100.0%) | 99.4% (96.7-99.9%) | 0.9653250279768749 |
| Capreomycin | Drprg | 2(13) | 4(93) | 84.6% (57.8-95.7%) | 95.7% (89.5-98.3%) | 0.7558571990231637 |
| Capreomycin | Mykrobe | 3(13) | 4(93) | 76.9% (49.7-91.8%) | 95.7% (89.5-98.3%) | 0.70359611692274 |
| Capreomycin | Tbprofiler | 2(13) | 4(93) | 84.6% (57.8-95.7%) | 95.7% (89.5-98.3%) | 0.7558571990231637 |
| Delamanid | Drprg | 1(1) | 0(93) | 0.0% (0.0-79.3%) | 100.0% (96.0-100.0%) | - |
| Delamanid | Mykrobe | 1(1) | 0(93) | 0.0% (0.0-79.3%) | 100.0% (96.0-100.0%) | - |
| Delamanid | Tbprofiler | 1(1) | 0(93) | 0.0% (0.0-79.3%) | 100.0% (96.0-100.0%) | - |
| Ethambutol | Drprg | 5(53) | 16(229) | 90.6% (79.7-95.9%) | 93.0% (89.0-95.7%) | 0.7795344823612268 |
| Ethambutol | Mykrobe | 5(53) | 17(229) | 90.6% (79.7-95.9%) | 92.6% (88.4-95.3%) | 0.7712443312992736 |
| Ethambutol | Tbprofiler | 5(53) | 18(229) | 90.6% (79.7-95.9%) | 92.1% (87.9-95.0%) | 0.7631197122668696 |
| Ethionamide | Drprg | 20(37) | 11(127) | 45.9% (31.0-61.6%) | 91.3% (85.2-95.1%) | 0.4141740735523773 |
| Ethionamide | Mykrobe | 11(37) | 15(127) | 70.3% (54.2-82.5%) | 88.2% (81.4-92.7%) | 0.5643018224168724 |
| Ethionamide | Tbprofiler | 10(37) | 16(127) | 73.0% (57.0-84.6%) | 87.4% (80.5-92.1%) | 0.5737592501981046 |
| Isoniazid | Drprg | 17(135) | 5(218) | 87.4% (80.8-92.0%) | 97.7% (94.7-99.0%) | 0.868118053524601 |
| Isoniazid | Mykrobe | 14(135) | 7(218) | 89.6% (83.3-93.7%) | 96.8% (93.5-98.4%) | 0.8735871080954598 |
| Isoniazid | Tbprofiler | 10(135) | 7(218) | 92.6% (86.9-95.9%) | 96.8% (93.5-98.4%) | 0.8977596305525896 |
| Kanamycin | Drprg | 4(25) | 2(168) | 84.0% (65.3-93.6%) | 98.8% (95.8-99.7%) | 0.8582554180919595 |
| Kanamycin | Mykrobe | 6(25) | 2(168) | 76.0% (56.6-88.5%) | 98.8% (95.8-99.7%) | 0.8066918414409607 |
| Kanamycin | Tbprofiler | 5(25) | 2(168) | 80.0% (60.9-91.1%) | 98.8% (95.8-99.7%) | 0.8327103314106743 |
| Levofloxacin | Drprg | 18(28) | 5(145) | 35.7% (20.7-54.2%) | 96.6% (92.2-98.5%) | 0.42231266491410374 |
| Levofloxacin | Mykrobe | 2(28) | 10(145) | 92.9% (77.4-98.0%) | 93.1% (87.8-96.2%) | 0.7799214704762708 |
| Levofloxacin | Tbprofiler | 1(28) | 11(145) | 96.4% (82.3-99.4%) | 92.4% (86.9-95.7%) | 0.7903590726235468 |
| Linezolid | Drprg | 0(0) | 0(128) | - | 100.0% (97.1-100.0%) | - |
| Linezolid | Mykrobe | 0(0) | 0(128) | - | 100.0% (97.1-100.0%) | - |
| Linezolid | Tbprofiler | 0(0) | 0(128) | - | 100.0% (97.1-100.0%) | - |
| Moxifloxacin | Drprg | 11(15) | 8(126) | 26.7% (10.9-52.0%) | 93.7% (88.0-96.7%) | 0.2244992239690201 |
| Moxifloxacin | Mykrobe | 2(15) | 18(126) | 86.7% (62.1-96.3%) | 85.7% (78.5-90.8%) | 0.5388625547887866 |
| Moxifloxacin | Tbprofiler | 2(15) | 20(126) | 86.7% (62.1-96.3%) | 84.1% (76.8-89.5%) | 0.5155328733959855 |
| Ofloxacin | Drprg | 3(3) | 0(5) | 0.0% (0.0-56.1%) | 100.0% (56.6-100.0%) | - |
| Ofloxacin | Mykrobe | 0(3) | 0(5) | 100.0% (43.9-100.0%) | 100.0% (56.6-100.0%) | 1.0 |
| Ofloxacin | Tbprofiler | 0(3) | 0(5) | 100.0% (43.9-100.0%) | 100.0% (56.6-100.0%) | 1.0 |
| Pyrazinamide | Drprg | 9(23) | 0(132) | 60.9% (40.8-77.8%) | 100.0% (97.2-100.0%) | 0.7548792871746883 |
| Pyrazinamide | Mykrobe | 2(23) | 2(132) | 91.3% (73.2-97.6%) | 98.5% (94.6-99.6%) | 0.8978919631093544 |
| Pyrazinamide | Tbprofiler | 2(23) | 2(132) | 91.3% (73.2-97.6%) | 98.5% (94.6-99.6%) | 0.8978919631093544 |
| Rifampicin | Drprg | 8(118) | 4(240) | 93.2% (87.2-96.5%) | 98.3% (95.8-99.4%) | 0.9237938244724586 |
| Rifampicin | Mykrobe | 6(118) | 4(240) | 94.9% (89.3-97.6%) | 98.3% (95.8-99.4%) | 0.936595807066951 |
| Rifampicin | Tbprofiler | 117(118) | 1(240) | 0.8% (0.1-4.6%) | 99.6% (97.7-99.9%) | 0.0271689721964718 |
| Streptomycin | Drprg | 12(42) | 1(64) | 71.4% (56.4-82.8%) | 98.4% (91.7-99.7%) | 0.7512240395539047 |
| Streptomycin | Mykrobe | 8(42) | 5(64) | 81.0% (66.7-90.0%) | 92.2% (83.0-96.6%) | 0.7418210904541338 |
| Streptomycin | Tbprofiler | 8(42) | 2(64) | 81.0% (66.7-90.0%) | 96.9% (89.3-99.1%) | 0.8037977341533646 |
Next avenues of investigation:
I'll probably wait until the Nanopore stuff is debugged and then run on a larger sample of data before debugging Illumina
Depth analysis
One piece of analysis that would be nice to investigate is how results scale as read depth is reduced.
My thoughts we randomly subsample each sample to discrete depth thresholds (y-axis) and plot the sensitivity/specificity? FN/FP? some other metric? on the x-axis
If everyone is happy with this analysis, I thin start with depth thresholds
- 100
- 60
- 30
- 15
- 10
- 5
If 5 still looks okay, we could go lower still.
Novel variants in false negative samples
It could be interesting for the discussion/results to see if there are any common novel mutations in false negative samples. These would either be gaps in the catalogue, which we could link back to the number stated in the WHO catalogue, or it could be mutations we find that aren't previously described
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.