phiweger / clonepool Goto Github PK

View Code? Open in Web Editor NEW

1.0 1.0 0.0 2.79 MB

Increased PCR screening capacity using a multi-replicate pooling scheme

License: Other

Python 1.22% R 0.03% TeX 98.42% Jupyter Notebook 0.32%

clonepool's People

Contributors

Stargazers

Watchers

clonepool's Issues

Add plot cmd and port plotting from R to Python

There is some R code floating around in the source files that creates a plot from aggregated simulation results. Since the plotting is done via ggplot, which is also available from Python, it should be simple to port the code and get rid of the R dependency. The plotting feature should be exposed to the user via a plot cmd.

Auto-build wheels on pushes / merges

Via Github Actions, it should be easy to implement a basic continuous integration workflow that builds wheels of clonepool. This would allow the user to simply download clonepool in a single file without having to mess with the Git repo at all.

fresh install issue (google colab) on master 5e5b344

i get in a new colab session now the following error while installing:
command:

!git clone https://github.com/phiweger/clonepool.git
!cd clonepool/ && pip install -e .
!clonepool

error:

ERROR: pytest 3.6.4 has requirement pluggy<0.8,>=0.5, but you'll have pluggy 0.13.1 which is incompatible.
ERROR: google-colab 1.0.0 has requirement six~=1.12.0, but you'll have six 1.14.0 which is incompatible.
ERROR: datascience 0.10.6 has requirement folium==0.2.1, but you'll have folium 0.8.3 which is incompatible.
ERROR: albumentations 0.1.12 has requirement imgaug<0.2.7,>=0.2.5, but you'll have imgaug 0.2.9 which is incompatible.

full install procedure:

Cloning into 'clonepool'...
remote: Enumerating objects: 75, done.
remote: Counting objects: 100% (75/75), done.
remote: Compressing objects: 100% (54/54), done.
remote: Total 493 (delta 32), reused 54 (delta 21), pack-reused 418
Receiving objects: 100% (493/493), 3.59 MiB | 17.17 MiB/s, done.
Resolving deltas: 100% (262/262), done.
Obtaining file:///content/clonepool
Collecting tox
  Downloading https://files.pythonhosted.org/packages/7b/4b/a90a0a89db60fc39fc92e31e7da436177a12c06038973c8b7199f47b47c0/tox-3.14.6-py2.py3-none-any.whl (81kB)
     |████████████████████████████████| 81kB 2.1MB/s 
Requirement already satisfied: pytest in /usr/local/lib/python3.6/dist-packages (from clonepool==0.1) (3.6.4)
Requirement already satisfied: click in /usr/local/lib/python3.6/dist-packages (from clonepool==0.1) (7.1.1)
Requirement already satisfied: networkx in /usr/local/lib/python3.6/dist-packages (from clonepool==0.1) (2.4)
Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from clonepool==0.1) (1.18.2)
Requirement already satisfied: tqdm in /usr/local/lib/python3.6/dist-packages (from clonepool==0.1) (4.38.0)
Requirement already satisfied: more-itertools in /usr/local/lib/python3.6/dist-packages (from clonepool==0.1) (8.2.0)
Requirement already satisfied: filelock<4,>=3.0.0 in /usr/local/lib/python3.6/dist-packages (from tox->clonepool==0.1) (3.0.12)
Collecting six<2,>=1.14.0
  Downloading https://files.pythonhosted.org/packages/65/eb/1f97cb97bfc2390a276969c6fae16075da282f5058082d4cb10c6c5c1dba/six-1.14.0-py2.py3-none-any.whl
Collecting virtualenv!=20.0.0,!=20.0.1,!=20.0.2,!=20.0.3,!=20.0.4,!=20.0.5,!=20.0.6,!=20.0.7,>=16.0.0
  Downloading https://files.pythonhosted.org/packages/23/1a/d55f5fb904564365c3d364fd41701a74ad497379f04fd548cccc3f227d98/virtualenv-20.0.18-py2.py3-none-any.whl (4.6MB)
     |████████████████████████████████| 4.6MB 8.0MB/s 
Collecting pluggy<1,>=0.12.0
  Downloading https://files.pythonhosted.org/packages/a0/28/85c7aa31b80d150b772fbe4a229487bc6644da9ccb7e427dd8cc60cb8a62/pluggy-0.13.1-py2.py3-none-any.whl
Requirement already satisfied: packaging>=14 in /usr/local/lib/python3.6/dist-packages (from tox->clonepool==0.1) (20.3)
Requirement already satisfied: importlib-metadata<2,>=0.12; python_version < "3.8" in /usr/local/lib/python3.6/dist-packages (from tox->clonepool==0.1) (1.6.0)
Collecting toml>=0.9.4
  Downloading https://files.pythonhosted.org/packages/a2/12/ced7105d2de62fa7c8fb5fce92cc4ce66b57c95fb875e9318dba7f8c5db0/toml-0.10.0-py2.py3-none-any.whl
Requirement already satisfied: py<2,>=1.4.17 in /usr/local/lib/python3.6/dist-packages (from tox->clonepool==0.1) (1.8.1)
Requirement already satisfied: setuptools in /usr/local/lib/python3.6/dist-packages (from pytest->clonepool==0.1) (46.1.3)
Requirement already satisfied: atomicwrites>=1.0 in /usr/local/lib/python3.6/dist-packages (from pytest->clonepool==0.1) (1.3.0)
Requirement already satisfied: attrs>=17.4.0 in /usr/local/lib/python3.6/dist-packages (from pytest->clonepool==0.1) (19.3.0)
Requirement already satisfied: decorator>=4.3.0 in /usr/local/lib/python3.6/dist-packages (from networkx->clonepool==0.1) (4.4.2)
Collecting importlib-resources<2,>=1.0; python_version < "3.7"
  Downloading https://files.pythonhosted.org/packages/8f/36/e678bc02cad98e566fbd7e6711ab2f389fc9adfb1edbb9f553eae41f9f97/importlib_resources-1.4.0-py2.py3-none-any.whl
Collecting appdirs<2,>=1.4.3
  Downloading https://files.pythonhosted.org/packages/56/eb/810e700ed1349edde4cbdc1b2a21e28cdf115f9faf263f6bbf8447c1abf3/appdirs-1.4.3-py2.py3-none-any.whl
Collecting distlib<1,>=0.3.0
  Downloading https://files.pythonhosted.org/packages/7d/29/694a3a4d7c0e1aef76092e9167fbe372e0f7da055f5dcf4e1313ec21d96a/distlib-0.3.0.zip (571kB)
     |████████████████████████████████| 573kB 47.2MB/s 
Requirement already satisfied: pyparsing>=2.0.2 in /usr/local/lib/python3.6/dist-packages (from packaging>=14->tox->clonepool==0.1) (2.4.7)
Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.6/dist-packages (from importlib-metadata<2,>=0.12; python_version < "3.8"->tox->clonepool==0.1) (3.1.0)
Building wheels for collected packages: distlib
  Building wheel for distlib (setup.py) ... done
  Created wheel for distlib: filename=distlib-0.3.0-cp36-none-any.whl size=340429 sha256=a37f357bfee341caff54d25376016a05eb71ce20b095e87c40ca477b05ad5c5f
  Stored in directory: /root/.cache/pip/wheels/6e/e8/db/c73dae4867666e89ba3cfbc4b5c092446f0e584eda6f409cbb
Successfully built distlib
ERROR: pytest 3.6.4 has requirement pluggy<0.8,>=0.5, but you'll have pluggy 0.13.1 which is incompatible.
ERROR: google-colab 1.0.0 has requirement six~=1.12.0, but you'll have six 1.14.0 which is incompatible.
ERROR: datascience 0.10.6 has requirement folium==0.2.1, but you'll have folium 0.8.3 which is incompatible.
ERROR: albumentations 0.1.12 has requirement imgaug<0.2.7,>=0.2.5, but you'll have imgaug 0.2.9 which is incompatible.
Installing collected packages: six, importlib-resources, appdirs, distlib, virtualenv, pluggy, toml, tox, clonepool
  Found existing installation: six 1.12.0
    Uninstalling six-1.12.0:
      Successfully uninstalled six-1.12.0
  Found existing installation: pluggy 0.7.1
    Uninstalling pluggy-0.7.1:
      Successfully uninstalled pluggy-0.7.1
  Running setup.py develop for clonepool
Successfully installed appdirs-1.4.3 clonepool distlib-0.3.0 importlib-resources-1.4.0 pluggy-0.13.1 six-1.14.0 toml-0.10.0 tox-3.14.6 virtualenv-20.0.18
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/pkg_resources/__init__.py", line 584, in _build_master
    ws.require(__requires__)
  File "/usr/local/lib/python3.6/dist-packages/pkg_resources/__init__.py", line 901, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/local/lib/python3.6/dist-packages/pkg_resources/__init__.py", line 792, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (pluggy 0.13.1 (/usr/local/lib/python3.6/dist-packages), Requirement.parse('pluggy<0.8,>=0.5'), {'pytest'})

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/clonepool", line 6, in <module>
    from pkg_resources import load_entry_point
  File "/usr/local/lib/python3.6/dist-packages/pkg_resources/__init__.py", line 3258, in <module>
    @_call_aside
  File "/usr/local/lib/python3.6/dist-packages/pkg_resources/__init__.py", line 3242, in _call_aside
    f(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/pkg_resources/__init__.py", line 3271, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/usr/local/lib/python3.6/dist-packages/pkg_resources/__init__.py", line 586, in _build_master
    return cls._build_from_requirements(__requires__)
  File "/usr/local/lib/python3.6/dist-packages/pkg_resources/__init__.py", line 599, in _build_from_requirements
    dists = ws.resolve(reqs, Environment())
  File "/usr/local/lib/python3.6/dist-packages/pkg_resources/__init__.py", line 787, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'pluggy<0.8,>=0.5' distribution was not found and is required by pytest

it would be good to release a candidate so i can install a specific and tested release on colab. otherwise some will face a few "master breaks install issue".

e.g. this works in google colab (this commit was okay)

!git clone https://github.com/phiweger/clonepool.git
!cd clonepool/ && git reset --hard 5da7bd05895cb62fe597b68274b758561365939f && pip install -e .
!clonepool

so either we add via git reset a specific working commit hash or you create a functional/tested release/tag to clone ;)

simulate FP, FN

this is just to remind us that this is on the TODO before writing the draft @xileF1337

the number does not matter

I remember we talked about whether the number of pools matters -- I just simulated this and it does not. ie the figures for 50 and 94 pools look indistinguishable, which to me makes sense as the params that determine the effective number of samples are just prevalence and and pool size (how many samples per pool) -- if you agree simply close this issue @xileF1337 :)

use sequential pool splits to increase efficiency even further

https://www.medrxiv.org/content/10.1101/2020.04.10.20061176v1

update README and rm sample n

Distribute samples to pools more uniformely

When sampling the positions, weigh each pool according to the number of its free slots. Low priority.

resolve routine bug

I think I found my mistake:

I basically assume in the above linked line that I can resolve a pool where the only remaining sample is uncertain as positive. However, I put this resolved sample in the same category (i.e. "0") as the samples that are negative. Then I iterate assuming that this means I can resolve "2nd degree" positives, which I cannot. So very good catch here @xileF1337 to find this logic error, many thanks!

feedback

@replikation @xileF1337 the first draft of the paper is now in the "write" branch, plz give me your feedback either under this issue or as comments in the pdf.

false positives in the current resolve strategy

# print out the positive samples
clonepool layout -n 235 -P 5 --simulate -p 0.1 test/layout.csv
[13, 14, 22, 40, 44, 49, 50, 76, 84, 86, 112, 114, 125, 140, 146, 147, 153, 179, 183, 187, 189, 198, 209, 219]
clonepool resolve --layout test/layout.csv test/results.csv
# Effective number of samples: 1.7153
grep "+" test/results.csv

127	+ --> FP
13	+
153	+
2	+ --> FP
219	+
50	+

results -- what are they?

two params matter, prevalence and pool size, given FP and FN == 0
the larger the pool size, the more effective are replicates, eg at pool size 10 it seems 2 replicates should be chosen while at pool size 20 we should select 3 to cover a wide range of prevalence.
at low prevalence/ small pool size, replicates make no sense
for coronavirus we suggest pool size 10 (virus detection should still be ok given published PCR Ct values from other studies) and 2 replicates, which would increase eff number of samples 2-5 fold (given prevalence) compared to one-sample-per-tube baseline and is about 50% more effective than naive "1-replicate" pools.

command line tool

API suggestion:

head -n3 samples.csv
# sample
# s1
# s2

# Assign samples to pools
clonepool layout samples.txt -n 94 -p 0.05 --size 10 -o layout.csv

head -n3 layout.csv
# pool,sample
# 1,s1
# 1,s2

# Now record result of the qPCR in a table
head -n3 pools.state.csv
# pool,state
# 1,-
# 2,+

# Resolve sample state
clonepool resolve --layout layout.csv -i pools.state.csv -o samples.state.csv

head -n4 samples.state.csv
# sample,state
# s1,+
# s2,-
# s3,NA

# All samples w/ NA need retesting

Add automatic testing

We should have at least some basic unit testing. For the beginning, have it check that clonepool --help runs properly and that import clonepool works.

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.