lasersonlab / single-cell-experiments Goto Github PK

It's not installing anndata: https://travis-ci.org/lasersonlab/single-cell-experiments/builds/416327302

To make sure we follow formatting and docstring guidelines.

See e.g. https://jeffknupp.com/blog/2016/12/09/how-python-linters-will-save-your-large-python-project/

This includes:

• log1p
• scale
• filter_genes
• filter_genes_dispersion
• normalize_per_cell

to support the Zheng17 recipe: https://github.com/theislab/scanpy/blob/master/scanpy/preprocessing/recipes.py#L61.

Anndata should be able to read and write to zarr storage

Pyspark vs Apache Beam vs Dask

So it can more closely track scanpy, and hopefully eventually be merged there. We would add it as a submodule of this repo, just like we do for anndata.

Like this but for 10x data: https://nbviewer.jupyter.org/github/theislab/scanpy_usage/blob/master/170505_seurat/seurat.ipynb

Might be relevant: Jupyter+Spark+Google Cloud
https://cloud.google.com/blog/big-data/2017/02/google-cloud-platform-for-data-scientists-using-jupyter-notebooks-with-apache-spark-on-google-cloud

Now that scverse/anndata#38 has been merged we should try using the upstream branch, and the next release when that's available.

Tracking some data-format-conversion needs/functionality here.

WIP script for some conversions is at cli.py

Notes:

• AnnData's read_loom method loads the whole dataset into memory, which we'd like to avoid
• h5py only reads from local files
  • adding support there for reading from cloud-storage would be nice, but seems pretty nontrivial
  • @tomwhite has an implementation of this in java that we might rather rely on

How does it compare to the single process equivalent in scanpy? Can we make it go faster (bigger cluster, better memory/core settings, etc)?

We'll need to do this on a single machine owing to the lack of support for reading hdf5 in Python in a distributed way.