Git Product home page Git Product logo

cosmos-tools's Introduction

COSMOS

Code to extract neuronal traces from a multi-focal microscope (COSMOS: Cortical Observation by Synchronous Multifocal Optical Sampling), and then to analyze these traces.

Installation instructions

It is recommended that you use an Anaconda environment for running this package. The following command line instructions assume Anaconda has been installed.

Download the repository

git clone https://github.com/izkula/cosmos-tools.git

Install the environment

Navigate to the cosmos-tools folder created by git and then run:

conda env create --file cosmos3requirements.yml -n cosmostools3

Activate the new environment

source activate cosmostools3

Setup the cosmos package

python setup.py develop

Optionally setup ipywidgets package for jupyter notebook (only required for trace_merge_script.ipynb, and sometimes causes weird problems)

conda install -c conda-forge ipywidgets

(For some reason, ipywidgets, which is necessary only for trace_merge_script.ipynb, appears to be required to be installed separately, after the environment has already been created. There have also been some issues when removing conda environments after ipywidgets has been installed, but remaking the environments appears to eventually work.)

Setup CNMF-E package for neural source extraction

The CNMF-E implementation we use is written in MATLAB (a python implementation is in progress, and may be ready by the time you are reading this). This package uses the version of CNMF-E from June 27, 2018 (commit # ddb865832f52c81725859df8b5e032b4acb421e9). A MATLAB license is thus required for a subset of this codebase (specifically, only for extracting sources from raw imaging data), all subsequent steps do not require MATLAB. You can download the most up-to-date version of CNMF_E to your desired install location with

git clone https://github.com/zhoupc/CNMF_E.git

Then, open MATLAB, change to the installation directory, and run

cnmfe_setup

Testing the code

From the top directory (in environment cosmos3) run:

pytest or pytest --pdb

If using Mac OSX, then because of an issue with matplotlib you instead need to do the following:

conda install python.app then pythonw -m pytest

Additional installation notes

Note: There is a small possibility that due to a certain quirk in loading ome.tif files in import_raw_cosmos_data.py, you may need to use python2 for that script. It is possible the library call to load the files has now been properly upgraded for python3 and this is irrelevant. If it is necessary, you can likely just call python2 within the cosmos3 environment, or start a new environment (this has not been very well tested) with

conda env create --file install_utils/cosmos2requirements.yml -n cosmos2 python=2.7

source activate cosmos2

python setup.py develop

Useful scripts (last updated 20200415)

Importing raw data from COSMOS microscope:

  1. Use import_raw_cosmos_data.py, following the directions at the top of that script. This crops the dual-lenslet ROIs, extracts timing information from the synchronization LED, uses CNMF_E to extract neural sources (requires MATLAB), and enables manual atlas alignment.

  2. Then use the interactive jupyter notebook: notebooks/processing_notebooks/trace_merge_script.ipynb to align and merge traces from the different lenslets, quality control the extract sources, and save out an .h5 file containing merged traces that will be used for all further analyses of the dataset.

  3. You can then run notebooks/processing_notebooks/quickly_assess_merged_traces.ipynb to quickly assess the sources and play with the traces. For general analysis, though, move the merged_traces.h5 file from processedData to ~/Dropbox/cosmos_data/[date]/[session_name]/, and add the relevant information about the session to cosmos.params.trace_analyze_params.py

Importing intrinsic imaging movies for atlas alignment:

  1. See notebooks/processing_notebooks/intrinsic_imaging_alignment.ipynb

  2. After extracting phase map from step 1, you can manually overlay the phase map with the the image that contains vasculature. This alignment of PM/V1 boundary can then be used to precisely align already-imported COSMOS data, using scripts/adjust_atlas.py

Importing two-photon data (i.e. for comparing visual stimulus orientation selectivity):

  1. Process two-photon data (i.e. visual stimulus). scripts/batch_cnmf_2p.py

  2. Analyzing orientation selective visual grating stimulation. notebooks/primary_notebooks/fig_visual_stimulation_two_photon.ipynb, notebooks/primary_notebooks/fig_visual_stimulation_cosmos.ipynb,

Analyzing COSMOS traces from lick-to-target task and generating figures:

DEMO analysis of processed traces.

ipynb/primary_notebooks/DEMO_trace_analyze_script.ipynb

Decode behavior from neural activity.

notebooks/primary_notebooks/classification_analysis.ipynb

Assign neural sources to task-related classes

notebooks/primary_notebooks/task_class_assignment.ipynb notebooks/primary_notebooks/fig_cluster_summary_with_mr2

Trace summary figure

notebooks/primary_notebooks/fig_trace_summary.ipynb

Optics summary

notebooks/primary_notebooks/fig_optics.ipynb

Characterization of COSMOS vs. macroscope

notebooks/primary_notebooks/fig_macroscope_comparison.ipynb

Lick decoding

notebooks/primary_notebooks/fig_classification_summary.ipynb

VGAT optogenetic inhibition

notebooks/primary_notebooks/vgat_inhibition_analysis.ipynb

Unaveraged vs. trial-averaged correlation

notebooks/primary_notebooks/fig_cluster_summary_SINGLE_TRIAL-11.13.19.ipynb

Optics simulations

matlab/scripts/trace_analysis_spont.m (for estimating background and signal photons) matlab/SNR/dof_snr_simulations.m (for plotting simulations)

Reference:

Please cite this paper when you use COSMOS in your research. Thanks!

Kauvar IV*, Machado TA*, et al. Cortical observation by synchronous multifocal optical sampling reveals widespread population encoding of actions. Neuron 2020.

License

Copyright 2020 Isaac Kauvar and Tim Machado

General notes:

To run jupyter notebooks, on a remote computer, go to top directory and run:

jupyter notebook --port=5558 --ip=* --no-browser

See jupyter documentation to learn how to set up a password so that this is secure.

cosmos-tools's People

Contributors

izkula avatar tamachado avatar

Watchers

 avatar  avatar  avatar

cosmos-tools's Issues

E ValueError: numpy.ufunc size changed, may indicate binary incompatibility. Expected 216 from C header, got 192 from PyObject

I've installed the environment successfully, but I get this error message when I run 'pytest'. How can I fix this problem?

=================================== ERRORS ====================================
_______________ ERROR collecting tests/test_clustering_utils.py _______________
tests\test_clustering_utils.py:1: in
import cosmos.traces.clustering_utils as cu
cosmos\traces\clustering_utils.py:1: in
import cosmos.traces.trace_analysis_utils as utils
cosmos\traces\trace_analysis_utils.py:29: in
import seaborn as sns
..\anaconda3\envs\cosmos3\lib\site-packages\seaborn_init_.py:9: in
from .regression import *
..\anaconda3\envs\cosmos3\lib\site-packages\seaborn\regression.py:27: in
from .axisgrid import FacetGrid, _facet_docs
..\anaconda3\envs\cosmos3\lib\site-packages\seaborn\axisgrid.py:16: in
from .distributions import distplot, kdeplot, freedman_diaconis_bins
..\anaconda3\envs\cosmos3\lib\site-packages\seaborn\distributions.py:13: in
import statsmodels.nonparametric.api as smnp
..\anaconda3\envs\cosmos3\lib\site-packages\statsmodels\nonparametric\api.py:7: in
from .kde import KDEUnivariate
..\anaconda3\envs\cosmos3\lib\site-packages\statsmodels\nonparametric\kde.py:23: in
from .linbin import fast_linbin
statsmodels\nonparametric\linbin.pyx:1: in init statsmodels.nonparametric.linbin
???
E ValueError: numpy.ufunc size changed, may indicate binary incompatibility. Expected 216 from C header, got 192 from PyObject
________________ ERROR collecting tests/test_cosmos_dataset.py ________________
tests\test_cosmos_dataset.py:3: in
from cosmos.imaging.cosmos_dataset import CosmosDataset
cosmos\imaging\cosmos_dataset.py:21: in
import cosmos.traces.trace_analysis_utils as trace_utils
cosmos\traces\trace_analysis_utils.py:29: in
import seaborn as sns
..\anaconda3\envs\cosmos3\lib\site-packages\seaborn_init
.py:9: in
from .regression import *
..\anaconda3\envs\cosmos3\lib\site-packages\seaborn\regression.py:27: in
from .axisgrid import FacetGrid, _facet_docs
..\anaconda3\envs\cosmos3\lib\site-packages\seaborn\axisgrid.py:16: in
from .distributions import distplot, kdeplot, freedman_diaconis_bins
..\anaconda3\envs\cosmos3\lib\site-packages\seaborn\distributions.py:13: in
import statsmodels.nonparametric.api as smnp
..\anaconda3\envs\cosmos3\lib\site-packages\statsmodels\nonparametric\api.py:7: in
from .kde import KDEUnivariate
..\anaconda3\envs\cosmos3\lib\site-packages\statsmodels\nonparametric\kde.py:23: in
from .linbin import fast_linbin
statsmodels\nonparametric\linbin.pyx:1: in init statsmodels.nonparametric.linbin
???
E ValueError: numpy.ufunc size changed, may indicate binary incompatibility. Expected 216 from C header, got 192 from PyObject
------------------------------- Captured stderr -------------------------------
c:\users\rhkd9\anaconda3\envs\cosmos3\lib\site-packages\h5py_init
.py:34: FutureWarning: Conversion of the second argument of issubdtype from float to np.floating is deprecated. In future, it will be treated as np.float64 == np.dtype(float).type.
from ._conv import register_converters as _register_converters
________________ ERROR collecting tests/test_decoding_utils.py ________________
tests\test_decoding_utils.py:1: in
import cosmos.traces.decoding_utils as utils
cosmos\traces\decoding_utils.py:25: in
from cosmos.traces.decoding.decoders import DenseNNDecoder, LSTMDecoder
cosmos\traces\decoding\decoders.py:8: in
import statsmodels.api as sm
..\anaconda3\envs\cosmos3\lib\site-packages\statsmodels\api.py:11: in
from .regression.recursive_ls import RecursiveLS
..\anaconda3\envs\cosmos3\lib\site-packages\statsmodels\regression\recursive_ls.py:14: in
from statsmodels.tsa.statespace.mlemodel import (
..\anaconda3\envs\cosmos3\lib\site-packages\statsmodels\tsa\statespace\mlemodel.py:32: in
from .simulation_smoother import SimulationSmoother
..\anaconda3\envs\cosmos3\lib\site-packages\statsmodels\tsa\statespace\simulation_smoother.py:9: in
from .kalman_smoother import KalmanSmoother
..\anaconda3\envs\cosmos3\lib\site-packages\statsmodels\tsa\statespace\kalman_smoother.py:11: in
from statsmodels.tsa.statespace.representation import OptionWrapper
..\anaconda3\envs\cosmos3\lib\site-packages\statsmodels\tsa\statespace\representation.py:9: in
from .tools import (
..\anaconda3\envs\cosmos3\lib\site-packages\statsmodels\tsa\statespace\tools.py:14: in
from . import (_initialization, _representation, _kalman_filter,
statsmodels\tsa\statespace_initialization.pyx:1: in init statsmodels.tsa.statespace.initialization
???
E ValueError: numpy.ufunc size changed, may indicate binary incompatibility. Expected 216 from C header, got 192 from PyObject
------------------------------- Captured stderr -------------------------------
Using TensorFlow backend.
_____________ ERROR collecting tests/test_trace_analysis_utils.py _____________
tests\test_trace_analysis_utils.py:2: in
import cosmos.traces.trace_analysis_utils as utils
cosmos\traces\trace_analysis_utils.py:29: in
import seaborn as sns
..\anaconda3\envs\cosmos3\lib\site-packages\seaborn_init
.py:9: in
from .regression import *
..\anaconda3\envs\cosmos3\lib\site-packages\seaborn\regression.py:27: in
from .axisgrid import FacetGrid, _facet_docs
..\anaconda3\envs\cosmos3\lib\site-packages\seaborn\axisgrid.py:16: in
from .distributions import distplot, kdeplot, _freedman_diaconis_bins
..\anaconda3\envs\cosmos3\lib\site-packages\seaborn\distributions.py:13: in
import statsmodels.nonparametric.api as smnp
..\anaconda3\envs\cosmos3\lib\site-packages\statsmodels\nonparametric\api.py:7: in
from .kde import KDEUnivariate
..\anaconda3\envs\cosmos3\lib\site-packages\statsmodels\nonparametric\kde.py:23: in
from .linbin import fast_linbin
statsmodels\nonparametric\linbin.pyx:1: in init statsmodels.nonparametric.linbin
???
E ValueError: numpy.ufunc size changed, may indicate binary incompatibility. Expected 216 from C header, got 192 from PyObject
!!!!!!!!!!!!!!!!!!! Interrupted: 4 errors during collection !!!!!!!!!!!!!!!!!!!
========================== 4 error in 20.29 seconds ===========================

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.