Git Product home page Git Product logo

telluride_decoding's Introduction

The telluride_decoding Library

(This is not an official Google product!)

This repository contains Python/Tensorflow code to decode perceptual signals from brain data. The perceptual signals we are using are generally audio features. And the brain data is one of several types of signals, such as EEG, MEG and ECoG.

The Telluride Decoding Toolbox is geared towards large-scale decoding experiments. You can run small experiments in a colab. But this package keeps all the data in files so the data doesn't have to fit into memory, and so it is easier to run tens or hundreds of tests in parallel.

License

This code uses the Apache License 2.0. See the LICENSE file for details.

Purpose

This code builds and trains models that connect perceptual signals, primarily audio, to brain signals, which can be EEG, MEG or ECOG. One common use for this type of algorithm is to decode auditory attention, as shown in the figure below.

Auditory attention decoding

In attention decoding, we wish to know which of two (or more) signals a user is attending. One signal (the upward pointing blue arrow) indicates that the signal is being processed by the entire brain and is "exciting" all areas. Another signal is heard by the auditory system, but is not attended, and doesn't recruit as much of the brain. The two signals are processed differently, and produce different brain signals.

This software helps to decide which signal the user is attending, by building a model that uses the EEG signal, for example, to predict the intensity of the attended audio signal.

Documentation

Documentation to follow (but there are extensive comments in the code.) The three primary parts of this code are:

  • ingest: Use to read various kinds of file formats and transform the data into TFRecords for use by the decoding program
  • decoding: Used to build, train and test models that connect audio and brain signals.
  • infer: Use to run a model on real or test data. This model can run in real time.

To install

To install this software you can execute this command, which will automatically install the necessary prerequisites:

pip install telluride-decoding

Using this code

This library is written in Python3 and uses Tensorflow2. The decoding code can be run as a standalone program, or as a library, or in a Colab, a form of Python wrapped in an interactive notebook.

There is a complete end-to-end codelab that describes how to use the programs to perform attention decoding on a public dataset. With a Unix machine and access to to GitHub you can verify that the code works and produces the same results as we get at Google. The codelab is here doc/DecodingCodelab.md

One can also use the program as a library in a Colab. Colab allows one to interactively ingest data, build models, and test their performance. Here is an example:

  • Telluride4: Ingest and test linear regression. colab

Documentation on the standalone decoding program can be found here. This version is useful in distributed environments, where one might want to do a parameter search. Each run of the program tests one set of parameters.

The code uses CPU or GPUs, as available. TPU support is forthcoming.

References

James O'Sullivan, AJ Power, Nima Mesgarani, S. Rajaram, John Foxe, Barbara Shinn-Cunningham, Malcolm Slaney, Shihab Shamma, Edward Lalor. Attentional Selection in a Cocktail Party Environment Can Be Decoded from Single-Trial EEG. Cereb Cortex. 2015 Jul;25(7):1697-706.

Daniel D.E. Wong, Søren A. Fuglsang, Jens Hjortkjær, Enea Ceolini, Malcolm Slaney, Alain de Cheveigné. A Comparison of Temporal Response Function Estimation Methods for Auditory Attention Decoding. Frontiers in Neuroscience. doi: 10.3389/fnins.2018.00531.

This package contains Python and Tensorflow code for the Deep Canonical Correlation Analysis algorithm. (An earlier MATLAB implementation can be found at https://ttic.uchicago.edu/~wwang5/dccae.html)

Wang, Weiran and Arora, Raman and Livescu, Karen and Bilmes, Jeff. On deep multi-view representation learning, International Conference on Machine Learning, pp. 1083-1092, 2015.

Andrew, Galen and Arora, Raman and Bilmes, Jeff and Livescu, Karen. Deep canonical correlation analysis. International conference on machine learning, pp. 1247-1255, 2013. https://ttic.uchicago.edu/~klivescu/papers/andrew_icml2013.pdf

telluride_decoding's People

Contributors

malcolmslaney avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar

telluride_decoding's Issues

is there any script missing in this box?

Hi, while i was reading this toolbox guideline, i found a picture about single trial's decoding accuracy(EEG data to data.wav ),which is what i want to try in my research, but seems no such a script can caculate this and draw this picture
image

also, I found that even though there are 32 trials ,but the result only shows one reconstruction accuracy, i am kind of confused.

looking forward to your reply!

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.