A project intended to generate music fitted to a person's emotional state as determined through real-time EEG recordings.

Python 2.13% HTML 0.07% JavaScript 0.03% CSS 0.01% MATLAB 0.39% Jupyter Notebook 97.37%

brain-computer-interface electroencephalography emotiv-epoc classification music

nrgyorku-brain-computer-music-interface's Introduction

NRGYorkU Brain Computer Music Interface

Goal:

To develop a Brain-Computer Music Interface (BCMI) to generate music based on a person's emotions, as determined by real-time EEG classification using an Emotiv kit.

Potential Steps:

There are 3 main parts of this project that can be developed in parallel. There may exist a number of technical difficulties involved in connecting multiple real-time data feeds to each other.

EEG data feed: We need to develop a pipeline for acquiring a real-time EEG data feed that can be attached to a classifier. This may be the most difficult step conceptually, as there are few good libraries available for acquiring this type of data from the Emotiv EPOC.
Emotion Classification: We need to extract robust EEG features and we need to train a classifier or regressor to determine the emotional content from EEG data. We will use publicly available EEG datasets for this task, and might want to incorporate some elements of semi-supervised learning (given the abundance of EEG data without labels).
Music Generation: We need to generate music based on the identified emotions. This can be done using Erlich's algorithm, which has been translated to Python at this point in time. As a brief note, it might potentially be better to do away with Erlich's algorithm entirely in the future and to use a generative adversarial music generation algorithm instead; however, this will be a technically difficult challenge.

Requirements:

Python 3.x. Anaconda installation highly recommended. Ability to run Jupyter Notebooks highly recommended.
All libraries listed in requirements.txt. Run pip install -r requirements.txt from command line to install dependencies.
VirtualMIDISynth. You can download this software from here.
Sound fonts for VirtualMIDISynth. I used these soundfonts.

Citations

Stefan Erlich's music generation algorithm.
We intend on using the DEAP dataset for classifier training in the future.
Ehrlich, S. K., Agres, K. R., Guan, C., & Cheng, G. (2019). A closed-loop, music-based brain-computer interface for emotion mediation. PloS one, 14(3), e0213516. URL/DOI: https://doi.org/10.1371/journal.pone.0213516

nrgyorku-brain-computer-music-interface's People

Contributors

Stargazers

Watchers

nrgyorku-brain-computer-music-interface's Issues

Obtain access from DEAP for their dataset

Investigate GANs

We need to swap out the Erlich algorithm at some point.

Determine how to control the valence and arousal in the Erlich code

Try wrapping Erlich's code and/or EEGlab code in Python

Incorporate important project information in the README.md

Anything relevant can go here, really.

Figure out how to do preprocessing of raw EEG data (if necessary)

Create MIDI output from Generative Algorithm

Try to get the Audio Toolbox working in composer_algorithm in MATLAB, and send it to a MIDI source output to see whether or not we can execute MIDI commands independent of EEG input.

Find Workflow Images for Presentation

Create background slide with an image explaining BCMI/music theory (Harley or Nicia)
Expand on the circumplex model (Nicia)
Explain process of the overall system (Harley)
EEG Signal Processing - how will Emotiv stream to the classifier in the testing phase, and how will DEAP stream to classifier in the training phase? (Harley)
Feature selection - find Python image to graphically represent features, and expand
Classifiers - use schematic to represent Random Forest, and KNN and explain how it may classify valence/arousal. Elaborate on Ramzan and Dawn's model (Harley or Matthew)
Generative Algorithm - explain parameters, how generative algorithm works, and how MIDI signals will be sent to a DAW (Danny)
Future direction - explain neurofeedback component and formalizing it methodologically, and additional applications for the BCMI (e.g. entertainment, music composition, coma interventions) (Danny)

Attempt to train a classifier using the data from the nadzeri repository (Python or MATLAB)

https://github.com/nadzeri/Realtime-EEG-Based-Emotion-Recognition

Create a pipeline for extracting features from EEG (can use previous pipelines as reference)

Implement FFT descriptive features

For each electrode, take the fast Fourier transform and separate it into frequency bands. Then, for each frequency band, find the Mean/Median/Variance/SD/Min_value/Max_value/Range/Skewness/whatever from the FFT coefficients and concatenate this with the existing final_features array (note that final_features should have the shape of (number of training samples, number of features)).

Realtime framework

Need to combine real-time EEG data feed with preprocessing, classification/regression, and then feed model outputs to generative model. Might require multiple threads and/or asyncio.

Program real-time EEG data into classifier

Find a way to capture a robust resampling rate (e.g last 10 sec) for real-time EEG data to be continuously sent to the classifier. If possible, stream this through a preprocessing pipeline either on Python (preferably) or EEGLab intermediary to being sent to the classifier.

matthewcso / nrgyorku-brain-computer-music-interface Goto Github PK