open-ephys / analysis-tools Goto Github PK
View Code? Open in Web Editor NEWArchived code for reading data saved by the Open Ephys GUI
Archived code for reading data saved by the Open Ephys GUI
My example file is here: https://iu.box.com/s/68nel6xs7g73z6i732dgp4guintjsnyr
The faster function skips chunks of data over 10001 samples or more long
[~, ts]=load_open_ephys_data_faster('100_CH19.continuous');
db_fast=diff(ts);
db_fast(db_fast>100)
returns three instances:
10001
10001
111245
They are located somewhere in the middle:
find(db_fast>100)
ans =
4659200
8309760
50656256
length(db_fast)
ans =
266704895
Wheres load_open_ephys_data.m does not:
[~, ts]=load_open_ephys_data('100_CH19.continuous');db=diff(ts);find(db>100)
ans =
Empty matrix: 0-by-1
Hello everyone,
We are using a 64 channel probe for recording and the data is stored in .continuous format. Each channel has a separate file. For spike sorting, I would like to convert the .continuous files to flat binary .dat for use in Klusta. Is there any script for such a purpose?
I would appreciate if you can give me instruction on how to proceed.
Is there a trivial way to convert between flat binary to the open-ephys formats? I see there is a tool to convert between open-ephys and kwik. Has this been generalized?
Thanks!
There's a discontinuity in the length of the array of timestamps that are outputted by load_open_ephys_faster and OpenEphys.py run independently on the same analog continuous channel. For whatever reason it seems that OpenEphys.py is only keeping every 512th entry in the timestamp matrix given by load_open_ephys_faster run in matlab.
Hello,
I saved data in flat binary format, but found out that the number of samples is not correct. Each my recording is 2 second long sampled at 20kHz, so 40000 samples should be in the continuous.dat file for each channel in each recording. However, when I load the data with the function in Binary.py and checked the data dimension, it turned out to be either more or less than 40000 samples for each recording.
Is there a way to fix this? How do I align the data from each recording in this case?
Appreciate it for any help,
Yu
I recently put a Julia package for loading Open Ephys .continuous data files on github. I want to keep it as a separate repository so that Julia's package manager can use it, but it might be useful here as well.
Hello,
I’m trying to import data from open ephys to neuroexplorer 4.
I first import data in matlab using load_open_ephys_data.m or load_open_ephys_data_faster.m.
Then, to bypass the well known import issues, I import data in neuroexplore by means of nex script GetMatlabVectorAsContVar.nsc for continuous variables and GetMatlabTimestampVector.nsc for events variables.
Everything works fine for .continuous data, but I have problems when trying to import .events.
In particular, I can import in Matlab only 1 column of data. In this column each number represents the number of channel-line changing phase in time (e.g. if I switch on dig 2, then dig 3 and then off 3 and 2 I’ll read 2332). However I cannot import timestamps and the flag (1 or 0, on/off).
Note that importing data with the demo version of neuroexplorer 5 and sending to Matlab, I have as a result a matrix of 5 columns: 1 timestmps; 2 event type; 3 processor ID; 4 event ID (i.e. 1 or 0 of the dig. Channel); 5 event channel, thus I believe all data are present in the .event file.
Do you think it is a problem in our Open-ephys configuration?
Did someone else had the same problem? Any solution or suggestion?
Thanks in advance for help
I noticed that load_open_ephys_data_faster.m currently outputs continuous data with lower resolution than load_open_ephys_data.m, which I don't think is intentional. Specifically, one change made in pull request #26 (line 150 in the current file) causes the data values read in from the continuous file to be stored in an int16 matlab array (which is fine so far, since they are integers). Then, when they are scaled by info.header.bitVolts in line 120, the result is automatically rounded to another int16 array. In contrast, load_open_ephys_data.m reads the data in as doubles, so they are not rounded after multiplying. In my case, the bitVolts value was about 0.2, so reading data with this function decreased the resolution by about 5 times.
Is there a good reason for writing to an int16 array in the fread function? Otherwise, that could just be reverted.
-Ethan
index
is undefined on line 163
Hi all,
I have two separate recording sessions corresponding to two different behavioral tasks, and I am interested in following a unit's response across the two tasks.
I think the best way to do this would be to concatenate the .spikes files from the two sessions together so that I can spike sort at once (*I use MATLAB and MClust).
Is there an easy way to do this?
I am new to openephys and am not very familiar with the structure of the files. I apologize in advance if this is a very basic question.
Thank you!
Hi all,
I recently started to use a new openephys acquisition board, and I usually acquire raw data and filtered data for LFP and unit analysis.
I recently noticed that the size of filter data is smaller than the raw data, and sometimes I got a few channels that have intact recordings, but sometimes none of the channels has intact recording, though I did not change the configuration file.
Does anyone have any idea what is wrong in my recording?
Many thanks!
Yu
I want to get the data from specific channels in binary data like the way that open_ephys format lists the channels. Can anyone help me with that ?
Hello,
I'm using "convert_open_ephys_to_kwik.m" to read 32-channel recording files in the OpenEphys GUI, however, it looks like the script only includes the 32 channels and not the ADC and AUX channels, which I also need to visualize. Is there a way to do this?
Thanks,
Jack
There is a difference between the way timestamps are outputted between load_open_ephys_data_faster.m and Openephys.py. Namely, the former outputs a set of timestamps equivalent to the length of the data in the channel. My reading of the code suggests this is not stored within the .continuous file itself, but is generated in the matlab script. Is there anyway to duplicate the same output in OpenEphys.py in future versions?
I’m new user to Open Ephys. When I try to compile the OE data to plexon, the ephys2plx script fails at ephys2plx_mex.c
Error using mex
I have tried the script in two versions of matlab (2013a and 2015a) and get the same failure.
Any suggestions as to why the complier is failing. Im using windows 7. Data format is in Openephys format.
line 190
info.header = header;
because header is a matlab native function it doesn't anymore.
I replaced the function getHeader with
function info = getHeader(hdr)
header='';
eval(char(hdr'));
info.header = header;
end
When I try to use loadEvents from OpenEphys.py to import event files into python I get the following error:
File "C:\Anaconda\lib\site-packages\OpenEphys\OpenEphys.py", line 241, in loadEvents
nodeId[index] = np.fromfile(f, np.dtype('<ui'), 1)
TypeError: data type "<ui" not understood
Changing line 241 to:
nodeId[index] = np.fromfile(f, np.dtype('<u1'), 1)
resolves the problem.
Hello,
I'm trying to use the analysis tools to load my OpenEphys data in Jupyter notebook, but am not having any success. Has anyone done this and could guide me through?
Ben
Hi there,
It seems that the recorded sampling rate in *.spikes files (30kHz) is different from set sampling rate during acquisition with the GUI (20kHz).
Does it mean that the spikes files sampling rate is by default 30 kHz or is it just a bug ?
Thx in advance for your help.
Karin
I was analyzing my continuous LFP data and trying to line-up manual timestamps that I had placed during recording (via comments entered in the GUI and pressing 'save',) when I noticed that clear events on the recording were not lining up with the timestamp. For a number of my recordings, I had hit the play button prior to hitting the record button in the GUI to make sure that I had a good signal before recording. I was wondering whether the timestamp on the messages.events file related to the sample number on the actual recording or the overall sample number (including non-recorded time). I did a test where I did a number of timestamps, paused the recording, and did additional timestamps. From that data, it looks like the timestamp includes non-recorded time (which means that the int64 stamp doesn't actually correspond to sample number in the recording.) I have a lot of data that hinges on accurate timestamps and is already recorded- is there any way to determine how these timestamps line up? It looks like the load_open_ephys.m reads the int64 timestamp in the .continuous files, which is what I believe is in the messages.events file, but I'm not clear how to correlate everything. Any help appreciated.
Hi!
I'd like to make some changes to this repo, but I just wanted to get some feedback first.
I would like to reorganize the python code into a python module. This allows people to install the tools as a python package. The downside is that is adds a bit of complexity to the structure. Also, I'm not sure the name "analysis-tools" is going to work as a python module. The pep8 has some guidance on how to name a python package here, I propose "openephys", or maybe "pyopenephys" (or "pyoephys"?) to distinguish it from the matlab code
The new structure of the repo would then be:
analysis-tools/
setup.py <- for installing the package
[py]openephys/
kwik.py
(etc).py
matlab/
*.m
Would love to hear what you think, possibly we could create a new branch for these changes?
Hi everyone;
newbee here.
i have some problems with load_open_ephys_data function.
I started in this days to use Open Ephys GUI, trying to record some samples and convert in Matfile to test the system. I'have dowloaded the analysis_tools and used the function, but some syntax problems are aroused from the fist time.
Example:
"Error: File: load_open_ephys_data.m Line: 140 Column: 6
Illegal use of reserved keyword "elseif".
line 140- elseif strcmp(filetype, 'continuous')
141
142- disp(['Loading ' filename '...']);
line 140: Parse error at ELSEIF: usage might be invalid in Matlab syntax
line 140: Parse error at ')': usage might be invalid in Matlab syntax"
load_open_ephys_data_faster does not give this issue.
Can you give me some help or suggestion to resolve this problems?
Thanks!
Matlab version: 2016a
When I import the OpenEphys module I can load the .continous data without issues.
But when I try to load the spikes it gives me the following error:
data = OpenEphys.load(r"T:\Dokumente\Data\test_2019-01-08_13-57-09\STp105.0n5.spikes")
loading spikes...
Traceback (most recent call last):
File "", line 1, in
data = OpenEphys.load(r"T:\Dokumente\Data\test_2019-01-08_13-57-09\STp105.0n5.spikes")
File "c:\anaconda3\lib\OpenEphys.py", line 37, in load
data = loadSpikes(filepath)
File "c:\anaconda3\lib\OpenEphys.py", line 208, in loadSpikes
waveforms = np.fromfile(f, np.dtype('<u2'), numChannels*numSamples)
TypeError: only integer scalar arrays can be converted to a scalar index
Does anybody know what I am doing wrong here? Any help would be appreciated.
My recording usually lasts for three days, but I want to pull out the data either from LFP, or Spikes for a specific hour (hour before and after the injection time). Can anyone help me with the code ? I also use MATLAB
When trying to load a continuous file (that I can open using other loaders) I get the following error:
In [8]: data = OpenEphys.load(path)
Loading continuous data...
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-8-79b386c02a12> in <module>()
----> 1 data = OpenEphys.load(path)
/home/galen/Documents/Code/analysis-tools/Python3/OpenEphys.py in load(filepath)
37 # redirects to code for individual file types
38 if 'continuous' in filepath:
---> 39 data = loadContinuous(filepath)
40 elif 'spikes' in filepath:
41 data = loadSpikes(filepath)
/home/galen/Documents/Code/analysis-tools/Python3/OpenEphys.py in loadContinuous(filepath, dtype)
159 data = np.fromfile(f,np.dtype('>i2'),N) # big-endian 16-bit signed integer
160 try:
--> 161 samples[indices[recordNumber]:indices[recordNumber+1]] = data
162 except ValueError:
163 raise
ValueError: could not broadcast input array from shape (1024) into shape (256)
Normally this error is not raised due to index
not being defined in the except block (#43), but once I remove the print statement this is the error that is raised.
I'm a new user of OpenEphy (and Matlab...), when I try to compile openephy data to plexon data it appear error in Matlab2014b:
ephys2plx
Error using getFileHeader (line 6)
Error: Unexpected MATLAB expression.
Error in ephysConv (line 6)
FileHeaders{1,1}=getFileHeader([pathName ephysInfo.events_file]);
Error in ephys2plx (line 15)
ephysConv; %get .xml file, and read information
Any idea what should I do ? my laptop is MAC OS X Yosemite, Matlab2014b for mac and Xcode 6.3.1 installed. I follow the steps in https://github.com/open-ephys/analysis-tools/tree/master/ephys2plx and execute mex ephys2plx_mex.c says successful installed complier. Data format is openEphy format stored in one fold with 32 channel continuous data , 8 Tetrode spike data and 8 ADC channel event.
The loadContinuous()
function in OpenEphys.py
produces several VisibleDeprecationWarning
s, for two reasons.
The first (lines 128, 129, 130) is because the constants defined around line 30 are floats instead of integers. This can be fixed by wrapping them in int()
.
The second (line 157) is because N
is an array ([1024]
) instead of an integer (1024
), because np.fromfile()
returns an array even when asked for a single value (line 147).
I'll submit a pull request with what I assume are the correct fixes.
Hi,
I recently had a problem with one of my recordins. At around 1/3 of the recording the mice chewed the cable returning basically noise and increasing the spike file size a lot. I intended to go through the data and find the initial point of disconnection but realized that loading file in matlab for big files is a problem: my .spikes files were ~800Mb and the load_open_ephys_data.m got stuck at around 40%.
I think it would be useful for en users to have a loader that would allow for:
I don't feel confident doing it in a clean way so i am suggesting it as an improvement of the analysis tool for matlab/python.It could be great to have that in the near future.
i have loaded the .continuous files in to work space but now i want see the signal which i stored is correct or not. how can i open them and see.
https://github.com/open-ephys/analysis-tools/blob/master/Python3/OpenEphys.py#L222
Should be wv = np.reshape(waveforms, (numChannels, numSamples)).T
What is the license of this repository?
Hi, opening event file from my last recording is returning an error
Undefined function or variable 'header'
I am not sure what the two lines
hdr = fread(fid, NUM_HEADER_BYTES, 'char*1');
eval(char(hdr'));
do in load_open_ephys_data (lines 83,84) but they appear to not be working this time. Previous recordings are ok. Last recording was done in Linux, OE compiled from the latest code on github. Events were generated using TTL. Event file attached here https://drive.google.com/file/d/0ByTsMM7VWmWKUG5nbVRCMTVDOWM/view?usp=sharing Thanks.
i have an issue running TT openephys data recorded in binary to Mclust.
i can open data using load_open_ephys_binary (both spikes and continuous) but then when i want to cut them in Mclust using LoadTT_openephys then it doesnt work.
any one got similar issue? any idea how to do it
thanks
When using load_open_ephys_data, we have an issue of timestamps missing and being out of order. They can skip ahead, which results in losing some samples or they can skip backwards, which makes them misaligned. This happens frequently throughout recording. I'm using the latest open ephys version compiled on linux computer.
The m file and spike, continuous and event file is in same dir. ( same error with continuous and event file)
I already tried with this commands:
load_open_ephys_data("C:\Open Ephys\2018-06-01_19-23-51\100_CH31.continuous");
anything else I need to do? I copied (https://github.com/open-ephys/analysis-tools/blob/master/load_open_ephys_data.m) this in new script file and named load_open_ephys_data.m
Hi,
I noticed a bug in the messages.events timestamps.
During my experiment, I send a TCP message containing a timestamp to open ephys. Once I opened my data I realised that the open ephys timestamps do not all correspond to the sender program timestamps.
I hope this plot makes sense.
Is there anyone who could solve the problem ?
Thx
Karin
These tools are simple and therefore are easy targets for an automated test suite. I think it would be great to have this...
Hi,
I am trying to align the continuous data with the spike data. I have several questions.
Thx for your help
Karin
When converting time stamps from loadSpikes in OpenEphys.py by dividing by the sample rate I get spike times that are larger than the recording duration e.g.
Spike time 1011.3367666666667 s exceeds duration 983.1765333333333 s
Am I converting time stamps wrongly?
Hello,
Im interested in the spike sorting feature of Open Ephys for upcoming neuro phys experiments -
Currently, Ive been able to setup a practice signal chain of the following:
File read (cortex file) > Bandpass (300-6000hz) > Spike Sorter to get the hang of using the OpenEphys software.
Im able to successfully set some arbitrary thresholds and record the signal - to then load the '.spikes' file of a particular channel into matlab.
Problem: Im attempting to visualize/analyze/plot the spike data and have no idea how to proceed. Upon loading the data, a 30x40x2 array is created. Im unsure how to perceive the contents of this array, what the values correspond to, and how to plot them in a manner that allows me to see the spikes as well as any other data that is usually acquired via the spike sorter.
Any information/elaboration/tutorial would be tremendous as I am a beginner attempting to learn this software.
Thank you very much.
Is there a way to load only parts of the recording? For example if I record over 3h and and I want to analyse a 2-3 min fraction in the middle of the recording. At the moment I would have to load the full 3h which is time consuming with 32 channels.
I am using OpenEphys.py
Does anyone know of a tool to convert .continuous files to the SpikeGLX binary format? Alternatively, if there isn't one, is there a good spec for the .continuous file so that I might build my own converter?
Hi.
I'm using open-ephys GUI and neuropixels.
I recorded 50 min of LFP and SPK.
Thanks for your help in advance.
Are the time-stamps in seconds, maybe I am wrong but that doesn't seems to be the case for me.
Please could you double check.
Thank you.
settings.xml doesn't always (at least when I'm running it) have the processor name as the second attribute (line 38 in the code). Not sure if this is an issue with the code writing the settings file or the way Matlab reads it. I wrote a quick loop to step through the attributes to identify the "name" and tried to submit it for revision but I'm new to working with this kind of collaborative system and am not sure I did it right. I changed lines 37 & 38:
nodeId = str2num(char(xProcessor.getAttributes.item(0).getValue));
processorName = char(xProcessor.getAttributes.item(2).getValue);
into the following loop:
for procAtt = 0:xProcessor.getAttributes.getLength-1
if strcmp(xProcessor.getAttributes.item(procAtt).getNodeName, 'NodeId')
nodeId = str2num(char(xProcessor.getAttributes.item(procAtt).getValue));
elseif strcmp(xProcessor.getAttributes.item(procAtt).getNodeName, 'name')
processorName = char(xProcessor.getAttributes.item(procAtt).getValue);
end
end
With the previous code the later code using strcmp to check for the 'Filters/Spike Detector' node (line 44 I believe):
if strcmp(processorName, 'Filters/Spike Detector')
was not working because it wasn't correctly pulling the "name" attribute. Additionally, the output from the file was not correctly including the processor names as strings in the second column of the output "info."
Edit: code blocks weren't being formatted properly.
Hello
I've noticed that load_open_ephys_data_faster.m requires a lot more memory to load .continuous files in Matlab 2020a. Specifically, I'm getting a memory error on line 84:
hdr = fread(fid, NUM_HEADER_BYTES, 'char*1');
when I have about 4 GB of memory left for a .continuous file that takes up only about 1 GB of space when it is loaded (and obviously the header is much smaller than the full 1 GB for all the data). In Matlab 2016, this was not a problem.
I found this post on the mathworks site that indicates something changed with fread in Matlab2020a that might explain this behavior:
https://www.mathworks.com/matlabcentral/answers/84385-out-of-memory-error-while-reading-binary-file
I'd appreciate it if any necessary updates could be made to load_open_ephys_data_faster.m so it will work properly with Matlab2020a, or if someone could provide suggestions on how to modify the code. I don't know much about the structure of the .continuous files, so I don't feel confident making this change myself. Thank you for all your help!
~Nick
Hello!
I'm trying to load open-ephys files in python3. After 2to3 script conversion, when I try to load a .continuous file, I got this error:
In [1]: import OpenEphys
In [2]: data = OpenEphys.load('100_CH1.continuous')
Loading continuous data...
Traceback (most recent call last):
File "<ipython-input-2-e99a956d2fe5>", line 1, in <module>
data = OpenEphys.load('100_CH1.continuous')
File "/home/malfatti/Documents/SciScripts/Python3/Deps/OpenEphys/OpenEphys.py", line 39, in load
data = loadContinuous(filepath)
File "/home/malfatti/Documents/SciScripts/Python3/Deps/OpenEphys/OpenEphys.py", line 136, in loadContinuous
header = readHeader(f)
File "/home/malfatti/Documents/SciScripts/Python3/Deps/OpenEphys/OpenEphys.py", line 294, in readHeader
h = f.read(1024).replace('\n','').replace('header.','')
TypeError: 'str' does not support the buffer interface
In line 134, f is defined as byte coding:
f = open(filepath,'rb')
so I'm gessing that something in the readHeader function is not byte-encoded. I tried to use .encode() method in some strings (in line 294) but it didn't work.
Could you help me?
Cheers,
T Malfatti
Hi,
I have the following error in Matlab when I use the file: convert_open_ephys_to_kwik.m.
Error using h5create>create_dataset (line 154)
The dataset '/kwik_version' already exists.
Error in h5create (line 64)
create_dataset(options)
Error in convert_open_ephys_to_kwik (line 128)
h5create(kwdfile, '/kwik_version', [1 1],'Datatype', 'int16');
To implement, I have copied and renamed the file load_open_ephys_data_faster.m as load_open_ephys_data.m.
I think that the problem is because I have several repetitions (blocks) in the same experiment. That is, in a given .continuous file I have three recording sections. So, I have three experiments joint in one.
Do you know how I can fix this problem?
Cheers,
nc
I acquired external analog signal with open-ephys format and load the continuous data with the "load_open_ephys_data.mat". I noticed that the timestamps extracted for the analog have lots of jitters.
For example, I acquired the analog with 30K, and loaded the continuous data, and directly ploted "timestamps" generated by the code. The graph is not a perfect line. Is it the problem of my data acquisition or the problem of the code?
When I try to create a .dat file from a folder of data saved in Open Ephys format using the pack function of OpenEphys.py I get the following error:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Anaconda\lib\site-packages\OpenEphys\OpenEphys.py", line 316, in pack
ch = data[''.join((source,'_CH',str(order[j]).replace(source+'_CH','')))]['data']
KeyError: '100_CH100_AUX2'
Changing line 316 from:
ch = data[''.join((source,'_CH',str(order[j]).replace(source+'_CH','')))]['data']
to:
ch = data[order[j]]['data']
prevents this error, and the program then outputs data files that appear to be correct.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.