Saving/Loading combined recording/sorting data about spikeinterface HOT 11 CLOSED

Currently I am trying to write my own script that will map my recording and sorting to a Neo object, and then export it to a Neo-Nix object using the NeoIO. I havent had much time to work on this though. When I am done, I could try to add this to SpikeInterface if you want. I am trying to generalize it as much as a I can so it is not only specific to my MEA design.

One thing that might be nice with the SpikeInterface documentation is for the Compatible Technology to have a table that shows the current capabilities of each extractor (whether it can read and/or write, and what data it can extract/write). It might also be nice to have an infographic overview of the SpikeInterface data structure, to make it easier for mapping the data?

from spikeinterface.

Hi All.
Using NixIO format can be easy.
I can do it.

from spikeinterface.

Ooh, I think I understand your question (I deleted my previous response). I misunderstood the first time. It seems like you want to export recordings and sorting to the same file. As you mentioned, it doesn’t appear that any exporter will do this although in principal, NWB or NIX should be capable of storing it in one file. I should maybe bring that up with those developers.

I will say that the toolkit.export_to_phy saves both a recording and sorting into the phy format although it is a bit heavy to run.

from spikeinterface.

Thanks! I can work with just the sorted data for now, but it would be a nice addition to NWB/NIX, and the Neo extractor as well when that come out.

I have tried to export_to_phy previously to try manual sorting, and unfortunately it crashes with a memory error (at least on Windows).

from spikeinterface.

As it is now, export_to_phy tries to extract all the spikes from the recording so you can do manual curation later. This will unfortunately not scale to large datasets. I am not sure exactly how to deal with that. @alejoe91 any ideas or plans for this? I am not sure where we left it.

from spikeinterface.

As it is now, export_to_phy tries to extract all the spikes from the recording so you can do manual curation later. This will unfortunately not scale to large datasets. I am not sure exactly how to deal with that. @alejoe91 any ideas or plans for this? I am not sure where we left it.

@colehurwitz unfortunately phy needs to extract all spikes to compute features from all of them. Otherwise the manual curation would be biased as some spikes will not show...I don't think there is a way around that

from spikeinterface.

Currently I am trying to write my own script that will map my recording and sorting to a Neo object, and then export it to a Neo-Nix object using the NeoIO. I havent had much time to work on this though. When I am done, I could try to add this to SpikeInterface if you want. I am trying to generalize it as much as a I can so it is not only specific to my MEA design.

One thing that might be nice with the SpikeInterface documentation is for the Compatible Technology to have a table that shows the current capabilities of each extractor (whether it can read and/or write, and what data it can extract/write). It might also be nice to have an infographic overview of the SpikeInterface data structure, to make it easier for mapping the data?

Hi @McD-NMI
I think having a function in spikeextractors that exports to NEO will be a good addition. The user should be able to pass any combination of recording and sorting extractor:

neo_block = se.export_to_neo(recording=rec_extractor, sorting=sort_extractor)
neo_block = se.export_to_neo(recording=rec_extractor, sorting=None)
neo_block = se.export_to_neo(recording=None, sorting=sort_extractor)

@samuelgarcia can probably help! I think it could go in the extraction_tools.

from spikeinterface.

@alejoe91 thanks for replying! You are much more knowledgeable about this part than I am.

Do you think the memory limitations of the phy exporter could potentially be overcome by extracting and writing out in chunks?

from spikeinterface.

mmm the waveforms are extracted to compute PCA features, so it's a bit more complicated than that. One could load a subset of features to compute the PCA model, then waveforms could be loaded, transformed, and saved out in chunks. But this would require an extensive rewrite.. I think that for large-scale recordings one needs an appropriate machine :)

from spikeinterface.

Right, we would still need to extract waveforms for PCA, although I think we can pick an appropriate amount of spikes for that. It would be quite nice if this worked on regular sized machines as well. Also, I think some recordings will be too big for even the largest machines!

I think I will make an issue for this that we can address at any point (low priority).

from spikeinterface.

Done in 0.90

from spikeinterface.