Slow curation and export about spikeinterface HOT 5 CLOSED

My guess is that the bottleneck is likely the function collecting spike cut-outs (both used in curation and phy export). What format is your raw data file? Also, do you apply any filters on the recording extractor?

from spikeinterface.

The raw data was acquired with Open Ephys GUI, stored in their binary format. I'm applying bandpass filtering and common mode rejection on the recording extractor, and using the cmr'd extractor for sorting, curation, phy, etc.

Is it possible to speed up the cut-out step somehow? or at least to save the cutouts in one step so they don't have to be re-computed several times?

from spikeinterface.

in spike interface there is a cache system to avoid to recompute the preprocessing chain and extractor several times waveforms snippet (very long) @alejoe91 do we have a notebook example for the cache workflow ?

from spikeinterface.

Hi @guitchounts

there are a few things you can do:

1. Cache the recording_cmr (VERY IMPORTANT): this dumps a binary version of your recording (in your case bandpass filtered + cmr) to a dat binary file. You can simply do: recording_cmr = se.CacheRecordingExtractor(recording_cmr, 'recording_cmr.dat'). The process will take a few minutes, but it'll speed up everything else (otherwise each single waveforms is filtered separately later!)

2. Use multiple jobs: when computing the SNR (and therefore the waveforms in the background) you can use the n_jobs=8 (or 4-16...). This will compute waveforms and templates in parallel. Same holds for the export_to_phy.

3. Note that the export to phy is slower because ALL waveforms need to be extracted to compute amplitudes and PC scores. If you are not interested in those (you'll still be able to look at templates, similarity, and cross-correlograms), you can try the to set the compute_amplitudes=False and compute_pc_features=False when calling the export_to_phy

Hope this helps! :)

from spikeinterface.

Potprocessing efficiency issues are solved with latest releases of spiketoolkit

from spikeinterface.