Inserting Noise and Complex Architectures about snn_toolbox HOT 5 CLOSED

Hi Saif,

Thanks for your interesting questions.

1. Yes, we have experimented with noise added to the ANN during training. See for instance the script snn_toolbox/scripts/ann_architectures/cifar10/noise.py. You are probably aware of the work by Hunsberger et al. in this direction (https://arxiv.org/abs/1510.08829). Were you thinking of adding noise during inference of the SNN as well (instead of just during training of the ANN)? Not sure if this will bring additional benefits, as the SNN is inherently noisy itself. A way to implement noise in the SNN would be to add a random charge to each neuron's membrane potential at every time step, or to change the threshold slightly. To do this, you would have to edit the method snntoolbox.simulation.backends.inisim.temporal_mean_rate_{theano or tensorflow}.SpikeLayer.get_new_mem.

2. I have not done it myself, but since the converted SNN is essentially a Keras model, you can do whatever you would normally do with a Keras network. And of course you can make the neurons inhibitory by constraining the weights to be negative.

3. No, we generally do not retrain the weights after ANN-to-SNN conversion. We only normalize the parameters of the ANN such that the maximum activation in each layer is at most 1, to avoid saturating spike rates in the SNN. However, since the converted SNN is implemented as a Keras model, it is in principle possible to continue training the converted SNN. You just have to decide what learning rule to use (spike-based, rate-based, ...) and implement it ...

Good luck and best regards,

Bodo

from snn_toolbox.

Hi Bodo,

1. My ANN is trained with noise too. For some weird reason, one of my layers have zero activations for some samples. I was wondering if the noise might help, given I was assuming online learning too. Thanks for the clear pointer.

2. I'm learning a lot from you!

3. I'll have to dig into Keras and snntoolbox source code now. If I could implement it, I'd also be able to train a fresh SNN and compare the results. As you might have guessed, I'm a novice. Am I being overly ambitious here?

Best,
Saif

from snn_toolbox.

3. There is a growing body of literature on spike-based learning in deep neural networks, with spiking versions of back-propagation, Hebbian / STDP rules etc.
   This problem is far from solved, and the existing methods may take some effort to implement. I don't want to discourage you, but yes, this can be a substantial project.

from snn_toolbox.

So there's nothing at all that implements online-learning in SNNs? (even if it's without a GPU)

3. What is the use of "batch_size" in the config file if there is no learning taking place?

from snn_toolbox.

3. The batch_size parameter in the config file can be used to run several examples in parallel during inference.

For supervised spike-based learning implementations, have a look at

https://github.com/petered/spiking-mlp
https://github.com/fzenke/ssbm

from snn_toolbox.