Are training trajectories of deep single-spike and deep ReLU network equivalent?

by   Ana Stanojevic, et al.

Communication by binary and sparse spikes is a key factor for the energy efficiency of biological brains. However, training deep spiking neural networks (SNNs) with backpropagation is harder than with artificial neural networks (ANNs), which is puzzling given that recent theoretical results provide exact mapping algorithms from ReLU to time-to-first-spike (TTFS) SNNs. Building upon these results, we analyze in theory and in simulation the learning dynamics of TTFS-SNNs. Our analysis highlights that even when an SNN can be mapped exactly to a ReLU network, it cannot always be robustly trained by gradient descent. The reason for that is the emergence of a specific instance of the vanishing-or-exploding gradient problem leading to a bias in the gradient descent trajectory in comparison with the equivalent ANN. After identifying this issue we derive a generic solution for the network initialization and SNN parameterization which guarantees that the SNN can be trained as robustly as its ANN counterpart. Our theoretical findings are illustrated in practice on image classification datasets. Our method achieves the same accuracy as deep ConvNets on CIFAR10 and enables fine-tuning on the much larger PLACES365 dataset without loss of accuracy compared to the ANN. We argue that the combined perspective of conversion and fine-tuning with robust gradient descent in SNN will be decisive to optimize SNNs for hardware implementations needing low latency and resilience to noise and quantization.


page 1

page 2

page 3

page 4


Recognizing Images with at most one Spike per Neuron

In order to port the performance of trained artificial neural networks (...

An Exact Mapping From ReLU Networks to Spiking Neural Networks

Deep spiking neural networks (SNNs) offer the promise of low-power artif...

Training Deep Spiking Neural Networks

Computation using brain-inspired spiking neural networks (SNNs) with neu...

Hybrid Spiking Neural Network Fine-tuning for Hippocampus Segmentation

Over the past decade, artificial neural networks (ANNs) have made tremen...

Controlled Descent Training

In this work, a novel and model-based artificial neural network (ANN) tr...

Sparse, guided feature connections in an Abstract Deep Network

We present a technique for developing a network of re-used features, whe...

Pairwise Neural Networks (PairNets) with Low Memory for Fast On-Device Applications

A traditional artificial neural network (ANN) is normally trained slowly...

Please sign up or login with your details

Forgot password? Click here to reset