Training Deep Spiking Neural Networks

by   Eimantas Ledinauskas, et al.
SRI International

Computation using brain-inspired spiking neural networks (SNNs) with neuromorphic hardware may offer orders of magnitude higher energy efficiency compared to the current analog neural networks (ANNs). Unfortunately, training SNNs with the same number of layers as state of the art ANNs remains a challenge. To our knowledge the only method which is successful in this regard is supervised training of ANN and then converting it to SNN. In this work we directly train deep SNNs using backpropagation with surrogate gradient and find that due to implicitly recurrent nature of feed forward SNN's the exploding or vanishing gradient problem severely hinders their training. We show that this problem can be solved by tuning the surrogate gradient function. We also propose using batch normalization from ANN literature on input currents of SNN neurons. Using these improvements we show that is is possible to train SNN with ResNet50 architecture on CIFAR100 and Imagenette object recognition datasets. The trained SNN falls behind in accuracy compared to analogous ANN but requires several orders of magnitude less inference time steps (as low as 10) to reach good accuracy compared to SNNs obtained by conversion from ANN which require on the order of 1000 time steps.


Joint A-SNN: Joint Training of Artificial and Spiking Neural Networks via Self-Distillation and Weight Factorization

Emerged as a biology-inspired method, Spiking Neural Networks (SNNs) mim...

Deep Directly-Trained Spiking Neural Networks for Object Detection

Spiking neural networks (SNNs) are brain-inspired energy-efficient model...

ARRID: ANN-based Rotordynamics for Robust and Integrated Design

The purpose of this study is to introduce ANN-based software for the fas...

SpikingBERT: Distilling BERT to Train Spiking Language Models Using Implicit Differentiation

Large language Models (LLMs), though growing exceedingly powerful, compr...

Encrypted Internet traffic classification using a supervised Spiking Neural Network

Internet traffic recognition is an essential tool for access providers s...

Examining the Robustness of Spiking Neural Networks on Non-ideal Memristive Crossbars

Spiking Neural Networks (SNNs) have recently emerged as the low-power al...

Are training trajectories of deep single-spike and deep ReLU network equivalent?

Communication by binary and sparse spikes is a key factor for the energy...

Code Repositories


Implementation of "Incorporating Learnable Membrane Time Constant to Enhance Learning of Spiking Neural Networks" by Fang et al. (2020)

view repo

Please sign up or login with your details

Forgot password? Click here to reset