Online Training Through Time for Spiking Neural Networks

by   Mingqing Xiao, et al.

Spiking neural networks (SNNs) are promising brain-inspired energy-efficient models. Recent progress in training methods has enabled successful deep SNNs on large-scale tasks with low latency. Particularly, backpropagation through time (BPTT) with surrogate gradients (SG) is popularly used to achieve high performance in a very small number of time steps. However, it is at the cost of large memory consumption for training, lack of theoretical clarity for optimization, and inconsistency with the online property of biological learning and rules on neuromorphic hardware. Other works connect spike representations of SNNs with equivalent artificial neural network formulation and train SNNs by gradients from equivalent mappings to ensure descent directions. But they fail to achieve low latency and are also not online. In this work, we propose online training through time (OTTT) for SNNs, which is derived from BPTT to enable forward-in-time learning by tracking presynaptic activities and leveraging instantaneous loss and gradients. Meanwhile, we theoretically analyze and prove that gradients of OTTT can provide a similar descent direction for optimization as gradients based on spike representations under both feedforward and recurrent conditions. OTTT only requires constant training memory costs agnostic to time steps, avoiding the significant memory costs of BPTT for GPU training. Furthermore, the update rule of OTTT is in the form of three-factor Hebbian learning, which could pave a path for online on-chip learning. With OTTT, it is the first time that two mainstream supervised SNN training methods, BPTT with SG and spike representation-based training, are connected, and meanwhile in a biologically plausible form. Experiments on CIFAR-10, CIFAR-100, ImageNet, and CIFAR10-DVS demonstrate the superior performance of our method on large-scale static and neuromorphic datasets in small time steps.


page 1

page 2

page 3

page 4


Training High-Performance Low-Latency Spiking Neural Networks by Differentiation on Spike Representation

Spiking Neural Network (SNN) is a promising energy-efficient AI model wh...

Towards Memory- and Time-Efficient Backpropagation for Training Spiking Neural Networks

Spiking Neural Networks (SNNs) are promising energy-efficient models for...

SPIDE: A Purely Spike-based Method for Training Feedback Spiking Neural Networks

Spiking neural networks (SNNs) with event-based computation are promisin...

Training Feedback Spiking Neural Networks by Implicit Differentiation on the Equilibrium State

Spiking neural networks (SNNs) are brain-inspired models that enable ene...

Training Spiking Neural Networks with Local Tandem Learning

Spiking neural networks (SNNs) are shown to be more biologically plausib...

Temporal Contrastive Learning for Spiking Neural Networks

Biologically inspired spiking neural networks (SNNs) have garnered consi...

H2Learn: High-Efficiency Learning Accelerator for High-Accuracy Spiking Neural Networks

Although spiking neural networks (SNNs) take benefits from the bio-plaus...

Please sign up or login with your details

Forgot password? Click here to reset