Motion Equivariant Networks for Event Cameras with the Temporal Normalization Transform

by   Alex Zihao Zhu, et al.

In this work, we propose a novel transformation for events from an event camera that is equivariant to optical flow under convolutions in the 3-D spatiotemporal domain. Events are generated by changes in the image, which are typically due to motion, either of the camera or the scene. As a result, different motions result in a different set of events. For learning based tasks based on a static scene such as classification which directly use the events, we must either rely on the learning method to learn the underlying object distinct from the motion, or to memorize all possible motions for each object with extensive data augmentation. Instead, we propose a novel transformation of the input event data which normalizes the x and y positions by the timestamp of each event. We show that this transformation generates a representation of the events that is equivariant to this motion when the optical flow is constant, allowing a deep neural network to learn the classification task without the need for expensive data augmentation. We test our method on the event based N-MNIST dataset, as well as a novel dataset N-MOVING-MNIST, with significantly more variety in motion compared to the standard N-MNIST dataset. In all sequences, we demonstrate that our transformed network is able to achieve similar or better performance compared to a network with a standard volumetric event input, and performs significantly better when the test set has a larger set of motions than seen at training.


page 1

page 2

page 3

page 4


Unsupervised Event-based Learning of Optical Flow, Depth, and Egomotion

In this work, we propose a novel framework for unsupervised learning for...

Motion Equivariance OF Event-based Camera Data with the Temporal Normalization Transform

In this work, we focus on using convolution neural networks (CNN) to per...

Object Localization and Motion Transfer learning with Capsules

Inspired by CapsNet's routing-by-agreement mechanism, with its ability t...

EventHPE: Event-based 3D Human Pose and Shape Estimation

Event camera is an emerging imaging sensor for capturing dynamics of mov...

TMA: Temporal Motion Aggregation for Event-based Optical Flow

Event cameras have the ability to record continuous and detailed traject...

Improved Regularization of Event-based Learning by Reversing and Drifting

Event camera has an enormous potential in challenging scenes for its adv...

Learning Optical Flow from Event Camera with Rendered Dataset

We study the problem of estimating optical flow from event cameras. One ...

Please sign up or login with your details

Forgot password? Click here to reset