Gesture Recognition with Keypoint and Radar Stream Fusion for Automated Vehicles

02/20/2023

∙

by   Adrian Holzbock, et al.

∙

Otto-von-Guericke-University Magdeburg

∙

Universität Ulm

∙

23

∙

share

We present a joint camera and radar approach to enable autonomous vehicles to understand and react to human gestures in everyday traffic. Initially, we process the radar data with a PointNet followed by a spatio-temporal multilayer perceptron (stMLP). Independently, the human body pose is extracted from the camera frame and processed with a separate stMLP network. We propose a fusion neural network for both modalities, including an auxiliary loss for each modality. In our experiments with a collected dataset, we show the advantages of gesture recognition with two modalities. Motivated by adverse weather conditions, we also demonstrate promising performance when one of the sensors lacks functionality.