DeepAI AI Chat
Log In Sign Up

Gesture Recognition with Keypoint and Radar Stream Fusion for Automated Vehicles

02/20/2023
by   Adrian Holzbock, et al.
Otto-von-Guericke-University Magdeburg
Universität Ulm
23

We present a joint camera and radar approach to enable autonomous vehicles to understand and react to human gestures in everyday traffic. Initially, we process the radar data with a PointNet followed by a spatio-temporal multilayer perceptron (stMLP). Independently, the human body pose is extracted from the camera frame and processed with a separate stMLP network. We propose a fusion neural network for both modalities, including an auxiliary loss for each modality. In our experiments with a collected dataset, we show the advantages of gesture recognition with two modalities. Motivated by adverse weather conditions, we also demonstrate promising performance when one of the sensors lacks functionality.

READ FULL TEXT
04/25/2022

A Spatio-Temporal Multilayer Perceptron for Gesture Recognition

Gesture recognition is essential for the interaction of autonomous vehic...
03/08/2023

Camera-Radar Perception for Autonomous Vehicles and ADAS: Concepts, Datasets and Metrics

One of the main paths towards the reduction of traffic accidents is the ...
11/18/2020

CGAP2: Context and gap aware predictive pose framework for early detection of gestures

With a growing interest in autonomous vehicles' operation, there is an e...
03/12/2021

A Continuous-Time Approach for 3D Radar-to-Camera Extrinsic Calibration

Reliable operation in inclement weather is essential to the deployment o...
08/28/2019

Efficient Convolutional Neural Network for FMCW Radar Based Hand Gesture Recognition

FMCW radar could detect object's range, speed and Angleof-Arrival, advan...
07/31/2020

Traffic Control Gesture Recognition for Autonomous Vehicles

A car driver knows how to react on the gestures of the traffic officers....