Two-Stream 3D Convolutional Neural Network for Skeleton-Based Action Recognition

by   Hong Liu, et al.

It remains a challenge to efficiently extract spatialtemporal information from skeleton sequences for 3D human action recognition. Although most recent action recognition methods are based on Recurrent Neural Networks which present outstanding performance, one of the shortcomings of these methods is the tendency to overemphasize the temporal information. Since 3D convolutional neural network(3D CNN) is a powerful tool to simultaneously learn features from both spatial and temporal dimensions through capturing the correlations between three dimensional signals, this paper proposes a novel two-stream model using 3D CNN. To our best knowledge, this is the first application of 3D CNN in skeleton-based action recognition. Our method consists of three stages. First, skeleton joints are mapped into a 3D coordinate space and then encoding the spatial and temporal information, respectively. Second, 3D CNN models are seperately adopted to extract deep features from two streams. Third, to enhance the ability of deep features to capture global relationships, we extend every stream into multitemporal version. Extensive experiments on the SmartHome dataset and the large-scale NTU RGB-D dataset demonstrate that our method outperforms most of RNN-based methods, which verify the complementary property between spatial and temporal information and the robustness to noise.


Dynamic Spatial-temporal Hypergraph Convolutional Network for Skeleton-based Action Recognition

Skeleton-based action recognition relies on the extraction of spatial-te...

Skeleton-Based Relational Modeling for Action Recognition

With the fast development of effective and low-cost human skeleton captu...

Global Temporal Representation based CNNs for Infrared Action Recognition

Infrared human action recognition has many advantages, i.e., it is insen...

A Fine-to-Coarse Convolutional Neural Network for 3D Human Action Recognition

This paper presents a new framework for human action recognition from 3D...

Infrared and 3D skeleton feature fusion for RGB-D action recognition

A challenge of skeleton-based action recognition is the difficulty to cl...

Combining ConvNets with Hand-Crafted Features for Action Recognition Based on an HMM-SVM Classifier

This paper proposes a new framework for RGB-D-based action recognition t...

Three-stream network for enriched Action Recognition

Understanding accurate information on human behaviours is one of the mos...

Please sign up or login with your details

Forgot password? Click here to reset