Action2video: Generating Videos of Human 3D Actions

by   Chuan Guo, et al.

We aim to tackle the interesting yet challenging problem of generating videos of diverse and natural human motions from prescribed action categories. The key issue lies in the ability to synthesize multiple distinct motion sequences that are realistic in their visual appearances. It is achieved in this paper by a two-step process that maintains internal 3D pose and shape representations, action2motion and motion2video. Action2motion stochastically generates plausible 3D pose sequences of a prescribed action category, which are processed and rendered by motion2video to form 2D videos. Specifically, the Lie algebraic theory is engaged in representing natural human motions following the physical law of human kinematics; a temporal variational auto-encoder (VAE) is developed that encourages diversity of output motions. Moreover, given an additional input image of a clothed human character, an entire pipeline is proposed to extract his/her 3D detailed shape, and to render in videos the plausible motions from different views. This is realized by improving existing methods to extract 3D human shapes and textures from single 2D images, rigging, animating, and rendering to form 2D videos of human motions. It also necessitates the curation and reannotation of 3D human motion datasets for training purpose. Thorough empirical experiments including ablation study, qualitative and quantitative evaluations manifest the applicability of our approach, and demonstrate its competitiveness in addressing related tasks, where components of our approach are compared favorably to the state-of-the-arts.


page 2

page 13

page 18

page 21

page 22

page 23

page 24

page 26


Action2Motion: Conditioned Generation of 3D Human Motions

Action recognition is a relatively established task, where givenan input...

HuMoR: 3D Human Motion Model for Robust Pose Estimation

We introduce HuMoR: a 3D Human Motion Model for Robust Estimation of tem...

Conditional Temporal Variational AutoEncoder for Action Video Prediction

To synthesize a realistic action sequence based on a single human image,...

Weakly-supervised Action Transition Learning for Stochastic Human Motion Prediction

We introduce the task of action-driven stochastic human motion predictio...

Implicit Neural Representations for Variable Length Human Motion Generation

We propose an action-conditional human motion generation method using va...

Landmark-Guided Elastic Shape Analysis of Human Character Motions

Motions of virtual characters in movies or video games are typically gen...

3D Pose Estimation and Future Motion Prediction from 2D Images

This paper considers to jointly tackle the highly correlated tasks of es...

Please sign up or login with your details

Forgot password? Click here to reset