Koopman pose predictions for temporally consistent human walking estimations

by   Marc Mitjans, et al.

We tackle the problem of tracking the human lower body as an initial step toward an automatic motion assessment system for clinical mobility evaluation, using a multimodal system that combines Inertial Measurement Unit (IMU) data, RGB images, and point cloud depth measurements. This system applies the factor graph representation to an optimization problem that provides 3-D skeleton joint estimations. In this paper, we focus on improving the temporal consistency of the estimated human trajectories to greatly extend the range of operability of the depth sensor. More specifically, we introduce a new factor graph factor based on Koopman theory that embeds the nonlinear dynamics of several lower-limb movement activities. This factor performs a two-step process: first, a custom activity recognition module based on spatial temporal graph convolutional networks recognizes the walking activity; then, a Koopman pose prediction of the subsequent skeleton is used as an a priori estimation to drive the optimization problem toward more consistent results. We tested the performance of this module on datasets composed of multiple clinical lowerlimb mobility tests, and we show that our approach reduces outliers on the skeleton form by almost 1 m, while preserving natural walking trajectories at depths up to more than 10 m.


Spatial Temporal Transformer Network for Skeleton-based Action Recognition

Skeleton-based Human Activity Recognition has achieved a great interest ...

Non-local Graph Convolutional Network for joint Activity Recognition and Motion Prediction

3D skeleton-based motion prediction and activity recognition are two int...

Skeleton Focused Human Activity Recognition in RGB Video

The data-driven approach that learns an optimal representation of vision...

Pose Uncertainty Aware Movement Synchrony Estimation via Spatial-Temporal Graph Transformer

Movement synchrony reflects the coordination of body movements between i...

We Hear Your Activities through Wi-Fi Signals

In this paper we focus on the problem of human activity recognition with...

Human Pose Estimation from Sparse Inertial Measurements through Recurrent Graph Convolution

We propose the adjacency adaptive graph convolutional long-short term me...

Please sign up or login with your details

Forgot password? Click here to reset