JOTR: 3D Joint Contrastive Learning with Transformers for Occluded Human Mesh Recovery

by   Jiahao Li, et al.

In this study, we focus on the problem of 3D human mesh recovery from a single image under obscured conditions. Most state-of-the-art methods aim to improve 2D alignment technologies, such as spatial averaging and 2D joint sampling. However, they tend to neglect the crucial aspect of 3D alignment by improving 3D representations. Furthermore, recent methods struggle to separate the target human from occlusion or background in crowded scenes as they optimize the 3D space of target human with 3D joint coordinates as local supervision. To address these issues, a desirable method would involve a framework for fusing 2D and 3D features and a strategy for optimizing the 3D space globally. Therefore, this paper presents 3D JOint contrastive learning with TRansformers (JOTR) framework for handling occluded 3D human mesh recovery. Our method includes an encoder-decoder transformer architecture to fuse 2D and 3D representations for achieving 2D&3D aligned results in a coarse-to-fine manner and a novel 3D joint contrastive learning approach for adding explicitly global supervision for the 3D feature space. The contrastive learning approach includes two contrastive losses: joint-to-joint contrast for enhancing the similarity of semantically similar voxels (i.e., human joints), and joint-to-non-joint contrast for ensuring discrimination from others (e.g., occlusions and background). Qualitative and quantitative analyses demonstrate that our method outperforms state-of-the-art competitors on both occlusion-specific and standard benchmarks, significantly improving the reconstruction of occluded humans.


page 1

page 4

page 8


Learning Dense UV Completion for Human Mesh Recovery

Human mesh reconstruction from a single image is challenging in the pres...

End-to-End Human Pose and Mesh Reconstruction with Transformers

We present a new method, called MEsh TRansfOrmer (METRO), to reconstruct...

Cross-Attention of Disentangled Modalities for 3D Human Mesh Recovery with Transformers

Transformer encoder architectures have recently achieved state-of-the-ar...

Occluded Human Mesh Recovery

Top-down methods for monocular human mesh recovery have two stages: (1) ...

Joint Contrastive Learning for Unsupervised Domain Adaptation

Enhancing feature transferability by matching marginal distributions has...

Hierarchical Kinematic Human Mesh Recovery

We consider the problem of estimating a parametric model of 3D human mes...

Please sign up or login with your details

Forgot password? Click here to reset