Learnable Triangulation for Deep Learning-based 3D Reconstruction of Objects of Arbitrary Topology from Single RGB Images

by   Tarek Ben Charrada, et al.

We propose a novel deep reinforcement learning-based approach for 3D object reconstruction from monocular images. Prior works that use mesh representations are template based. Thus, they are limited to the reconstruction of objects that have the same topology as the template. Methods that use volumetric grids as intermediate representations are computationally expensive, which limits their application in real-time scenarios. In this paper, we propose a novel end-to-end method that reconstructs 3D objects of arbitrary topology from a monocular image. It is composed of of (1) a Vertex Generation Network (VGN), which predicts the initial 3D locations of the object's vertices from an input RGB image, (2) a differentiable triangulation layer, which learns in a non-supervised manner, using a novel reinforcement learning algorithm, the best triangulation of the object's vertices, and finally, (3) a hierarchical mesh refinement network that uses graph convolutions to refine the initial mesh. Our key contribution is the learnable triangulation process, which recovers in an unsupervised manner the topology of the input shape. Our experiments on ShapeNet and Pix3D benchmarks show that the proposed method outperforms the state-of-the-art in terms of visual quality, reconstruction accuracy, and computational time.


page 7

page 14

page 15

page 16

page 17

page 18


Deep Mesh Reconstruction from Single RGB Images via Topology Modification Networks

Reconstructing the 3D mesh of a general object from a single image is no...

HumanMeshNet: Polygonal Mesh Recovery of Humans

3D Human Body Reconstruction from a monocular image is an important prob...

Learning monocular 3D reconstruction of articulated categories from motion

Monocular 3D reconstruction of articulated object categories is challeng...

Neural Template: Topology-aware Reconstruction and Disentangled Generation of 3D Meshes

This paper introduces a novel framework called DTNet for 3D mesh reconst...

Deep Shape-from-Template: Wide-Baseline, Dense and Fast Registration and Deformable Reconstruction from a Single Image

We present Deep Shape-from-Template (DeepSfT), a novel Deep Neural Netwo...

FOF: Learning Fourier Occupancy Field for Monocular Real-time Human Reconstruction

The advent of deep learning has led to significant progress in monocular...

Cut-and-Approximate: 3D Shape Reconstruction from Planar Cross-sections with Deep Reinforcement Learning

Current methods for 3D object reconstruction from a set of planar cross-...

Please sign up or login with your details

Forgot password? Click here to reset