3DLatNav: Navigating Generative Latent Spaces for Semantic-Aware 3D Object Manipulation

by   Amaya Dharmasiri, et al.

3D generative models have been recently successful in generating realistic 3D objects in the form of point clouds. However, most models do not offer controllability to manipulate the shape semantics of component object parts without extensive semantic attribute labels or other reference point clouds. Moreover, beyond the ability to perform simple latent vector arithmetic or interpolations, there is a lack of understanding of how part-level semantics of 3D shapes are encoded in their corresponding generative latent spaces. In this paper, we propose 3DLatNav; a novel approach to navigating pretrained generative latent spaces to enable controlled part-level semantic manipulation of 3D objects. First, we propose a part-level weakly-supervised shape semantics identification mechanism using latent representations of 3D shapes. Then, we transfer that knowledge to a pretrained 3D object generative latent space to unravel disentangled embeddings to represent different shape semantics of component parts of an object in the form of linear subspaces, despite the unavailability of part-level labels during the training. Finally, we utilize those identified subspaces to show that controllable 3D object part manipulation can be achieved by applying the proposed framework to any pretrained 3D generative model. With two novel quantitative metrics to evaluate the consistency and localization accuracy of part-level manipulations, we show that 3DLatNav outperforms existing unsupervised latent disentanglement methods in identifying latent directions that encode part-level shape semantics of 3D objects. With multiple ablation studies and testing on state-of-the-art generative models, we show that 3DLatNav can implement controlled part-level semantic manipulations on an input point cloud while preserving other features and the realistic nature of the object.


Latent-Space Laplacian Pyramids for Adversarial Representation Learning with 3D Point Clouds

Constructing high-quality generative models for 3D shapes is a fundament...

StylePart: Image-based Shape Part Manipulation

Due to a lack of image-based "part controllers", shape manipulation of m...

Learning to Manipulate Individual Objects in an Image

We describe a method to train a generative model with latent factors tha...

LPMNet: Latent Part Modification and Generation for 3D Point Clouds

In this paper, we focus on latent modification and generation of 3D poin...

StructureNet: Hierarchical Graph Networks for 3D Shape Generation

The ability to generate novel, diverse, and realistic 3D shapes along wi...

Automatic generation of object shapes with desired functionalities

Functional design of objects is slow and still largely an artisanal acti...

ContraFeat: Contrasting Deep Features for Semantic Discovery

StyleGAN has shown strong potential for disentangled semantic control, t...

Please sign up or login with your details

Forgot password? Click here to reset