GINA-3D: Learning to Generate Implicit Neural Assets in the Wild

by   Bokui Shen, et al.

Modeling the 3D world from sensor data for simulation is a scalable way of developing testing and validation environments for robotic learning problems such as autonomous driving. However, manually creating or re-creating real-world-like environments is difficult, expensive, and not scalable. Recent generative model techniques have shown promising progress to address such challenges by learning 3D assets using only plentiful 2D images – but still suffer limitations as they leverage either human-curated image datasets or renderings from manually-created synthetic 3D environments. In this paper, we introduce GINA-3D, a generative model that uses real-world driving data from camera and LiDAR sensors to create realistic 3D implicit neural assets of diverse vehicles and pedestrians. Compared to the existing image datasets, the real-world driving setting poses new challenges due to occlusions, lighting-variations and long-tail distributions. GINA-3D tackles these challenges by decoupling representation learning and generative modeling into two stages with a learned tri-plane latent structure, inspired by recent advances in generative modeling of images. To evaluate our approach, we construct a large-scale object-centric dataset containing over 520K images of vehicles and pedestrians from the Waymo Open Dataset, and a new set of 80K images of long-tail instances such as construction equipment, garbage trucks, and cable cars. We compare our model with existing approaches and demonstrate that it achieves state-of-the-art performance in quality and diversity for both generated images and geometries.


page 1

page 5

page 7

page 8

page 15

page 21

page 22

page 23


A*3D Dataset: Towards Autonomous Driving in Challenging Environments

With the increasing global popularity of self-driving cars, there is an ...

AADS: Augmented Autonomous Driving Simulation using Data-driven Algorithms

Simulation systems have become an essential component in the development...

SurfelGAN: Synthesizing Realistic Sensor Data for Autonomous Driving

Autonomous driving system development is critically dependent on the abi...

Recovering and Simulating Pedestrians in the Wild

Sensor simulation is a key component for testing the performance of self...

Deep Object Centric Policies for Autonomous Driving

While learning visuomotor skills in an end-to-end manner is appealing, d...

Brand Label Albedo Extraction of eCommerce Products using Generative Adversarial Network

In this paper we present our solution to extract albedo of branded label...

InfiniteForm: A synthetic, minimal bias dataset for fitness applications

The growing popularity of remote fitness has increased the demand for hi...

Please sign up or login with your details

Forgot password? Click here to reset