Physics-based Differentiable Depth Sensor Simulation

by   Benjamin Planche, et al.

Gradient-based algorithms are crucial to modern computer-vision and graphics applications, enabling learning-based optimization and inverse problems. For example, photorealistic differentiable rendering pipelines for color images have been proven highly valuable to applications aiming to map 2D and 3D domains. However, to the best of our knowledge, no effort has been made so far towards extending these gradient-based methods to the generation of depth (2.5D) images, as simulating structured-light depth sensors implies solving complex light transport and stereo-matching problems. In this paper, we introduce a novel end-to-end differentiable simulation pipeline for the generation of realistic 2.5D scans, built on physics-based 3D rendering and custom block-matching algorithms. Each module can be differentiated w.r.t sensor and scene parameters; e.g., to automatically tune the simulation for new devices over some provided scans or to leverage the pipeline as a 3D-to-2.5D transformer within larger computer-vision applications. Applied to the training of deep-learning methods for various depth-based recognition tasks (classification, pose estimation, semantic segmentation), our simulation greatly improves the performance of the resulting models on real scans, thereby demonstrating the fidelity and value of its synthetic depth data compared to previous static simulations and learning-based domain adaptation schemes.


page 1

page 2

page 4

page 5

page 8

page 12

page 13


Differentiable Rendering for Pose Estimation in Proximity Operations

Differentiable rendering aims to compute the derivative of the image ren...

DepthSynth: Real-Time Realistic Synthetic Data Generation from CAD Models for 2.5D Recognition

Recent progress in computer vision has been dominated by deep neural net...

Hair Color Digitization through Imaging and Deep Inverse Graphics

Hair appearance is a complex phenomenon due to hair geometry and how the...

A Non-linear Differential CNN-Rendering Module for 3D Data Enhancement

In this work we introduce a differential rendering module which allows n...

NaRPA: Navigation and Rendering Pipeline for Astronautics

This paper presents Navigation and Rendering Pipeline for Astronautics (...

Deep Depth From Focus

Depth from Focus (DFF) is one of the classical ill-posed inverse problem...

Deep Phase Correlation for End-to-End Heterogeneous Sensor Measurements Matching

The crucial step for localization is to match the current observation to...

Please sign up or login with your details

Forgot password? Click here to reset