Neural Projection Mapping Using Reflectance Fields

by   Yotam Erel, et al.

We introduce a high resolution spatially adaptive light source, or a projector, into a neural reflectance field that allows to both calibrate the projector and photo realistic light editing. The projected texture is fully differentiable with respect to all scene parameters, and can be optimized to yield a desired appearance suitable for applications in augmented reality and projection mapping. Our neural field consists of three neural networks, estimating geometry, material, and transmittance. Using an analytical BRDF model and carefully selected projection patterns, our acquisition process is simple and intuitive, featuring a fixed uncalibrated projected and a handheld camera with a co-located light source. As we demonstrate, the virtual projector incorporated into the pipeline improves scene understanding and enables various projection mapping applications, alleviating the need for time consuming calibration steps performed in a traditional setting per view or projector location. In addition to enabling novel viewpoint synthesis, we demonstrate state-of-the-art performance projector compensation for novel viewpoints, improvement over the baselines in material and scene reconstruction, and three simply implemented scenarios where projection image optimization is performed, including the use of a 2D generative model to consistently dictate scene appearance from multiple viewpoints. We believe that neural projection mapping opens up the door to novel and exciting downstream tasks, through the joint optimization of the scene and projection images.


page 1

page 3

page 5

page 6

page 7

page 8

page 9


Is Projection Mapping Natural? Towards Physical World Augmentation Consistent with Light Field Context

Projection mapping seamlessly merges real and virtual worlds. Although m...

Neural Reflectance Fields for Appearance Acquisition

We present Neural Reflectance Fields, a novel deep scene representation ...

A Generic Multi-Projection-Center Model and Calibration Method for Light Field Cameras

Light field cameras can capture both spatial and angular information of ...

Lightform: Procedural Effects for Projected AR

Projected augmented reality, also called projection mapping or video map...

A Multiple-View Geometric Model for Specularity Prediction on Non-Uniformly Curved Surfaces

Specularity prediction is essential to many computer vision applications...

Single View Refractive Index Tomography with Neural Fields

Refractive Index Tomography is an inverse problem in which we seek to re...

Neural 360^∘ Structured Light with Learned Metasurfaces

Structured light has proven instrumental in 3D imaging, LiDAR, and holog...

Please sign up or login with your details

Forgot password? Click here to reset