Factorized and Controllable Neural Re-Rendering of Outdoor Scene for Photo Extrapolation

by   Boming Zhao, et al.

Expanding an existing tourist photo from a partially captured scene to a full scene is one of the desired experiences for photography applications. Although photo extrapolation has been well studied, it is much more challenging to extrapolate a photo (i.e., selfie) from a narrow field of view to a wider one while maintaining a similar visual style. In this paper, we propose a factorized neural re-rendering model to produce photorealistic novel views from cluttered outdoor Internet photo collections, which enables the applications including controllable scene re-rendering, photo extrapolation and even extrapolated 3D photo generation. Specifically, we first develop a novel factorized re-rendering pipeline to handle the ambiguity in the decomposition of geometry, appearance and illumination. We also propose a composited training strategy to tackle the unexpected occlusion in Internet images. Moreover, to enhance photo-realism when extrapolating tourist photographs, we propose a novel realism augmentation process to complement appearance details, which automatically propagates the texture details from a narrow captured photo to the extrapolated neural rendered image. The experiments and photo editing examples on outdoor scenes demonstrate the superior performance of our proposed method in both photo-realism and downstream applications.


page 1

page 4

page 5

page 6

page 7

page 8


Neural Parameterization for Dynamic Human Head Editing

Implicit radiance functions emerged as a powerful scene representation f...

NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections

We present a learning-based method for synthesizing novel views of compl...

What Makes Kevin Spacey Look Like Kevin Spacey

We reconstruct a controllable model of a person from a large photo colle...

Plausible Shading Decomposition For Layered Photo Retouching

Photographers routinely compose multiple manipulated photos of the same ...

Shape and Reflectance Reconstruction in Uncontrolled Environments by Differentiable Rendering

Simultaneous reconstruction of geometry and reflectance properties in un...

Predicting Surface Reflectance Properties of Outdoor Scenes Under Unknown Natural Illumination

Estimating and modelling the appearance of an object under outdoor illum...

Unsupervised Contrastive Photo-to-Caricature Translation based on Auto-distortion

Photo-to-caricature translation aims to synthesize the caricature as a r...

Please sign up or login with your details

Forgot password? Click here to reset