Transforming Radiance Field with Lipschitz Network for Photorealistic 3D Scene Stylization

by   Zicheng Zhang, et al.

Recent advances in 3D scene representation and novel view synthesis have witnessed the rise of Neural Radiance Fields (NeRFs). Nevertheless, it is not trivial to exploit NeRF for the photorealistic 3D scene stylization task, which aims to generate visually consistent and photorealistic stylized scenes from novel views. Simply coupling NeRF with photorealistic style transfer (PST) will result in cross-view inconsistency and degradation of stylized view syntheses. Through a thorough analysis, we demonstrate that this non-trivial task can be simplified in a new light: When transforming the appearance representation of a pre-trained NeRF with Lipschitz mapping, the consistency and photorealism across source views will be seamlessly encoded into the syntheses. That motivates us to build a concise and flexible learning framework namely LipRF, which upgrades arbitrary 2D PST methods with Lipschitz mapping tailored for the 3D scene. Technically, LipRF first pre-trains a radiance field to reconstruct the 3D scene, and then emulates the style on each view by 2D PST as the prior to learn a Lipschitz network to stylize the pre-trained appearance. In view of that Lipschitz condition highly impacts the expressivity of the neural network, we devise an adaptive regularization to balance the reconstruction and stylization. A gradual gradient aggregation strategy is further introduced to optimize LipRF in a cost-efficient manner. We conduct extensive experiments to show the high quality and robust performance of LipRF on both photorealistic 3D stylization and object appearance editing.


page 2

page 6

page 7

page 8


Stylizing 3D Scene via Implicit Representation and HyperNetwork

In this work, we aim to address the 3D scene stylization problem - gener...

SNeRF: Stylized Neural Implicit Representations for 3D Scenes

This paper presents a stylized novel view synthesis method. Applying sta...

NeRF-Art: Text-Driven Neural Radiance Fields Stylization

As a powerful representation of 3D scenes, the neural radiance field (Ne...

Locally Stylized Neural Radiance Fields

In recent years, there has been increasing interest in applying stylizat...

NeRFEditor: Differentiable Style Decomposition for Full 3D Scene Editing

We present NeRFEditor, an efficient learning framework for 3D scene edit...

StyleTRF: Stylizing Tensorial Radiance Fields

Stylized view generation of scenes captured casually using a camera has ...

General Neural Gauge Fields

The recent advance of neural fields, such as neural radiance fields, has...

Please sign up or login with your details

Forgot password? Click here to reset