SC-NeRF: Self-Correcting Neural Radiance Field with Sparse Views

by   Liang Song, et al.

In recent studies, the generalization of neural radiance fields for novel view synthesis task has been widely explored. However, existing methods are limited to objects and indoor scenes. In this work, we extend the generalization task to outdoor scenes, trained only on object-level datasets. This approach presents two challenges. Firstly, the significant distributional shift between training and testing scenes leads to black artifacts in rendering results. Secondly, viewpoint changes in outdoor scenes cause ghosting or missing regions in rendered images. To address these challenges, we propose a geometric correction module and an appearance correction module based on multi-head attention mechanisms. We normalize rendered depth and combine it with light direction as query in the attention mechanism. Our network effectively corrects varying scene structures and geometric features in outdoor scenes, generalizing well from object-level to unseen outdoor scenes. Additionally, we use appearance correction module to correct appearance features, preventing rendering artifacts like blank borders and ghosting due to viewpoint changes. By combining these modules, our approach successfully tackles the challenges of outdoor scene generalization, producing high-quality rendering results. When evaluated on four datasets (Blender, DTU, LLFF, Spaces), our network outperforms previous methods. Notably, compared to MVSNeRF, our network improves average PSNR from 19.369 to 25.989, SSIM from 0.838 to 0.889, and reduces LPIPS from 0.265 to 0.224 on Spaces outdoor scenes.


page 1

page 3

page 6

page 7

page 8

page 10


Neural Reflectance Fields for Appearance Acquisition

We present Neural Reflectance Fields, a novel deep scene representation ...

GARF:Geometry-Aware Generalized Neural Radiance Field

Neural Radiance Field (NeRF) has revolutionized free viewpoint rendering...

Predicting Surface Reflectance Properties of Outdoor Scenes Under Unknown Natural Illumination

Estimating and modelling the appearance of an object under outdoor illum...

GenLayNeRF: Generalizable Layered Representations with 3D Model Alignment for Multi-Human View Synthesis

Novel view synthesis (NVS) of multi-human scenes imposes challenges due ...

Learning to Dehaze From Realistic Scene with A Fast Physics Based Dehazing Network

Dehaze is one of the popular computer vision research topics for long. A...

Learning Geo-Temporal Image Features

We propose to implicitly learn to extract geo-temporal image features, w...

Mega-NeRF: Scalable Construction of Large-Scale NeRFs for Virtual Fly-Throughs

We use neural radiance fields (NeRFs) to build interactive 3D environmen...

Please sign up or login with your details

Forgot password? Click here to reset