Neural Residual Radiance Fields for Streamably Free-Viewpoint Videos

by   Liao Wang, et al.

The success of the Neural Radiance Fields (NeRFs) for modeling and free-view rendering static objects has inspired numerous attempts on dynamic scenes. Current techniques that utilize neural rendering for facilitating free-view videos (FVVs) are restricted to either offline rendering or are capable of processing only brief sequences with minimal motion. In this paper, we present a novel technique, Residual Radiance Field or ReRF, as a highly compact neural representation to achieve real-time FVV rendering on long-duration dynamic scenes. ReRF explicitly models the residual information between adjacent timestamps in the spatial-temporal feature space, with a global coordinate-based tiny MLP as the feature decoder. Specifically, ReRF employs a compact motion grid along with a residual feature grid to exploit inter-frame feature similarities. We show such a strategy can handle large motions without sacrificing quality. We further present a sequential training scheme to maintain the smoothness and the sparsity of the motion/residual grids. Based on ReRF, we design a special FVV codec that achieves three orders of magnitudes compression rate and provides a companion ReRF player to support online streaming of long-duration FVVs of dynamic scenes. Extensive experiments demonstrate the effectiveness of ReRF for compactly representing dynamic radiance fields, enabling an unprecedented free-viewpoint viewing experience in speed and quality.


page 1

page 6

page 7

page 8

page 16


Editable Free-viewpoint Video Using a Layered Neural Representation

Generating free-viewpoint videos is critical for immersive VR/AR experie...

Representing Volumetric Videos as Dynamic MLP Maps

This paper introduces a novel representation of volumetric videos for re...

Fast Dynamic Radiance Fields with Time-Aware Neural Voxels

Neural radiance fields (NeRF) have shown great success in modeling 3D sc...

NeRFPlayer: A Streamable Dynamic Scene Representation with Decomposed Neural Radiance Fields

Visually exploring in a real-world 4D spatiotemporal space freely in VR ...

Temporal Interpolation Is All You Need for Dynamic Neural Radiance Fields

Temporal interpolation often plays a crucial role to learn meaningful re...

Real-time Neural Radiance Talking Portrait Synthesis via Audio-spatial Decomposition

While dynamic Neural Radiance Fields (NeRF) have shown success in high-f...

Envisioning a Next Generation Extended Reality Conferencing System with Efficient Photorealistic Human Rendering

Meeting online is becoming the new normal. Creating an immersive experie...

Please sign up or login with your details

Forgot password? Click here to reset