Instance Neural Radiance Field

by   Benran Hu, et al.

This paper presents one of the first learning-based NeRF 3D instance segmentation pipelines, dubbed as Instance Neural Radiance Field, or Instance NeRF. Taking a NeRF pretrained from multi-view RGB images as input, Instance NeRF can learn 3D instance segmentation of a given scene, represented as an instance field component of the NeRF model. To this end, we adopt a 3D proposal-based mask prediction network on the sampled volumetric features from NeRF, which generates discrete 3D instance masks. The coarse 3D mask prediction is then projected to image space to match 2D segmentation masks from different views generated by existing panoptic segmentation models, which are used to supervise the training of the instance field. Notably, beyond generating consistent 2D segmentation maps from novel views, Instance NeRF can query instance information at any 3D point, which greatly enhances NeRF object segmentation and manipulation. Our method is also one of the first to achieve such results without ground-truth instance information during inference. Experimented on synthetic and real-world NeRF datasets with complex indoor scenes, Instance NeRF surpasses previous NeRF segmentation works and competitive 2D segmentation methods in segmentation performance on unseen views. See the demo video at


page 1

page 4

page 7

page 11

page 12

page 13


3D Instance Segmentation of MVS Buildings

We present a novel framework for instance segmentation of 3D buildings f...

UnScene3D: Unsupervised 3D Instance Segmentation for Indoor Scenes

3D instance segmentation is fundamental to geometric understanding of th...

Panoptic Lifting for 3D Scene Understanding with Neural Fields

We propose Panoptic Lifting, a novel approach for learning panoptic 3D v...

Real-time GeoAI for High-resolution Mapping and Segmentation of Arctic Permafrost Features

This paper introduces a real-time GeoAI workflow for large-scale image a...

OpenIns3D: Snap and Lookup for 3D Open-vocabulary Instance Segmentation

Current 3D open-vocabulary scene understanding methods mostly utilize we...

Segment Anything in 3D with NeRFs

The Segment Anything Model (SAM) has demonstrated its effectiveness in s...

Please sign up or login with your details

Forgot password? Click here to reset