Learning to Recover Spectral Reflectance from RGB Images

by   Dong Huo, et al.

This paper tackles spectral reflectance recovery (SRR) from RGB images. Since capturing ground-truth spectral reflectance and camera spectral sensitivity are challenging and costly, most existing approaches are trained on synthetic images and utilize the same parameters for all unseen testing images, which are suboptimal especially when the trained models are tested on real images because they never exploit the internal information of the testing images. To address this issue, we adopt a self-supervised meta-auxiliary learning (MAXL) strategy that fine-tunes the well-trained network parameters with each testing image to combine external with internal information. To the best of our knowledge, this is the first work that successfully adapts the MAXL strategy to this problem. Instead of relying on naive end-to-end training, we also propose a novel architecture that integrates the physical relationship between the spectral reflectance and the corresponding RGB images into the network based on our mathematical analysis. Besides, since the spectral reflectance of a scene is independent to its illumination while the corresponding RGB images are not, we recover the spectral reflectance of a scene from its RGB images captured under multiple illuminations to further reduce the unknown. Qualitative and quantitative evaluations demonstrate the effectiveness of our proposed network and of the MAXL. Our code and data are available at https://github.com/Dong-Huo/SRR-MAXL.


page 1

page 4

page 8

page 9

page 10


Semantic-embedded Unsupervised Spectral Reconstruction from Single RGB Images in the Wild

This paper investigates the problem of reconstructing hyperspectral (HS)...

NTIRE 2020 Challenge on Spectral Reconstruction from an RGB Image

This paper reviews the second challenge on spectral reconstruction from ...

Hierarchical Regression Network for Spectral Reconstruction from RGB Images

Capturing visual image with a hyperspectral camera has been successfully...

Towards Spectral Estimation from a Single RGB Image in the Wild

In contrast to the current literature, we address the problem of estimat...

Deep Amended Gradient Descent for Efficient Spectral Reconstruction from Single RGB Images

This paper investigates the problem of recovering hyperspectral (HS) ima...

Spectral Graphormer: Spectral Graph-based Transformer for Egocentric Two-Hand Reconstruction using Multi-View Color Images

We propose a novel transformer-based framework that reconstructs two hig...

BioFaceNet: Deep Biophysical Face Image Interpretation

In this paper we present BioFaceNet, a deep CNN that learns to decompose...

Please sign up or login with your details

Forgot password? Click here to reset