Unsupervised Hyperspectral and Multispectral Images Fusion Based on the Cycle Consistency

by   Shuaikai Shi, et al.

Hyperspectral images (HSI) with abundant spectral information reflected materials property usually perform low spatial resolution due to the hardware limits. Meanwhile, multispectral images (MSI), e.g., RGB images, have a high spatial resolution but deficient spectral signatures. Hyperspectral and multispectral image fusion can be cost-effective and efficient for acquiring both high spatial resolution and high spectral resolution images. Many of the conventional HSI and MSI fusion algorithms rely on known spatial degradation parameters, i.e., point spread function, spectral degradation parameters, spectral response function, or both of them. Another class of deep learning-based models relies on the ground truth of high spatial resolution HSI and needs large amounts of paired training images when working in a supervised manner. Both of these models are limited in practical fusion scenarios. In this paper, we propose an unsupervised HSI and MSI fusion model based on the cycle consistency, called CycFusion. The CycFusion learns the domain transformation between low spatial resolution HSI (LrHSI) and high spatial resolution MSI (HrMSI), and the desired high spatial resolution HSI (HrHSI) are considered to be intermediate feature maps in the transformation networks. The CycFusion can be trained with the objective functions of marginal matching in single transform and cycle consistency in double transforms. Moreover, the estimated PSF and SRF are embedded in the model as the pre-training weights, which further enhances the practicality of our proposed model. Experiments conducted on several datasets show that our proposed model outperforms all compared unsupervised fusion methods. The codes of this paper will be available at this address: https: //github.com/shuaikaishi/CycFusion for reproducibility.


page 1

page 4

page 6

page 7

page 8

page 9

page 10


Hyperspectral and Multispectral Image Fusion Using the Conditional Denoising Diffusion Probabilistic Model

Hyperspectral images (HSI) have a large amount of spectral information r...

DiriNet: A network to estimate the spatial and spectral degradation functions

The spatial and spectral degradation functions are critical to hyper- an...

Enhanced Hyperspectral Image Super-Resolution via RGB Fusion and TV-TV Minimization

Hyperspectral (HS) images contain detailed spectral information that has...

Unsupervised Deep Learning-based Pansharpening with Jointly-Enhanced Spectral and Spatial Fidelity

In latest years, deep learning has gained a leading role in the pansharp...

Hyperspectral Pansharpening Based on Improved Deep Image Prior and Residual Reconstruction

Hyperspectral pansharpening aims to synthesize a low-resolution hyperspe...

Unsupervised Spectral Demosaicing with Lightweight Spectral Attention Networks

This paper presents a deep learning-based spectral demosaicing technique...

Unsupervised Cycle-consistent Generative Adversarial Networks for Pan-sharpening

Deep learning based pan-sharpening has received significant research int...

Please sign up or login with your details

Forgot password? Click here to reset