Local-Global Transformer Enhanced Unfolding Network for Pan-sharpening

by   Mingsong Li, et al.

Pan-sharpening aims to increase the spatial resolution of the low-resolution multispectral (LrMS) image with the guidance of the corresponding panchromatic (PAN) image. Although deep learning (DL)-based pan-sharpening methods have achieved promising performance, most of them have a two-fold deficiency. For one thing, the universally adopted black box principle limits the model interpretability. For another thing, existing DL-based methods fail to efficiently capture local and global dependencies at the same time, inevitably limiting the overall performance. To address these mentioned issues, we first formulate the degradation process of the high-resolution multispectral (HrMS) image as a unified variational optimization problem, and alternately solve its data and prior subproblems by the designed iterative proximal gradient descent (PGD) algorithm. Moreover, we customize a Local-Global Transformer (LGT) to simultaneously model local and global dependencies, and further formulate an LGT-based prior module for image denoising. Besides the prior module, we also design a lightweight data module. Finally, by serially integrating the data and prior modules in each iterative stage, we unfold the iterative algorithm into a stage-wise unfolding network, Local-Global Transformer Enhanced Unfolding Network (LGTEUN), for the interpretable MS pan-sharpening. Comprehensive experimental results on three satellite data sets demonstrate the effectiveness and efficiency of LGTEUN compared with state-of-the-art (SOTA) methods. The source code is available at https://github.com/lms-07/LGTEUN.


page 5

page 6

page 7


Dense residual Transformer for image denoising

Image denoising is an important low-level computer vision task, which ai...

PanFormer: a Transformer Based Model for Pan-sharpening

Pan-sharpening aims at producing a high-resolution (HR) multi-spectral (...

Pixel Adaptive Deep Unfolding Transformer for Hyperspectral Image Reconstruction

Hyperspectral Image (HSI) reconstruction has made gratifying progress wi...

Deep Generalized Unfolding Networks for Image Restoration

Deep neural networks (DNN) have achieved great success in image restorat...

SUMD: Super U-shaped Matrix Decomposition Convolutional neural network for Image denoising

In this paper, we propose a novel and efficient CNN-based framework that...

HQDec: Self-Supervised Monocular Depth Estimation Based on a High-Quality Decoder

Decoders play significant roles in recovering scene depths. However, the...

RCDNet: An Interpretable Rain Convolutional Dictionary Network for Single Image Deraining

As a common weather, rain streaks adversely degrade the image quality. H...

Please sign up or login with your details

Forgot password? Click here to reset