A Dive into SAM Prior in Image Restoration

by   Zeyu Xiao, et al.

The goal of image restoration (IR), a fundamental issue in computer vision, is to restore a high-quality (HQ) image from its degraded low-quality (LQ) observation. Multiple HQ solutions may correspond to an LQ input in this poorly posed problem, creating an ambiguous solution space. This motivates the investigation and incorporation of prior knowledge in order to effectively constrain the solution space and enhance the quality of the restored images. In spite of the pervasive use of hand-crafted and learned priors in IR, limited attention has been paid to the incorporation of knowledge from large-scale foundation models. In this paper, we for the first time leverage the prior knowledge of the state-of-the-art segment anything model (SAM) to boost the performance of existing IR networks in an parameter-efficient tuning manner. In particular, the choice of SAM is based on its robustness to image degradations, such that HQ semantic masks can be extracted from it. In order to leverage semantic priors and enhance restoration quality, we propose a lightweight SAM prior tuning (SPT) unit. This plug-and-play component allows us to effectively integrate semantic priors into existing IR networks, resulting in significant improvements in restoration quality. As the only trainable module in our method, the SPT unit has the potential to improve both efficiency and scalability. We demonstrate the effectiveness of the proposed method in enhancing a variety of methods across multiple tasks, such as image super-resolution and color image denoising.


page 1

page 4

page 5

page 7

page 8

page 9


Denoising Prior Driven Deep Neural Network for Image Restoration

Deep neural networks (DNNs) have shown very promising results for variou...

SAM-Deblur: Let Segment Anything Boost Image Deblurring

Image deblurring is a critical task in the field of image restoration, a...

Denoising Diffusion Models for Plug-and-Play Image Restoration

Plug-and-play Image Restoration (IR) has been widely recognized as a fle...

RAMiT: Reciprocal Attention Mixing Transformer for Lightweight Image Restoration

Although many recent works have made advancements in the image restorati...

Zero-Shot Image Restoration Using Denoising Diffusion Null-Space Model

Most existing Image Restoration (IR) models are task-specific, which can...

Image Restoration using Autoencoding Priors

We propose to leverage denoising autoencoder networks as priors to addre...

Image Restoration with Mean-Reverting Stochastic Differential Equations

This paper presents a stochastic differential equation (SDE) approach fo...

Please sign up or login with your details

Forgot password? Click here to reset