Can SAM Boost Video Super-Resolution?

by   Zhihe Lu, et al.

The primary challenge in video super-resolution (VSR) is to handle large motions in the input frames, which makes it difficult to accurately aggregate information from multiple frames. Existing works either adopt deformable convolutions or estimate optical flow as a prior to establish correspondences between frames for the effective alignment and fusion. However, they fail to take into account the valuable semantic information that can greatly enhance it; and flow-based methods heavily rely on the accuracy of a flow estimate model, which may not provide precise flows given two low-resolution frames. In this paper, we investigate a more robust and semantic-aware prior for enhanced VSR by utilizing the Segment Anything Model (SAM), a powerful foundational model that is less susceptible to image degradation. To use the SAM-based prior, we propose a simple yet effective module – SAM-guidEd refinEment Module (SEEM), which can enhance both alignment and fusion procedures by the utilization of semantic information. This light-weight plug-in module is specifically designed to not only leverage the attention mechanism for the generation of semantic-aware feature but also be easily and seamlessly integrated into existing methods. Concretely, we apply our SEEM to two representative methods, EDVR and BasicVSR, resulting in consistently improved performance with minimal implementation effort, on three widely used VSR datasets: Vimeo-90K, REDS and Vid4. More importantly, we found that the proposed SEEM can advance the existing methods in an efficient tuning manner, providing increased flexibility in adjusting the balance between performance and the number of training parameters. Code will be open-source soon.


page 1

page 6

page 7

page 8


FDAN: Flow-guided Deformable Alignment Network for Video Super-Resolution

Most Video Super-Resolution (VSR) methods enhance a video reference fram...

Optical-Flow-Reuse-Based Bidirectional Recurrent Network for Space-Time Video Super-Resolution

In this paper, we consider the task of space-time video super-resolution...

Deep Video Super-Resolution using HR Optical Flow Estimation

Video super-resolution (SR) aims at generating a sequence of high-resolu...

Optical Flow for Video Super-Resolution: A Survey

Video super-resolution is currently one of the most active research topi...

Learning for Video Super-Resolution through HR Optical Flow Estimation

Video super-resolution (SR) aims to generate a sequence of high-resoluti...

BSRT: Improving Burst Super-Resolution with Swin Transformer and Flow-Guided Deformable Alignment

This work addresses the Burst Super-Resolution (BurstSR) task using a ne...

Please sign up or login with your details

Forgot password? Click here to reset