Learning Physical-Spatio-Temporal Features for Video Shadow Removal

by   Zhihao Chen, et al.

Shadow removal in a single image has received increasing attention in recent years. However, removing shadows over dynamic scenes remains largely under-explored. In this paper, we propose the first data-driven video shadow removal model, termed PSTNet, by exploiting three essential characteristics of video shadows, i.e., physical property, spatio relation, and temporal coherence. Specifically, a dedicated physical branch was established to conduct local illumination estimation, which is more applicable for scenes with complex lighting and textures, and then enhance the physical features via a mask-guided attention strategy. Then, we develop a progressive aggregation module to enhance the spatio and temporal characteristics of features maps, and effectively integrate the three kinds of features. Furthermore, to tackle the lack of datasets of paired shadow videos, we synthesize a dataset (SVSRD-85) with aid of the popular game GTAV by controlling the switch of the shadow renderer. Experiments against 9 state-of-the-art models, including image shadow removers and image/video restoration methods, show that our method improves the best SOTA in terms of RMSE error for the shadow area by 14.7. In addition, we develop a lightweight model adaptation strategy to make our synthetic-driven model effective in real world scenes. The visual comparison on the public SBU-TimeLapse dataset verifies the generalization ability of our model in real scenes.


page 1

page 3

page 4

page 7

page 8

page 9

page 10

page 12


Real-World Video for Zoom Enhancement based on Spatio-Temporal Coupling

In recent years, single-frame image super-resolution (SR) has become mor...

Triple-cooperative Video Shadow Detection

Shadow detection in a single image has received significant research int...

A Survey on Rain Removal from Video and Single Image

Rain streaks might severely degenerate the performance of video/image pr...

PastNet: Introducing Physical Inductive Biases for Spatio-temporal Video Prediction

In this paper, we investigate the challenge of spatio-temporal video pre...

Toward Accurate Person-level Action Recognition in Videos of Crowded Scenes

Detecting and recognizing human action in videos with crowded scenes is ...

Efficient Spatio-Temporal Recurrent Neural Network for Video Deblurring

Real-time video deblurring still remains a challenging task due to the c...

Mask-Guided Image Person Removal with Data Synthesis

As a special case of common object removal, image person removal is play...

Please sign up or login with your details

Forgot password? Click here to reset