Fast video object segmentation with Spatio-Temporal GANs

by   Sergi Caelles, et al.

Learning descriptive spatio-temporal object models from data is paramount for the task of semi-supervised video object segmentation. Most existing approaches mainly rely on models that estimate the segmentation mask based on a reference mask at the first frame (aided sometimes by optical flow or the previous mask). These models, however, are prone to fail under rapid appearance changes or occlusions due to their limitations in modelling the temporal component. On the other hand, very recently, other approaches learned long-term features using a convolutional LSTM to leverage the information from all previous video frames. Even though these models achieve better temporal representations, they still have to be fine-tuned for every new video sequence. In this paper, we present an intermediate solution and devise a novel GAN architecture, FaSTGAN, to learn spatio-temporal object models over finite temporal windows. To achieve this, we concentrate all the heavy computational load to the training phase with two critics that enforce spatial and temporal mask consistency over the last K frames. Then at test time, we only use a relatively light regressor, which reduces the inference time considerably. As a result, our approach combines a high resiliency to sudden geometric and photometric object changes with efficiency at test time (no need for fine-tuning nor post-processing). We demonstrate that the accuracy of our method is on par with state-of-the-art techniques on the challenging YouTube-VOS and DAVIS datasets, while running at 32 fps, about 4x faster than the closest competitor.


page 1

page 3

page 8


Make One-Shot Video Object Segmentation Efficient Again

Video object segmentation (VOS) describes the task of segmenting a set o...

ReConvNet: Video Object Segmentation with Spatio-Temporal Features Modulation

We introduce ReConvNet, a recurrent convolutional architecture for semi-...

CNN in MRF: Video Object Segmentation via Inference in A CNN-Based Higher-Order Spatio-Temporal MRF

This paper addresses the problem of video object segmentation, where the...

Unsupervised Video Segmentation via Spatio-Temporally Nonlocal Appearance Learning

Video object segmentation is challenging due to the factors like rapidly...

TTVOS: Lightweight Video Object Segmentation with Adaptive Template Attention Module and Temporal Consistency Loss

Semi-supervised video object segmentation (semi-VOS) is widely used in m...

Online Adaptation of Convolutional Neural Networks for Video Object Segmentation

We tackle the task of semi-supervised video object segmentation, i.e. se...

Click Carving: Segmenting Objects in Video with Point Clicks

We present a novel form of interactive video object segmentation where a...

Please sign up or login with your details

Forgot password? Click here to reset