Learning a Fast 3D Spectral Approach to Object Segmentation and Tracking over Space and Time

by   Elena Burceanu, et al.

We pose video object segmentation as spectral graph clustering in space and time, with one graph node for each pixel and edges forming local space-time neighborhoods. We claim that the strongest cluster in this video graph represents the salient object. We start by introducing a novel and efficient method based on 3D filtering for approximating the spectral solution, as the principal eigenvector of the graph's adjacency matrix, without explicitly building the matrix. This key property allows us to have a fast parallel implementation on GPU, orders of magnitude faster than classical approaches for computing the eigenvector. Our motivation for a spectral space-time clustering approach, unique in video semantic segmentation literature, is that such clustering is dedicated to preserving object consistency over time, which we evaluate using our novel segmentation consistency measure. Further on, we show how to efficiently learn the solution over multiple input feature channels. Finally, we extend the formulation of our approach beyond the segmentation task, into the realm of object tracking. In extensive experiments we show significant improvements over top methods, as well as over powerful ensembles that combine them, achieving state-of-the-art on multiple benchmarks, both for tracking and segmentation.


page 2

page 11

page 13

page 16

page 17

page 21

page 23

page 26


A Spectral Approach to Unsupervised Object Segmentation in Video

We formulate object segmentation in video as a graph partitioning proble...

SFTrack++: A Fast Learnable Spectral Segmentation Approach for Space-Time Consistent Tracking

We propose an object tracking method, SFTrack++, that smoothly learns to...

Efficient Activity Detection in Untrimmed Video with Max-Subgraph Search

We propose an efficient approach for activity detection in video that un...

Iterative Knowledge Exchange Between Deep Learning and Space-Time Spectral Clustering for Unsupervised Segmentation in Videos

We propose a dual system for unsupervised object segmentation in video, ...

Spacetime Graph Optimization for Video Object Segmentation

In this paper we address the challenging task of object discovery and se...

Hybrid Diffusion: Spectral-Temporal Graph Filtering for Manifold Ranking

State of the art image retrieval performance is achieved with CNN featur...

A fast-convolution based space-time Chebyshev spectral method for peridynamic models

Peridynamics is a nonlocal generalization of continuum mechanics theory ...

Please sign up or login with your details

Forgot password? Click here to reset