C2F-TCN: A Framework for Semi and Fully Supervised Temporal Action Segmentation

by   Dipika Singhania, et al.

Temporal action segmentation tags action labels for every frame in an input untrimmed video containing multiple actions in a sequence. For the task of temporal action segmentation, we propose an encoder-decoder-style architecture named C2F-TCN featuring a "coarse-to-fine" ensemble of decoder outputs. The C2F-TCN framework is enhanced with a novel model agnostic temporal feature augmentation strategy formed by the computationally inexpensive strategy of the stochastic max-pooling of segments. It produces more accurate and well-calibrated supervised results on three benchmark action segmentation datasets. We show that the architecture is flexible for both supervised and representation learning. In line with this, we present a novel unsupervised way to learn frame-wise representation from C2F-TCN. Our unsupervised learning approach hinges on the clustering capabilities of the input features and the formation of multi-resolution features from the decoder's implicit structure. Further, we provide the first semi-supervised temporal action segmentation results by merging representation learning with conventional supervised learning. Our semi-supervised learning scheme, called “Iterative-Contrastive-Classify (ICC)”, progressively improves in performance with more labeled data. The ICC semi-supervised learning in C2F-TCN, with 40 labeled videos, performs similar to fully supervised counterparts.


page 3

page 4

page 5

page 7

page 18

page 20


Iterative Frame-Level Representation Learning And Classification For Semi-Supervised Temporal Action Segmentation

Temporal action segmentation classifies the action of each frame in (lon...

Coarse to Fine Multi-Resolution Temporal Convolutional Network

Temporal convolutional networks (TCNs) are a commonly used architecture ...

Unified Fully and Timestamp Supervised Temporal Action Segmentation via Sequence to Sequence Translation

This paper introduces a unified framework for video action segmentation ...

Leveraging Action Affinity and Continuity for Semi-supervised Temporal Action Segmentation

We present a semi-supervised learning approach to the temporal action se...

Learning Morphological Feature Perturbations for Calibrated Semi-Supervised Segmentation

We propose MisMatch, a novel consistency-driven semi-supervised segmenta...

Ensemble Manifold Segmentation for Model Distillation and Semi-supervised Learning

Manifold theory has been the central concept of many learning methods. H...

Incorporating Temporal Prior from Motion Flow for Instrument Segmentation in Minimally Invasive Surgery Video

Automatic instrument segmentation in video is an essentially fundamental...

Please sign up or login with your details

Forgot password? Click here to reset