Re^2TAL: Rewiring Pretrained Video Backbones for Reversible Temporal Action Localization

11/25/2022
by   Chen Zhao, et al.
0

Temporal action localization (TAL) requires long-form reasoning to predict actions of various lengths and complex content. Given limited GPU memory, training TAL end-to-end on such long-form videos (i.e., from videos to predictions) is a significant challenge. Most methods can only train on pre-extracted features without optimizing them for the localization problem, consequently limiting localization performance. In this work, to extend the potential in TAL networks, we propose a novel end-to-end method Re2TAL, which rewires pretrained video backbones for reversible TAL. Re2TAL builds a backbone with reversible modules, where the input can be recovered from the output such that the bulky intermediate activations can be cleared from memory during training. Instead of designing one single type of reversible module, we propose a network rewiring mechanism, to transform any module with a residual connection to a reversible module without changing any parameters. This provides two benefits: (1) a large variety of reversible networks are easily obtained from existing and even future model designs, and (2) the reversible models require much less training effort as they reuse the pre-trained parameters of their original non-reversible versions. Re2TAL reaches 37.01 average mAP, a new state-of-the-art record on ActivityNet-v1.3, and mAP 64.9 at tIoU=0.5 on THUMOS-14 without using optimal flow.

READ FULL TEXT
research
03/28/2021

Low-Fidelity End-to-End Video Encoder Pre-training for Temporal Action Localization

Temporal action localization (TAL) is a fundamental yet challenging task...
research
04/26/2022

Contrastive Language-Action Pre-training for Temporal Localization

Long-form video understanding requires designing approaches that are abl...
research
04/04/2022

TALLFormer: Temporal Action Localization with Long-memory Transformer

Most modern approaches in temporal action localization divide this probl...
research
01/08/2018

Adaptive Reversible Watermarking Based on Linear Prediction for Medical Videos

Reversible video watermarking can guarantee that the original watermark ...
research
06/01/2023

Make Your Pre-trained Model Reversible: From Parameter to Memory Efficient Fine-Tuning

Parameter-efficient fine-tuning (PEFT) of pre-trained language models (P...
research
06/28/2022

RevBiFPN: The Fully Reversible Bidirectional Feature Pyramid Network

This work introduces the RevSilo, the first reversible module for bidire...
research
11/11/2022

Soft-Landing Strategy for Alleviating the Task Discrepancy Problem in Temporal Action Localization Tasks

Temporal Action Localization (TAL) methods typically operate on top of f...

Please sign up or login with your details

Forgot password? Click here to reset