ABN: Agent-Aware Boundary Networks for Temporal Action Proposal Generation

by   Khoa Vo, et al.

Temporal action proposal generation (TAPG) aims to estimate temporal intervals of actions in untrimmed videos, which is a challenging yet plays an important role in many tasks of video analysis and understanding. Despite the great achievement in TAPG, most existing works ignore the human perception of interaction between agents and the surrounding environment by applying a deep learning model as a black-box to the untrimmed videos to extract video visual representation. Therefore, it is beneficial and potentially improve the performance of TAPG if we can capture these interactions between agents and the environment. In this paper, we propose a novel framework named Agent-Aware Boundary Network (ABN), which consists of two sub-networks (i) an Agent-Aware Representation Network to obtain both agent-agent and agents-environment relationships in the video representation, and (ii) a Boundary Generation Network to estimate the confidence score of temporal intervals. In the Agent-Aware Representation Network, the interactions between agents are expressed through local pathway, which operates at a local level to focus on the motions of agents whereas the overall perception of the surroundings are expressed through global pathway, which operates at a global level to perceive the effects of agents-environment. Comprehensive evaluations on 20-action THUMOS-14 and 200-action ActivityNet-1.3 datasets with different backbone networks (i.e C3D, SlowFast and Two-Stream) show that our proposed ABN robustly outperforms state-of-the-art methods regardless of the employed backbone network on TAPG. We further examine the proposal quality by leveraging proposals generated by our method onto temporal action detection (TAD) frameworks and evaluate their detection performances. The source code can be found in this URL https://github.com/vhvkhoa/TAPG-AgentEnvNetwork.git.


page 2

page 5

page 7

page 8

page 9

page 10

page 13

page 14


Agent-Environment Network for Temporal Action Proposal Generation

Temporal action proposal generation is an essential and challenging task...

Contextual Explainable Video Representation: Human Perception-based Understanding

Video understanding is a growing field and a subject of intense research...

AEI: Actors-Environment Interaction with Adaptive Attention for Temporal Action Proposals Generation

Humans typically perceive the establishment of an action in a video thro...

AOE-Net: Entities Interactions Modeling with Adaptive Attention Mechanism for Temporal Action Proposals Generation

Temporal action proposal generation (TAPG) is a challenging task, which ...

Fast Learning of Temporal Action Proposal via Dense Boundary Generator

Generating temporal action proposals remains a very challenging problem,...

Relation-Aware Pyramid Network (RapNet) for temporal action proposal

In this technical report, we describe our solution to temporal action pr...

A Self-Adaptive Proposal Model for Temporal Action Detection based on Reinforcement Learning

Existing action detection algorithms usually generate action proposals t...

Please sign up or login with your details

Forgot password? Click here to reset