Human Action Co-occurrence in Lifestyle Vlogs using Graph Link Prediction

by   Oana Ignat, et al.

We introduce the task of automatic human action co-occurrence identification, i.e., determine whether two human actions can co-occur in the same interval of time. We create and make publicly available the ACE (Action Co-occurrencE) dataset, consisting of a large graph of  12k co-occurring pairs of visual actions and their corresponding video clips. We describe graph link prediction models that leverage visual and textual information to automatically infer if two actions are co-occurring. We show that graphs are particularly well suited to capture relations between human actions, and the learned graph representations are effective for our task and capture novel and relevant information across different data domains. The ACE dataset and the code introduced in this paper are publicly available at


page 14

page 15


WhyAct: Identifying Action Reasons in Lifestyle Vlogs

We aim to automatically identify human action reasons in online videos. ...

An Effective Graph Learning based Approach for Temporal Link Prediction: The First Place of WSDM Cup 2022

Temporal link prediction, as one of the most crucial work in temporal gr...

Link-Backdoor: Backdoor Attack on Link Prediction via Node Injection

Link prediction, inferring the undiscovered or potential links of the gr...

A Benchmark for Structured Procedural Knowledge Extraction from Cooking Videos

Procedural knowledge, which we define as concrete information about the ...

Learning to Refactor Action and Co-occurrence Features for Temporal Action Localization

The main challenge of Temporal Action Localization is to retrieve subtle...

Wiki-CS: A Wikipedia-Based Benchmark for Graph Neural Networks

We present Wiki-CS, a novel dataset derived from Wikipedia for benchmark...

Near Real-Time Data Labeling Using a Depth Sensor for EMG Based Prosthetic Arms

Recognizing sEMG (Surface Electromyography) signals belonging to a parti...

Please sign up or login with your details

Forgot password? Click here to reset