Expression, Affect, Action Unit Recognition: Aff-Wild2, Multi-Task Learning and ArcFace

by   Dimitrios Kollias, et al.

Affective computing has been largely limited in terms of available data resources. The need to collect and annotate diverse in-the-wild datasets has become apparent with the rise of deep learning models, as the default approach to address any computer vision task. Some in-the-wild databases have been recently proposed. However: i) their size is small, ii) they are not audiovisual, iii) only a small part is manually annotated, iv) they contain a small number of subjects, or v) they are not annotated for all main behavior tasks (valence-arousal estimation, action unit detection and basic expression classification). To address these, we substantially extend the largest available in-the-wild database (Aff-Wild) to study continuous emotions such as valence and arousal. Furthermore, we annotate parts of the database with basic expressions and action units. As a consequence, for the first time, this allows the joint study of all three types of behavior states. We call this database Aff-Wild2. We conduct extensive experiments with CNN and CNN-RNN architectures that use visual and audio modalities; these networks are trained on Aff-Wild2 and their performance is then evaluated on 10 publicly available emotion databases. We show that the networks achieve state-of-the-art performance for the emotion recognition tasks. Additionally, we adapt the ArcFace loss function in the emotion recognition context and use it for training two new networks on Aff-Wild2 and then re-train them in a variety of diverse expression recognition databases. The networks are shown to improve the existing state-of-the-art. The database, emotion recognition models and source code are available at


A Multi-Task Learning & Generation Framework: Valence-Arousal, Action Units & Primary Expressions

Over the past few years many research efforts have been devoted to the f...

ABAW: Valence-Arousal Estimation, Expression Recognition, Action Unit Detection Multi-Task Learning Challenges

This paper describes the third Affective Behavior Analysis in-the-wild (...

MAFW: A Large-scale, Multi-modal, Compound Affective Database for Dynamic Facial Expression Recognition in the Wild

Dynamic facial expression recognition (FER) databases provide important ...

Distribution Matching for Heterogeneous Multi-Task Learning: a Large-scale Face Study

Multi-Task Learning has emerged as a methodology in which multiple tasks...

AffWild Net and Aff-Wild Database

Emotions recognition is the task of recognizing people's emotions. Usual...

Face Behavior à la carte: Expressions, Affect and Action Units in a Single Network

Automatic facial behavior analysis has a long history of studies in the ...

Emotion Recognition with Incomplete Labels Using Modified Multi-task Learning Technique

The task of predicting affective information in the wild such as seven b...

Please sign up or login with your details

Forgot password? Click here to reset