Face Behavior à la carte: Expressions, Affect and Action Units in a Single Network

by   Dimitrios Kollias, et al.

Automatic facial behavior analysis has a long history of studies in the intersection of computer vision, physiology and psychology. However it is only recently, with the collection of large-scale datasets and powerful machine learning methods such as deep neural networks, that automatic facial behavior analysis started to thrive. Three of its iconic tasks are automatic recognition of basic expressions (e.g. happy, sad, surprised), estimation of continuous emotions (e.g., valence and arousal), and detection of facial action units (activations of e.g. upper/inner eyebrows, nose wrinkles). Up until now these tasks have been mostly studied independently collecting a dataset for the task. We present the first and the largest study of all facial behaviour tasks learned jointly in a single multi-task, multi-domain and multi-label network, which we call FaceBehaviorNet. For this we utilize all publicly available datasets in the community (around 5M images) that study facial behaviour tasks in-the-wild. We demonstrate that training jointly an end-to-end network for all tasks has consistently better performance than training each of the single-task networks. Furthermore, we propose two simple strategies for coupling the tasks during training, co-annotation and distribution matching, and show the advantages of this approach. Finally we show that FaceBehaviorNet has learned features that encapsulate all aspects of facial behaviour, and can be successfully applied to perform tasks (compound emotion recognition) beyond the ones that it has been trained in a zero- and few-shot learning setting.


Distribution Matching for Heterogeneous Multi-Task Learning: a Large-scale Face Study

Multi-Task Learning has emerged as a methodology in which multiple tasks...

RAF-AU Database: In-the-Wild Facial Expressions with Subjective Emotion Judgement and Objective AU Annotations

Much of the work on automatic facial expression recognition relies on da...

Multi-Task Transformer with uncertainty modelling for Face Based Affective Computing

Face based affective computing consists in detecting emotions from face ...

Automatic Detection of Sentimentality from Facial Expressions

Emotion recognition has received considerable attention from the Compute...

Expression, Affect, Action Unit Recognition: Aff-Wild2, Multi-Task Learning and ArcFace

Affective computing has been largely limited in terms of available data ...

Affective Behaviour Analysis Using Pretrained Model with Facial Priori

Affective behaviour analysis has aroused researchers' attention due to i...

SchiNet: Automatic Estimation of Symptoms of Schizophrenia from Facial Behaviour Analysis

Patients with schizophrenia often display impairments in the expression ...

Please sign up or login with your details

Forgot password? Click here to reset