Robust Facial Expression Recognition with Convolutional Visual Transformers

by   Fuyan Ma, et al.

Facial Expression Recognition (FER) in the wild is extremely challenging due to occlusions, variant head poses, face deformation and motion blur under unconstrained conditions. Although substantial progresses have been made in automatic FER in the past few decades, previous studies are mainly designed for lab-controlled FER. Real-world occlusions, variant head poses and other issues definitely increase the difficulty of FER on account of these information-deficient regions and complex backgrounds. Different from previous pure CNNs based methods, we argue that it is feasible and practical to translate facial images into sequences of visual words and perform expression recognition from a global perspective. Therefore, we propose Convolutional Visual Transformers to tackle FER in the wild by two main steps. First, we propose an attentional selective fusion (ASF) for leveraging the feature maps generated by two-branch CNNs. The ASF captures discriminative information by fusing multiple features with global-local attention. The fused feature maps are then flattened and projected into sequences of visual words. Second, inspired by the success of Transformers in natural language processing, we propose to model relationships between these visual words with global self-attention. The proposed method are evaluated on three public in-the-wild facial expression datasets (RAF-DB, FERPlus and AffectNet). Under the same settings, extensive experiments demonstrate that our method shows superior performance over other methods, setting new state of the art on RAF-DB with 88.14 cross-dataset evaluation on CK+ show the generalization capability of the proposed method.


page 1

page 9


Region Attention Networks for Pose and Occlusion Robust Facial Expression Recognition

Occlusion and pose variations, which can change facial appearance signif...

Distract Your Attention: Multi-head Cross Attention Network for Facial Expression Recognition

We present a novel facial expression recognition network, called Distrac...

FaceTopoNet: Facial Expression Recognition using Face Topology Learning

Prior work has shown that the order in which different components of the...

Your "Attention" Deserves Attention: A Self-Diversified Multi-Channel Attention for Facial Action Analysis

Visual attention has been extensively studied for learning fine-grained ...

PAtt-Lite: Lightweight Patch and Attention MobileNet for Challenging Facial Expression Recognition

Facial Expression Recognition (FER) is a machine learning problem that d...

Lossless Attention in Convolutional Networks for Facial Expression Recognition in the Wild

Unlike the constraint frontal face condition, faces in the wild have var...

Landmark Guidance Independent Spatio-channel Attention and Complementary Context Information based Facial Expression Recognition

A recent trend to recognize facial expressions in the real-world scenari...

Please sign up or login with your details

Forgot password? Click here to reset