AU R-CNN: Encoding Expert Prior Knowledge into R-CNN for Action Unit Detection

by   Chen Ma, et al.

Modeling action units (AUs) on human faces is challenging because various AUs cause subtle facial appearance changes over various regions at different scales. Current works have attempted to recognize AUs by emphasizing important regions. However, the incorporation of prior knowledge into region definition remains under-exploited, and current AU detection systems do not use regional convolutional neural networks (R-CNN) with expert prior knowledge to directly focus on AU-related regions adaptively. By incorporating expert prior knowledge, we propose a novel R-CNN based model named AU R-CNN. The proposed solution offers two main contributions: (1) AU R-CNN directly observes different facial regions, where various AUs are located. Expert prior knowledge is encoded in the region and the RoI-level label definition. This design produces considerably better detection performance than do existing approaches. (2) We also integrate various dynamic models (including convolutional long short-term memory, two stream network, conditional random field, and temporal action localization network) into AU R-CNN and then investigate and analyze the reason behind the performance of dynamic models. Experiment results demonstrate that only static RGB image information and no optical flow-based AU R-CNN surpasses the one fused with dynamic models. AU R-CNN is also superior to traditional CNNs that use the same backbone on varying image resolutions. State-of-the-art recognition performance of AU detection is achieved. The complete network is end-to-end trainable. Experiments on BP4D and DISFA datasets show the effectiveness of our approach. Code will be made available.


page 2

page 4

page 5

page 6

page 7

page 8

page 14


Multi-Stream Single Shot Spatial-Temporal Action Detection

We present a 3D Convolutional Neural Networks (CNNs) based single shot d...

A Deep Learning Perspective on the Origin of Facial Expressions

Facial expressions play a significant role in human communication and be...

Face Animation with Multiple Source Images

Face animation has received a lot of attention from researchers in recen...

LoRRaL: Facial Action Unit Detection Based on Local Region Relation Learning

End-to-end convolution representation learning has been proved to be ver...

Modeling Spatial and Temporal Cues for Multi-label Facial Action Unit Detection

Facial action units (AUs) are essential to decode human facial expressio...

Action Unit Detection with Region Adaptation, Multi-labeling Learning and Optimal Temporal Fusing

Action Unit (AU) detection becomes essential for facial analysis. Many p...

Toward Metrics for Differentiating Out-of-Distribution Sets

Vanilla CNNs, as uncalibrated classifiers, suffer from classifying out-o...

Please sign up or login with your details

Forgot password? Click here to reset