Affect-DML: Context-Aware One-Shot Recognition of Human Affect using Deep Metric Learning

by   Kunyu Peng, et al.

Human affect recognition is a well-established research area with numerous applications, e.g., in psychological care, but existing methods assume that all emotions-of-interest are given a priori as annotated training examples. However, the rising granularity and refinements of the human emotional spectrum through novel psychological theories and the increased consideration of emotions in context brings considerable pressure to data collection and labeling work. In this paper, we conceptualize one-shot recognition of emotions in context – a new problem aimed at recognizing human affect states in finer particle level from a single support sample. To address this challenging task, we follow the deep metric learning paradigm and introduce a multi-modal emotion embedding approach which minimizes the distance of the same-emotion embeddings by leveraging complementary information of human appearance and the semantic scene context obtained through a semantic segmentation network. All streams of our context-aware model are optimized jointly using weighted triplet loss and weighted cross entropy loss. We conduct thorough experiments on both, categorical and numerical emotion recognition tasks of the Emotic dataset adapted to our one-shot recognition problem, revealing that categorizing human affect from a single example is a hard task. Still, all variants of our model clearly outperform the random baseline, while leveraging the semantic scene context consistently improves the learnt representations, setting state-of-the-art results in one-shot emotion recognition. To foster research of more universal representations of human affect states, we will make our benchmark and models publicly available to the community under


page 1

page 2

page 5


Context-Aware Emotion Recognition Networks

Traditional techniques for emotion recognition have focused on the facia...

Context Based Emotion Recognition using EMOTIC Dataset

In our everyday lives and social interactions we often try to perceive t...

End-to-end Triplet Loss based Emotion Embedding System for Speech Emotion Recognition

In this paper, an end-to-end neural embedding system based on triplet lo...

Using Emotion Embeddings to Transfer Knowledge Between Emotions, Languages, and Annotation Formats

The need for emotional inference from text continues to diversify as mor...

Cluster-Level Contrastive Learning for Emotion Recognition in Conversations

A key challenge for Emotion Recognition in Conversations (ERC) is to dis...

EmotiCon: Context-Aware Multimodal Emotion Recognition using Frege's Principle

We present EmotiCon, a learning-based algorithm for context-aware percei...

Few-shot Learning in Emotion Recognition of Spontaneous Speech Using a Siamese Neural Network with Adaptive Sample Pair Formation

Speech-based machine learning (ML) has been heralded as a promising solu...

Please sign up or login with your details

Forgot password? Click here to reset