Privileged Information for Modeling Affect In The Wild

by   Konstantinos Makantasis, et al.

A key challenge of affective computing research is discovering ways to reliably transfer affect models that are built in the laboratory to real world settings, namely in the wild. The existing gap between in vitro and in vivo affect applications is mainly caused by limitations related to affect sensing including intrusiveness, hardware malfunctions, availability of sensors, but also privacy and security. As a response to these limitations in this paper we are inspired by recent advances in machine learning and introduce the concept of privileged information for operating affect models in the wild. The presence of privileged information enables affect models to be trained across multiple modalities available in a lab setting and ignore modalities that are not available in the wild with no significant drop in their modeling performance. The proposed privileged information framework is tested in a game arousal corpus that contains physiological signals in the form of heart rate and electrodermal activity, game telemetry, and pixels of footage from two dissimilar games that are annotated with arousal traces. By training our arousal models using all modalities (in vitro) and using solely pixels for testing the models (in vivo), we reach levels of accuracy obtained from models that fuse all modalities both for training and testing. The findings of this paper make a decisive step towards realizing affect interaction in the wild.


page 1

page 3

page 5

page 7


From the Lab to the Wild: Affect Modeling via Privileged Information

How can we reliably transfer affect models trained in controlled laborat...

Introducing Representations of Facial Affect in Automated Multimodal Deception Detection

Automated deception detection systems can enhance health, justice, and s...

Supervised Contrastive Learning for Affect Modelling

Affect modeling is viewed, traditionally, as the process of mapping meas...

Towards Intercultural Affect Recognition: Audio-Visual Affect Recognition in the Wild Across Six Cultures

In our multicultural world, affect-aware AI systems that support humans ...

More to Less (M2L): Enhanced Health Recognition in the Wild with Reduced Modality of Wearable Sensors

Accurately recognizing health-related conditions from wearable data is c...

Overprotective Training Environments Fall Short at Testing Time: Let Models Contribute to Their Own Training

Despite important progress, conversational systems often generate dialog...

From Pixels to Affect: A Study on Games and Player Experience

Is it possible to predict the affect of a user just by observing her beh...

Please sign up or login with your details

Forgot password? Click here to reset