Multi-view Multi-label Fine-grained Emotion Decoding from Human Brain Activity

by   Kaicheng Fu, et al.

Decoding emotional states from human brain activity plays an important role in brain-computer interfaces. Existing emotion decoding methods still have two main limitations: one is only decoding a single emotion category from a brain activity pattern and the decoded emotion categories are coarse-grained, which is inconsistent with the complex emotional expression of human; the other is ignoring the discrepancy of emotion expression between the left and right hemispheres of human brain. In this paper, we propose a novel multi-view multi-label hybrid model for fine-grained emotion decoding (up to 80 emotion categories) which can learn the expressive neural representations and predicting multiple emotional states simultaneously. Specifically, the generative component of our hybrid model is parametrized by a multi-view variational auto-encoder, in which we regard the brain activity of left and right hemispheres and their difference as three distinct views, and use the product of expert mechanism in its inference network. The discriminative component of our hybrid model is implemented by a multi-label classification network with an asymmetric focal loss. For more accurate emotion decoding, we first adopt a label-aware module for emotion-specific neural representations learning and then model the dependency of emotional states by a masked self-attention mechanism. Extensive experiments on two visually evoked emotional datasets show the superiority of our method.


Basic and Depression Specific Emotion Identification in Tweets: Multi-label Classification Experiments

In this paper, we present empirical analysis on basic and depression spe...

A Deep-Learning-Based Neural Decoding Framework for Emotional Brain-Computer Interfaces

Reading emotions precisely from segments of neural activity is crucial f...

Emotional Brain State Classification on fMRI Data Using Deep Residual and Convolutional Networks

The goal of emotional brain state classification on functional MRI (fMRI...

Fine-grained Emotion Strength Transfer, Control and Prediction for Emotional Speech Synthesis

This paper proposes a unified model to conduct emotion transfer, control...

Semi-supervised Bayesian Deep Multi-modal Emotion Recognition

In emotion recognition, it is difficult to recognize human's emotional s...

Joint Emotion Label Space Modelling for Affect Lexica

Emotion lexica are commonly used resources to combat data poverty in aut...

Interpretability of Fine-grained Classification of Sadness and Depression

While sadness is a human emotion that people experience at certain times...

Please sign up or login with your details

Forgot password? Click here to reset