General Debiasing for Multimodal Sentiment Analysis

by   Teng Sun, et al.

Existing work on Multimodal Sentiment Analysis (MSA) utilizes multimodal information for prediction yet unavoidably suffers from fitting the spurious correlations between multimodal features and sentiment labels. For example, if most videos with a blue background have positive labels in a dataset, the model will rely on such correlations for prediction, while “blue background” is not a sentiment-related feature. To address this problem, we define a general debiasing MSA task, which aims to enhance the Out-Of-Distribution (OOD) generalization ability of MSA models by reducing their reliance on spurious correlations. To this end, we propose a general debiasing framework based on Inverse Probability Weighting (IPW), which adaptively assigns small weights to the samples with larger bias i.e., the severer spurious correlations). The key to this debiasing framework is to estimate the bias of each sample, which is achieved by two steps: 1) disentangling the robust features and biased features in each modality, and 2) utilizing the biased features to estimate the bias. Finally, we employ IPW to reduce the effects of large-biased samples, facilitating robust feature learning for sentiment prediction. To examine the model's generalization ability, we keep the original testing sets on two benchmarks and additionally construct multiple unimodal and multimodal OOD testing sets. The empirical results demonstrate the superior generalization ability of our proposed framework. We have released the code and data to facilitate the reproduction.


page 2

page 8


Counterfactual Reasoning for Out-of-distribution Multimodal Sentiment Analysis

Existing studies on multimodal sentiment analysis heavily rely on textua...

M-SENA: An Integrated Platform for Multimodal Sentiment Analysis

M-SENA is an open-sourced platform for Multimodal Sentiment Analysis. It...

CubeMLP: An MLP-based Model for Multimodal Sentiment Analysis and Depression Estimation

Multimodal sentiment analysis and depression estimation are two importan...

EiHi Net: Out-of-Distribution Generalization Paradigm

This paper develops a new EiHi net to solve the out-of-distribution (OoD...

CausalAPM: Generalizable Literal Disentanglement for NLU Debiasing

Dataset bias, i.e., the over-reliance on dataset-specific literal heuris...

Learning Robust Representation for Joint Grading of Ophthalmic Diseases via Adaptive Curriculum and Feature Disentanglement

Diabetic retinopathy (DR) and diabetic macular edema (DME) are leading c...

Efficient Multimodal Transformer with Dual-Level Feature Restoration for Robust Multimodal Sentiment Analysis

With the proliferation of user-generated online videos, Multimodal Senti...

Please sign up or login with your details

Forgot password? Click here to reset