MM-Align: Learning Optimal Transport-based Alignment Dynamics for Fast and Accurate Inference on Missing Modality Sequences

by   Wei Han, et al.

Existing multimodal tasks mostly target at the complete input modality setting, i.e., each modality is either complete or completely missing in both training and test sets. However, the randomly missing situations have still been underexplored. In this paper, we present a novel approach named MM-Align to address the missing-modality inference problem. Concretely, we propose 1) an alignment dynamics learning module based on the theory of optimal transport (OT) for indirect missing data imputation; 2) a denoising training algorithm to simultaneously enhance the imputation results and backbone network performance. Compared with previous methods which devote to reconstructing the missing inputs, MM-Align learns to capture and imitate the alignment dynamics between modality sequences. Results of comprehensive experiments on three datasets covering two multimodal tasks empirically demonstrate that our method can perform more accurate and faster inference and relieve overfitting under various missing conditions.


page 2

page 12


Missing Data Imputation using Optimal Transport

Missing data is a crucial issue when applying machine learning algorithm...

Maximum Likelihood Estimation for Multimodal Learning with Missing Modality

Multimodal learning has achieved great successes in many scenarios. Comp...

Multi-Modality Guidance Network For Missing Modality Inference

Multimodal models have gained significant success in recent years. Stand...

SMIL: Multimodal Learning with Severely Missing Modality

A common assumption in multimodal learning is the completeness of traini...

Are you doing what I say? On modalities alignment in ALFRED

ALFRED is a recently proposed benchmark that requires a model to complet...

Unsupervised domain adaptation with non-stochastic missing data

We consider unsupervised domain adaptation (UDA) for classification prob...

Relaxation Labeling Meets GANs: Solving Jigsaw Puzzles with Missing Borders

This paper proposes JiGAN, a GAN-based method for solving Jigsaw puzzles...

Please sign up or login with your details

Forgot password? Click here to reset