Variational Inference based on Robust Divergences

by   Futoshi Futami, et al.

Robustness to outliers is a central issue in real-world machine learning applications. While replacing a model to a heavy-tailed one (e.g., from Gaussian to Student-t) is a standard approach for robustification, it can only be applied to simple models. In this paper, based on Zellner's optimization and variational formulation of Bayesian inference, we propose an outlier-robust pseudo-Bayesian variational method by replacing the Kullback-Leibler divergence used for data fitting to a robust divergence such as the beta and gamma-divergences. An advantage of our approach is that complex models such as deep networks can be handled. We theoretically prove that, for deep networks with ReLU activation functions, the influence function in our proposed method is bounded, while it is unbounded in the ordinary variational inference. This implies that our proposed method is robust to both of input and output outliers, while the ordinary variational method is not. We experimentally demonstrate that our robust variational method outperforms ordinary variational inference in regression and classification with deep networks.


page 1

page 2

page 3

page 4


Alpha-Beta Divergence For Variational Inference

This paper introduces a variational approximation framework using direct...

β-Cores: Robust Large-Scale Bayesian Data Summarization in the Presence of Outliers

Modern machine learning applications should be able to address the intri...

Exploring Variational Deep Q Networks

This study provides both analysis and a refined, research-ready implemen...

Generalized Variational Inference

This paper introduces a generalized representation of Bayesian inference...

Variational Inference: Posterior Threshold Improves Network Clustering Accuracy in Sparse Regimes

Variational inference has been widely used in machine learning literatur...

Good Initializations of Variational Bayes for Deep Models

Stochastic variational inference is an established way to carry out appr...

Variational Learning for the Inverted Beta-Liouville Mixture Model and Its Application to Text Categorization

The finite invert Beta-Liouville mixture model (IBLMM) has recently gain...

Please sign up or login with your details

Forgot password? Click here to reset