On the marginal likelihood and cross-validation

05/21/2019
by   Edwin Fong, et al.
University of Oxford
6

In Bayesian statistics, the marginal likelihood, also known as the evidence, is used to evaluate model fit as it quantifies the joint probability of the data under the prior. In contrast, non-Bayesian models are typically compared using cross-validation on held-out data, either through k-fold partitioning or leave-p-out subsampling. We show that the marginal likelihood is formally equivalent to exhaustive leave-p-out cross-validation averaged over all values of p and all held-out test sets when using the log posterior predictive probability as the scoring rule. Moreover, the log posterior predictive is the only coherent scoring rule under data exchangeability. This offers new insight into the marginal likelihood and cross-validation and highlights the potential sensitivity of the marginal likelihood to the setting of the prior. We suggest an alternative approach using aggregate cross-validation following a preparatory training phase. Our work has connections to prequential analysis and intrinsic Bayes factors but is motivated through a different course.

READ FULL TEXT

page 1

page 2

page 3

page 4

08/23/2019

A relation between log-likelihood and cross-validation log-scores

It is shown that the log-likelihood of a hypothesis or model given some ...
06/11/2022

Mathematical Theory of Bayesian Statistics for Unknown Information Source

In statistical inference, uncertainty is unknown and all models are wron...
02/12/2021

Efficient Selection Between Hierarchical Cognitive Models: Cross-validation With Variational Bayes

Model comparison is the cornerstone of theoretical progress in psycholog...
11/18/2022

Prediction scoring of data-driven discoveries for reproducible research

Predictive modeling uncovers knowledge and insights regarding a hypothes...
07/07/2008

Catching Up Faster by Switching Sooner: A Prequential Solution to the AIC-BIC Dilemma

Bayesian model averaging, model selection and its approximations such as...

Please sign up or login with your details

Forgot password? Click here to reset