Random weighting to approximate posterior inference in LASSO regression
share
We consider a general-purpose approximation approach to Bayesian inference in which repeated optimization of a randomized objective function provides surrogate samples from the joint posterior distribution. In the context of LASSO regression, we repeatedly assign independently-drawn standard-exponential random weights to terms in the objective function, and optimize to obtain the surrogate samples. We establish the asymptotic properties of this method under different regularization parameters λ_n. In particular, if λ_n = o(√(n)), then the random-weighting (weighted bootstrap) samples are equivalent (up to the first order) to the Bayesian posterior samples. If λ_n = O( n^c ) for some 1/2 < c < 1, then these samples achieve conditional model selection consistency. We also establish the asymptotic properties of the random-weighting method when weights are drawn from other distributions, and also if weights are assigned to the LASSO penalty terms.
READ FULL TEXT