Universality laws for Gaussian mixtures in generalized linear models

02/17/2023

∙

by   Yatin Dandi, et al.

∙

1

∙

share

Let (x_i, y_i)_i=1,…,n denote independent samples from a general mixture distribution ∑_c∈𝒞ρ_cP_c^x, and consider the hypothesis class of generalized linear models ŷ = F(Θ^⊤x). In this work, we investigate the asymptotic joint statistics of the family of generalized linear estimators (Θ_1, …, Θ_M) obtained either from (a) minimizing an empirical risk R̂_n(Θ;X,y) or (b) sampling from the associated Gibbs measure exp(-β n R̂_n(Θ;X,y)). Our main contribution is to characterize under which conditions the asymptotic joint statistics of this family depends (on a weak sense) only on the means and covariances of the class conditional features distribution P_c^x. In particular, this allow us to prove the universality of different quantities of interest, such as the training and generalization errors, redeeming a recent line of work in high-dimensional statistics working under the Gaussian mixture hypothesis. Finally, we discuss the applications of our results to different machine learning tasks of interest, such as ensembling and uncertainty