Second order Stein: SURE for SURE and other applications in high-dimensional inference
share
Stein's formula states that a random variable of the form z^ f(z) - div f(z) is mean-zero for all functions f with integrable gradient. Here, div f is the divergence of the function f and z is a standard normal vector. A Second Order Stein formula is proposed to characterize the variance of such random variables. In the Gaussian sequence model, a remarkable consequence of Stein's formula is Stein's Unbiased Risk Estimate (SURE) of the mean square risk of almost any given estimator μ̂ for the unknown mean vector. A first application of the Second Order Stein formula is an Unbiased Risk Estimate of the risk of SURE itself (SURE for SURE): a simple unbiased estimate provides information about the squared distance between SURE and the squared estimation error of μ̂. SURE for SURE has a simple form and can be computed explicitly for differentiable μ̂, for example the Lasso and the Elastic Net. Other applications of the Second Order Stein formula are provided in high-dimensional regression. This includes novel bounds on the variance of the size of the model selected by the Lasso, and a general semi-parametric scheme to de-bias an almost differentiable initial estimator in order to estimate a low-dimensional projection of the unknown regression coefficient vector.
READ FULL TEXT