Cross-validation: what does it estimate and how well does it do it?

04/01/2021
by   Stephen Bates, et al.
33

Cross-validation is a widely-used technique to estimate prediction error, but its behavior is complex and not fully understood. Ideally, one would like to think that cross-validation estimates the prediction error for the model at hand, fit to the training data. We prove that this is not the case for the linear model fit by ordinary least squares; rather it estimates the average prediction error of models fit on other unseen training sets drawn from the same population. We further show that this phenomenon occurs for most popular estimates of prediction error, including data splitting, bootstrapping, and Mallow's Cp. Next, the standard confidence intervals for prediction error derived from cross-validation may have coverage far below the desired level. Because each data point is used for both training and testing, there are correlations among the measured accuracies for each fold, and so the usual estimate of variance is too small. We introduce a nested cross-validation scheme to estimate this variance more accurately, and show empirically that this modification leads to intervals with approximately correct coverage in many examples where traditional cross-validation intervals fail. Lastly, our analysis also shows that when producing confidence intervals for prediction accuracy with simple data splitting, one should not re-fit the model on the combined data, since this invalidates the confidence intervals.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
01/26/2022

Confidence intervals for the Cox model test error from cross-validation

Cross-validation (CV) is one of the most widely used techniques in stati...
research
07/01/2023

Bootstrapping the Cross-Validation Estimate

Cross-validation is a widely used technique for evaluating the performan...
research
07/24/2020

Cross-validation Confidence Intervals for Test Error

This work develops central limit theorems for cross-validation and consi...
research
02/10/2023

The out-of-sample R^2: estimation and inference

Out-of-sample prediction is the acid test of predictive models, yet an i...
research
05/08/2019

Predictive inference with the jackknife+

This paper introduces the jackknife+, which is a novel method for constr...
research
11/19/2016

A Bayesian approach to type-specific conic fitting

A perturbative approach is used to quantify the effect of noise in data ...
research
06/09/2021

Robust Prediction Interval estimation for Gaussian Processes by Cross-Validation method

Probabilistic regression models typically use the Maximum Likelihood Est...

Please sign up or login with your details

Forgot password? Click here to reset