Fetching the paper…

Reading the bibliography…

2019

Cross-Validation, Risk Estimation, and Model Selection

Wager, Stefan

Original
PDF
Post on X

Understand

Cross-validation is a popular non-parametric method for evaluating the accuracy of a predictive rule.

The usefulness of cross-validation depends on the task we want to employ it for.
In this note, I discuss a simple non-parametric setting, and find that cross-validation is asymptotically uninformative about the expected test error of any given predictive rule, but allows for asymptotically consistent model selection.
The reason for this phenomenon is that the leading-order error term of cross-validation doesn't depend on the model being evaluated, and so cancels out when we compare two models.

Cross-Validation, Risk Estimation, and Model Selection · Around

Built on

Some comments on Cp
Colin L Mallows · 1973
Earlier work this paper cites.
Estimating the error rate of a prediction rule: improvement on cross-validation
Bradley Efron · 1983
Earlier work this paper cites.
Random forests
Leo Breiman · 2001
Earlier work this paper cites.
The estimation of prediction error: covariance penalties and cross-validation
Bradley Efron · 2004
Earlier work this paper cites.

Similar

Consistency of cross validation for comparing regression procedures
Original
Yuhong Yang · 2007
Cited alongside, same era.
The Elements of Statistical Learning: Data Mining, Inference, and Prediction
Trevor Hastie, Robert Tibshirani, and Jerome Friedman · 2009
Cited alongside, same era.
A survey of cross-validation procedures for model selection
Sylvain Arlot and Alain Celisse · 2010
Cited alongside, same era.
Xgboost: A scalable tree boosting system
Tianqi Chen and Carlos Guestrin · 2016
Cited alongside, same era.

Then

On cross-validated lasso
Original
Denis Chetverikov, Zhipeng Liao, and Victor Chernozhukov · 2016
Later among the works it cites.
Quasi-oracle estimation of heterogeneous treatment effects
Original
Xinkun Nie and Stefan Wager · 2017
Later among the works it cites.
Generalized random forests
Susan Athey, Julie Tibshirani, and Stefan Wager · 2019
Closest in time.
From fixed-X to random-X regression: Bias-variance decompositions, covariance penalties, and prediction error estimation
Saharon Rosset and Ryan J Tibshirani · 2019
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…