The Theory Behind Overfitting, Cross Validation, Regularization, Bagging, and Boosting: Tutorial

Understand

In this tutorial paper, we first define mean squared error, variance, covariance, and bias of both random variables and classification/predictor models.

• Then, we formulate the true and generalization errors of the model for both training and validation/test instances where we make use of the Stein's Unbiased Risk Estimator (SURE).
• We define overfitting, underfitting, and generalization using the obtained true and generalization errors.
• We introduce cross validation and two well-known examples which are $K$-fold and leave-one-out cross validations.