Estimating the standard error of cross-Validation-Based estimators of classifier performance

Understand

First, we analyze the variance of the Cross Validation (CV)-based estimators used for estimating the performance of classification rules.

• Second, we propose a novel estimator to estimate this variance using the Influence Function (IF) approach that had been used previously very successfully to estimate the variance of the bootstrap-based estimators.
• The motivation for this research is that, as the best of our knowledge, the literature lacks a rigorous method for estimating the variance of the CV-based estimators.
• What is available is a set of ad-hoc procedures that have no mathematical foundation since they ignore the covariance structure among dependent random variables.