2018

Model Evaluation, Model Selection, and Algorithm Selection in Machine Learning

Raschka, Sebastian

Understand

The correct use of model evaluation, model selection, and algorithm selection techniques is vital in academic machine learning research as well as in many industrial settings.

This article reviews different techniques that can be used for each of these three subtasks and discusses the main advantages and disadvantages of each technique with references to theoretical and empirical studies.
Further, recommendations are given to encourage best yet feasible practices in research and applications of machine learning.
Common methods such as the holdout method for model evaluation and selection are covered, which are not recommended when working with small datasets.

Built on

Approximate statistical tests for comparing supervised classification learning algorithms
Dietterich, T. G. (1998) · 1923
Earlier work this paper cites.
Teoria statistica delle classi e calcolo delle probabilita
Bonferroni, C. (1936) · 1936
Earlier work this paper cites.
Note on the sampling error of the difference between correlated proportions or percentages
McNemar, Q. (1947) · 1947
Earlier work this paper cites.
Note on the “correction for continuity” in testing the significance of the difference between correlated proportions
Edwards, A. L. (1948) · 1948
Earlier work this paper cites.
The comparison of percentages in matched samples
Cochran, W. G. (1950) · 1950
Earlier work this paper cites.
Multiple comparisons among means
Dunn, O. J. (1961) · 1961
Earlier work this paper cites.
Nonparametric standard errors and confidence intervals
Efron, B. (1981) · 1981
Earlier work this paper cites.
Estimating the error rate of a prediction rule: improvement on cross-validation
Efron, B. (1983) · 1983
Earlier work this paper cites.
Classification and regression trees
Breiman, L., Friedman, J., Stone, C. J., and Olshen, R. A. (1984) · 1984
Earlier work this paper cites.
A statistical technique for comparing the accuracies of several classifiers
Looney, S. W. (1988) · 1988
Earlier work this paper cites.
No adjustments are needed for multiple comparisons
Rothman, K. J. (1990) · 1990
Earlier work this paper cites.

Similar

Bootstrap methods: another look at the Jackknife
Efron, B. (1992) · 1992
Cited alongside, same era.
An Introduction to the Bootstrap
Efron, B. and Tibshirani, R. J. (1994) · 1994
Cited alongside, same era.
A study of cross-validation and bootstrap for accuracy estimation and model selection
Kohavi, R. (1995) · 1995
Cited alongside, same era.
Improvements on cross-validation: the .632+ bootstrap method
Efron, B. and Tibshirani, R. (1997) · 1997
Cited alongside, same era.
What’s wrong with bonferroni adjustments
Perneger, T. V. (1998) · 1998
Cited alongside, same era.
Combined 5x2cv F test for comparing supervised classification learning algorithms
Alpaydin, E. (1999) · 1999
Cited alongside, same era.
Assessing model fit by cross-validation
Hawkins, D. M., Basak, S. C., and Mills, D. (2003) · 2003
Cited alongside, same era.
Oligonucleotide microarray for prediction of early intrahepatic recurrence of hepatocellular carcinoma after curative resection
Iizuka, N., Oka, M., Yamada-Okabe, H., Nishida, M., Maeda, Y., Mori, N., Takao, T., Tamesa, T., Tangoku, A., Tabuchi, H., et al. (2003) · 2003
Cited alongside, same era.
No unbiased estimator of the variance of k-fold cross-validation
Bengio, Y. and Grandvalet, Y. (2004) · 2004
Cited alongside, same era.
Combining Pattern Classifiers: Methods and Algorithms
Kuncheva, L. I. (2004) · 2004
Cited alongside, same era.
Prediction error estimation: a comparison of resampling methods
Molinaro, A. M., Simon, R., and Pfeiffer, R. M. (2005) · 2005
Cited alongside, same era.

Then

In Introduction to Data Mining
Tan, P.-N., Steinbach, M., and Kumar, V. (2005) · 2005
Later among the works it cites.
Bias in error estimation when using cross-validation for model selection
Varma, S. and Simon, R. (2006) · 2006
Later among the works it cites.
On comparison of feature selection algorithms
Refaeilzadeh, P., Tang, L., and Liu, H. (2007) · 2007
Later among the works it cites.
In The Elements of Statistical Learning: Data Mining, Inference, and Prediction
Hastie, T., Tibshirani, R., and Friedman, J. H. (2009) · 2009
Later among the works it cites.
Estimating classification error rate: Repeated cross-validation, repeated hold-out and bootstrap
Kim, J.-H. (2009) · 2009
Later among the works it cites.
Multiple McNemar tests
Westfall, P. H., Troendle, J. F., and Pennello, G. (2010) · 2010
Later among the works it cites.
Scikit-learn: Machine learning in python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., et al. (2011) · 2011
Later among the works it cites.
Statistical Methods for Rates and Proportions
Fleiss, J. L., Levin, B., and Paik, M. C. (2013) · 2013
Later among the works it cites.
In An Introduction to Statistical Learning: With Applications in R
James, G., Witten, D., Hastie, T., and Tibshirani, R. (2013) · 2013
Later among the works it cites.
Cross-validation failure: small sample sizes lead to large error bars
Varoquaux, G. (2017) · 2017
Later among the works it cites.
Mlxtend: Providing machine learning and data science utilities and extensions to python’s scientific computing stack
Raschka, S. (2018) · 2018
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…