2016

Classification accuracy as a proxy for two sample testing

Kim, Ilmun, Ramdas, Aaditya, Singh, Aarti et al.

Understand

When data analysts train a classifier and check if its accuracy is significantly different from chance, they are implicitly performing a two-sample test.

We investigate the statistical properties of this flexible approach in the high-dimensional setting.
We prove two results that hold for all classifiers in any dimensions: if its true error remains $\epsilon$-better than chance for some $\epsilon>0$ as $d,n \to \infty$, then (a) the permutation-based test is consistent (has power approaching to one), (b) a computationally efficient test based on a Gaussian approximation of the null distribution is also consistent.
To get a finer understanding of the rates of consistency, we study a specialized setting of distinguishing Gaussians with mean-difference $\delta$ and common (known or unknown) covariance $\Sigma$, when $d/n \to c \in (0,\infty)$.

Built on

On the use of random forest for two-sample testing
Original
Hediger, S., Michel, L., and Näf, J. (2019) · 1903
Earlier work this paper cites.
The generalization of student’s ratio
Hotelling, H. (1931) · 1931
Earlier work this paper cites.
The use of multiple measurements in taxonomic problems
Fisher, R. A. (1936) · 1936
Earlier work this paper cites.
The precision of discriminant functions
Fisher, R. A. (1940) · 1940
Earlier work this paper cites.
The accuracy of the gaussian approximation to the sum of independent variates
Berry, A. C. (1941) · 1941
Earlier work this paper cites.
On an optimum property of two important statistical tests
Simaika, J. (1941) · 1941
Earlier work this paper cites.
On a statistical problem arising in the classification of an individual into one of two groups
Wald, A. (1944) · 1944
Earlier work this paper cites.
Classification by multivariate analysis
Anderson, T. W. (1951) · 1951
Earlier work this paper cites.
An introduction to multivariate statistical analysis
Anderson, T. W. (1958) · 1958
Earlier work this paper cites.
Minimax Character of Hotelling’s T 2 T^{2} Test in the Simplest Case
Giri, N., Kiefer, J., and Stein, C. (1963) · 1963
Earlier work this paper cites.
Local and asymptotic minimax properties of multivariate tests
Giri, N. and Kiefer, J. (1964) · 1964
Earlier work this paper cites.
Minimax Character of Hotelling’s T 2 T^{2} Test. I
Salaevskii, O. (1971) · 1971
Earlier work this paper cites.
Multivariate generalizations of the Wald-Wolfowitz and Smirnov two-sample tests
Friedman, J. H. and Rafsky, L. C. (1979) · 1979
Earlier work this paper cites.
A robustness property of Hotelling’s T 2 T^{2} -test
Kariya, T. (1981) · 1981
Earlier work this paper cites.
Minimax character of the two-sample χ 2 \chi^{2} -test
Luschgy, H. (1982) · 1982
Earlier work this paper cites.
Multivariate two-sample tests based on nearest neighbors
Schilling, M. F. (1986) · 1986
Earlier work this paper cites.
A multivariate two-sample test based on the number of nearest neighbor type coincidences
Henze, N. (1988) · 1988
Earlier work this paper cites.
Effect of high dimension: by an example of a two sample problem
Bai, Z. D. and Saranadasa, H. (1996) · 1996
Earlier work this paper cites.
Fisher discriminant analysis with kernels
Mika, S., Rätsch, G., Weston, J., Schölkopf, B., Müller, K., Hu, Y.-H., Larsen, J., Wilson, E., and Douglas, S. (1999) · 1999
Earlier work this paper cites.
Asymptotic statistics
Van der Vaart, A. W. (2000) · 2000
Earlier work this paper cites.

Similar

Permutation tests for classification: towards statistical significance in image-based studies
Golland, P. and Fischl, B. (2003) · 2003
Cited alongside, same era.
A survey on continuous elliptical vector distributions
Gómez, E., Gómez-Villegas, M. A., and Marín, J. M. (2003) · 2003
Cited alongside, same era.
Some theory for Fisher’s linear discriminant function,‘naive Bayes’, and some alternatives when there are many more variables than observations
Bickel, P. J. and Levina, E. (2004) · 2004
Cited alongside, same era.
Generalized elliptical distributions: theory and applications
Frahm, G. (2004) · 2004
Cited alongside, same era.
On multivariate goodness-of-fit and two-sample testing
Friedman, J. (2004) · 2004
Cited alongside, same era.
Results in statistical discriminant analysis: A review of the former soviet union literature
Raudys, Š. and Young, D. M. (2004) · 2004
Cited alongside, same era.
An exact distribution-free test comparing two multivariate distributions based on adjacency
Rosenbaum, P. R. (2005) · 2005
Cited alongside, same era.
A Neyman-Pearson approach to statistical learning
Scott, C. and Nowak, R. (2005) · 2005
Cited alongside, same era.
Analysis of representations for domain adaptation
Ben-David, S., Blitzer, J., Crammer, K., and Pereira, F. (2007) · 2007
Cited alongside, same era.
Two-sample Comparison Based on Prediction Error, with Applications to Candidate Gene Association Studies
Yu, K., Martin, R., Rothman, N., Zheng, T., and Lan, Q. (2007) · 2007
Cited alongside, same era.
Testing for homogeneity with kernel fisher discriminant analysis
Eric, M., Bach, F. R., and Harchaoui, Z. (2008) · 2008
Cited alongside, same era.
Linear models in statistics
Rencher, A. C. and Schaalje, G. B. (2008) · 2008
Cited alongside, same era.

Then

Convex analysis and nonlinear optimization: theory and examples
Borwein, J. and Lewis, A. S. (2010) · 2010
Later among the works it cites.
A two-sample test for high-dimensional data with applications to gene-set testing
Chen, S. X. and Qin, Y.-L. (2010) · 2010
Later among the works it cites.
Analytic study of performance of error estimators for linear discriminant analysis
Zollanvari, A., Braga-Neto, U. M., and Dougherty, E. R. (2011) · 2011
Later among the works it cites.
A kernel two-sample test
Gretton, A., Borgwardt, K., Rasch, M., Schoelkopf, B., and Smola, A. (2012) · 2012
Later among the works it cites.
Bilinear forms and zonal polynomials
Mathai, A. M., Provost, S. B., and Hayakawa, T. (2012) · 2012
Later among the works it cites.
Induction in neuroscience with classification: issues and solutions
Olivetti, E., Greiner, S., and Avesani, P. (2012) · 2012
Later among the works it cites.
A two sample test in high dimensional data
Srivastava, M. S., Katayama, S., and Kano, Y. (2013) · 2013
Later among the works it cites.
Statistical inference and multiple testing correction in classification-based multi-voxel pattern analysis (MVPA): random permutations and cluster size control
Stelzer, J., Chen, Y., and Turner, R. (2013) · 2013
Later among the works it cites.
A transfer learning based classifier ensemble model for customer credit scoring
Xiao, J., Wang, R., Teng, G., and Hu, Y. (2014) · 2014
Later among the works it cites.
Feature-selection-based dynamic transfer ensemble model for customer churn prediction
Xiao, J., Xiao, Y., Huang, A., Liu, D., and Wang, S. (2015) · 2015
Later among the works it cites.
A review of 20 years of naive tests of significance for high-dimensional mean vectors and covariance matrices
Hu, J. and Bai, Z. (2016) · 2016
Closest in time.
Revisiting classifier two-sample tests
Original
Lopez-Paz, D. and Oquab, M. (2016) · 2016
Closest in time.
Remember the curse of dimensionality: the case of goodness-of-fit testing in arbitrary dimension
Arias-Castro, E., Pelletier, B., and Saligrama, V. (2018) · 2018
Closest in time.
Symmetric multivariate and related distributions
Fang, K. W., Kotz, S., and Ng, K. W. (2018) · 2018
Closest in time.
False discovery proportion estimation by permutations: confidence for significance analysis of microarrays
Hemerik, J. and Goeman, J. J. (2018) · 2018
Closest in time.
Classifier two-sample test for video anomaly detections
Liu, Y., Li, C.-L., and Póczos, B. (2018) · 2018
Closest in time.
Pros and cons of GAN evaluation measures
Borji, A. (2019) · 2019
Closest in time.
The classification permutation test: A flexible approach to testing for covariate imbalance in observational studies
Gagnon-Bartsch, J. and Shem-Tov, Y. (2019) · 2019
Closest in time.
Better-than-chance classification for signal detection
Rosenblatt, J. D., Benjamini, Y., Gilron, R., Mukamel, R., and Goeman, J. J. (2019) · 2019
Closest in time.
Two-sample tests based on geometric graphs: Asymptotic distribution and detection thresholds
Bhattacharya, B. B. (2020) · 2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…