2016

Revisiting Classifier Two-Sample Tests

Lopez-Paz, David, Oquab, Maxime

Understand

The goal of two-sample tests is to assess whether two samples, $S_P \sim P^n$ and $S_Q \sim Q^m$, are drawn from the same distribution.

Perhaps intriguingly, one relatively unexplored method to build two-sample tests is the use of binary classifiers.
In particular, construct a dataset by pairing the $n$ examples in $S_P$ with a positive label, and by pairing the $m$ examples in $S_Q$ with a negative label.
If the null hypothesis "$P = Q$" is true, then the classification accuracy of a binary classifier on a held-out subset of this dataset should remain near chance-level.

Built on

The probable error of a mean
Student · 1908
Earlier work this paper cites.
Sulla determinazione empirica di una legge di distribuzione
A. N. Kolmogorov · 1933
Earlier work this paper cites.
On the estimation of the discrepancy between empirical curves of distribution for two independent samples
N. V. Smirnov · 1939
Earlier work this paper cites.
Individual comparisons by ranking methods
F. Wilcoxon · 1945
Earlier work this paper cites.
On a test of whether one of two random variables is stochastically larger than the other
H. B. Mann and D. R. Whitney · 1947
Earlier work this paper cites.
Tests concerning random points on a circle
N. H. Kuiper · 1962
Earlier work this paper cites.
Sampling and bayes’ inference in scientific modelling and robustness
G. E. P. Box · 1980
Earlier work this paper cites.
Binomial approximation to the poisson binomial distribution
W. Ehm · 1991
Earlier work this paper cites.
Effect of high dimension: by an example of a two sample problem
Z. Bai and H. Saranadasa · 1996
Earlier work this paper cites.
Gradient-based learning applied to document recognition
Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner · 1998
Earlier work this paper cites.
Learning with kernels
A. J. Smola and B. Schölkopf · 1998
Earlier work this paper cites.
Statistical learning theory
V. Vapnik · 1998
Earlier work this paper cites.
The elements of statistical learning
J. Friedman, T. Hastie, and R. Tibshirani · 2001
Earlier work this paper cites.
On multivariate goodness of fit and two sample testing
J. H. Friedman · 2003
Earlier work this paper cites.
Measuring statistical dependence with hilbert-schmidt norms
A. Gretton, O. Bousquet, A. Smola, and B. Schölkopf · 2005
Earlier work this paper cites.
Testing statistical hypotheses
E. L. Lehmann and J. P. Romano · 2006
Earlier work this paper cites.
Labeled faces in the wild: A database for studying face recognition in unconstrained environments
G. B. Huang, M. Ramesh, T. Berg, and E. Learned-Miller · 2007
Earlier work this paper cites.
Kernel choice and classifiability for rkhs embeddings of probability distributions
K. Fukumizu, A. Gretton, Gert R. L., B. Schölkopf, and B. Sriperumbudur · 2009
Earlier work this paper cites.

Similar

Causality
J. Pearl · 2009
Cited alongside, same era.
Machine learning classifiers and fMRI: a tutorial overview
F. Pereira, T. Mitchell, and M. Botvinick · 2009
Cited alongside, same era.
Estimation of information theoretic measures for continuous random variables
F. Pérez-Cruz · 2009
Cited alongside, same era.
f-divergence estimation and two-sample homogeneity test under semiparametric density-ratio models
T. Kanamori, T. Suzuki, and M. Sugiyama · 2010
Cited alongside, same era.
Information, divergence and risk for binary experiments
M. D. Reid and R. C. Williamson · 2011
Cited alongside, same era.
Noise-contrastive estimation of unnormalized statistical models, with applications to natural image statistics
M. U. Gutmann and A. Hyvärinen · 2012
Cited alongside, same era.
Induction in neuroscience with classification: issues and solutions
E. Olivetti, S. Greiner, and P. Avesani · 2012
Cited alongside, same era.
Bounding the test log-likelihood of generative models
Y. Bengio, L. Yao, and K. Cho · 2013
Cited alongside, same era.
Distributed representations of words and phrases and their compositionality
T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean · 2013
Cited alongside, same era.
Generative adversarial nets
I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio · 2014
Cited alongside, same era.
Conditional generative adversarial nets
M. Mirza and S. Osindero · 2014
Cited alongside, same era.
A test of relative similarity for model selection in generative models
W. Bounliphone, E. Belilovsky, M. B. Blaschko, I. Antonoglou, and A. Gretton · 2015
Cited alongside, same era.

Then

Adaptivity and Computation-Statistics Tradeoffs for Kernel and Distance based High Dimensional Two Sample Testing
A. Ramdas, S. J. Reddi, B. Poczos, A. Singh, and L. Wasserman · 2015
Later among the works it cites.
On the high dimensional power of a linear-time two sample test under mean-shift alternatives
S. J. Reddi, A. Ramdas, B. Póczos, A. Singh, and L. A. Wasserman · 2015
Later among the works it cites.
ImageNet large scale visual recognition challenge
O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei · 2015
Later among the works it cites.
LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop
F. Yu, A. Seff, Y. Zhang, S. Song, T. Funkhouser, and J. Xiao · 2015
Later among the works it cites.
Training and investigating residual nets, 2016
S. Gross and M. Wilber · 2016
Closest in time.
Interpretable Distribution Features with Maximum Testing Power
W. Jitkrittum, Z. Szabo, K. Chwialkowski, and A. Gretton · 2016
Closest in time.
Perceptual Losses for Real-Time Style Transfer and Super-Resolution
J. Johnson, A. Alahi, and L. Fei-Fei · 2016
Closest in time.
Linking losses for density ratio and class-probability estimation
A. K. Menon and C. S. Ong · 2016
Closest in time.
Learning in Implicit Generative Models
S. Mohamed and B. Lakshminarayanan · 2016
Closest in time.
Distinguishing cause from effect using observational data: methods and benchmarks
J. M. Mooij, J. Peters, D. Janzing, J. Zscheischler, and B. Schölkopf · 2016
Closest in time.
f-GAN: Training generative neural samplers using variational divergence minimization
S. Nowozin, B. Cseke, and R. Tomioka · 2016
Closest in time.
Deconvolution and checkerboard artifacts
A. Odena, V. Dumoulin, and C. Olah · 2016
Closest in time.
Unsupervised representation learning with deep convolutional generative adversarial networks
A. Radford, L. Metz, and S. Chintala · 2016
Closest in time.
Classification accuracy as a proxy for two sample testing
A. Ramdas, A. Singh, and L. Wasserman · 2016
Closest in time.
Improved techniques for training GANs
T. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford, and X. Chen · 2016
Closest in time.
Generative Models and Model Criticism via Optimized Maximum Mean Discrepancy
D. J. Sutherland, H.-Y. Tung, H. Strathmann, S. De, A. Ramdas, A. Smola, and A. Gretton · 2016
Closest in time.
A note on the evaluation of generative models
L. Theis, A. van den Oord, and M. Bethge · 2016
Closest in time.
Two-sample homogeneity tests based on divergence measures
M. Wornowizki and R. Fried · 2016
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…