2015

How much does your data exploration overfit? Controlling bias via information usage

Russo, Daniel, Zou, James

Understand

Modern data is messy and high-dimensional, and it is often not clear a priori what are the right questions to ask.

Instead, the analyst typically needs to use the data to search for interesting analyses to perform and hypotheses to test.
This is an adaptive process, where the choice of analysis to be performed next depends on the results of the previous analyses on the same data.
Ultimately, which results are reported can be heavily influenced by the data.

Built on

Y. Benjamini and Y. Hochberg, “Controlling the false discovery rate: a practical and powerful approach to multiple testing,” Journal of the Royal Statistical Society. Series B (Methodological) , pp. 289–300, 1995
1995
Earlier work this paper cites.
Y. Benjamini and D. Yekutieli, “The control of the false discovery rate in multiple testing under dependency,” Annals of statistics , pp. 1165–1188, 2001
2001
Earlier work this paper cites.
O. Bousquet and A. Elisseeff, “Stability and generalization,” Journal of Machine Learning Research , vol. 2, no. Mar, pp. 499–526, 2002
2002
Earlier work this paper cites.
C. Genovese and L. Wasserman, “Operating characteristics and extensions of the false discovery rate procedure,” Journal of the Royal Statistical Society: Series B (Statistical Methodology) , vol. 64, no. 3, pp. 499–517, 2002
2002
Earlier work this paper cites.
T. Poggio, R. Rifkin, S. Mukherjee, and P. Niyogi, “General conditions for predictivity in learning theory,” Nature , vol. 428, no. 6981, pp. 419–422, 2004
2004
Earlier work this paper cites.
B. Efron, T. Hastie, I. Johnstone, R. Tibshirani et al. , “Least angle regression,” The Annals of statistics , vol. 32, no. 2, pp. 407–499, 2004
2004
Earlier work this paper cites.
N. Cesa-Bianchi and G. Lugosi, Prediction, learning, and games . Cambridge University Press, 2006
2006
Earlier work this paper cites.
S. Shalev-Shwartz, O. Shamir, N. Srebro, and K. Sridharan, “Learnability, stability and uniform convergence,” Journal of Machine Learning Research , vol. 11, no. Oct, pp. 2635–2670, 2010
2010
Earlier work this paper cites.
R. Bourgon, R. Gentleman, and W. Huber, “Independent filtering increases detection power for high-throughput experiments,” Proceedings of the National Academy of Sciences , vol. 107, no. 21, pp. 9546–9551, 2010
2010
Earlier work this paper cites.
J. P. Simmons, L. D. Nelson, and U. Simonsohn, “False-positive psychology undisclosed flexibility in data collection and analysis allows presenting anything as significant,” Psychological science , p. 0956797611417632, 2011
2011
Earlier work this paper cites.
I. Lee, G. Lushington, and M. Visvanathan, “A filter-based feature selection approach for identifying potential biomarkers for lung cancer,” Journal of Clinical Bioinformatics , 2011
2011
Earlier work this paper cites.
K. Chaudhuri, C. Monteleoni, and A. D. Sarwate, “Differentially private empirical risk minimization,” Journal of Machine Learning Research , vol. 12, no. Mar, pp. 1069–1109, 2011
2011
Earlier work this paper cites.
R. Gray, Entropy and information theory . Springer, 2011
2011
Earlier work this paper cites.

Similar

T. Cover and J. Thomas, Elements of information theory . John Wiley & Sons, 2012
2012
Cited alongside, same era.
D. McAllester, “A pac-Bayesian tutorial with a dropout bound,” arXiv preprint arXiv:1307.2118 , 2013
Original
2013
Cited alongside, same era.
V. Anantharam, A. Gohari, S. Kamath, and C. Nair, “On maximal correlation, hypercontractivity, and the data processing inequality studied by Erkip and Cover,” arXiv preprint arXiv:1304.6133 , 2013
Original
2013
Cited alongside, same era.
V. Buldygin and K. Moskvichova, “The sub-gaussian norm of a binary random variable,” Theory of Probability and Mathematical Statistics , vol. 86, pp. 33–49, 2013
2013
Cited alongside, same era.
A. Belloni, V. Chernozhukov, and C. Hansen, “Inference on treatment effects after selection among high-dimensional controls,” Review of Economic Studies , vol. 81, no. 287, pp. 608–650, 2014
2014
Cited alongside, same era.
S. Van de Geer, P. Buhlmann, Y. Ritov, R. Dezeure et al. , “On asymptotically optimal confidence regions and tests for high-dimensional models,” The Annals of Statistics , vol. 42, no. 3, pp. 1166–1202, 2014
2014
Cited alongside, same era.
A. Javanmard and A. Montanari, “Confidence intervals and hypothesis testing for high-dimensional regression,” The Journal of Machine Learning Research , vol. 15, no. 1, pp. 2869–2909, 2014
2014
Cited alongside, same era.
R. Lockhart, J. Taylor, R. J. Tibshirani, and R. Tibshirani, “A significance test for the lasso,” Annals of Statistics , vol. 42, no. 2, p. 413, 2014
2014
Cited alongside, same era.
W. Fithian, D. Sun, and J. Taylor, “Optimal inference after model selection,” arXiv preprint arXiv:1410.2597 , 2014
Original
2014
Cited alongside, same era.
J. Taylor, R. Lockhart, R. J. Tibshirani, and R. Tibshirani, “Exact post-selection inference for forward stepwise and least angle regression,” arXiv preprint arXiv:1401.3889 , 2014
Original
2014
Cited alongside, same era.
M. Hardt and J. Ullman, “Preventing false discovery in interactive data analysis is hard,” in Foundations of Computer Science (FOCS), 2014 IEEE 55th Annual Symposium on . IEEE, 2014, pp. 454–463
2014
Cited alongside, same era.
J. Zou, C. Lippert, D. Heckerman, M. Aryee, and J. Listgarten, “Epigenome-wide association studies without the need for cell-type composition,” Nature Methods , pp. 309–11, 2014
2014
Cited alongside, same era.

Then

C. Dwork, V. Feldman, M. Hardt, T. Pitassi, O. Reingold, and A. Roth, “Preserving statistical validity in adaptive data analysis,” in STOC 2015 . ACM, 2014
2014
Later among the works it cites.
J. Taylor and R. J. Tibshirani, “Statistical learning and selective inference,” Proceedings of the National Academy of Sciences , vol. 112, no. 25, pp. 7629–7634, 2015
2015
Closest in time.
A. Blum and M. Hardt, “The ladder: A reliable leaderboard for machine learning competitions,” in International Conference on Machine Learning , 2015, pp. 1006–1014
2015
Closest in time.
C. Dwork, V. Feldman, M. Hardt, T. Pitassi, O. Reingold, and A. Roth, “Generalization in adaptive data analysis and holdout reuse,” in Advances in Neural Information Processing Systems , 2015, pp. 2350–2358
2015
Closest in time.
——, “The reusable holdout: Preserving validity in adaptive data analysis,” Science , vol. 349, no. 6248, pp. 636–638, 2015
2015
Closest in time.
S. Kamath and C. Nair, “The strong data processing constant for sums of iid random variables,” in Information Theory (ISIT), 2015 IEEE International Symposium on . IEEE, 2015, pp. 2550–2552
2015
Closest in time.
R. MacCoun and S. Perlmutter, “Blind analysis: Hide results to seek the truth.” Nature , vol. 526, no. 7572, pp. 187–189, 2015
2015
Closest in time.
M. Wainwright, “Basic tail and concentration bounds,” 2015
2015
Closest in time.
J. D. Lee, D. L. Sun, Y. Sun, J. E. Taylor et al. , “Exact post-selection inference, with application to the lasso,” The Annals of Statistics , vol. 44, no. 3, pp. 907–927, 2016
2016
Closest in time.
S. Wu, A. Joseph, A. S. Hammonds, S. E. Celniker, B. Yu, and E. Frise, “Stability-driven nonnegative matrix factorization to interpret spatial gene expression and build local gene networks,” Proceedings of the National Academy of Sciences , p. 201521171, 2016
2016
Closest in time.
Y. Polyanskiy and Y. Wu, “Dissipation of information in channels with input constraints,” IEEE Transactions on Information Theory , vol. 62, no. 1, pp. 35–55, 2016
2016
Closest in time.
O. Bousquet, S. Boucheron, and G. Lugosi, “Introduction to statistical learning theory,” in Advanced lectures on machine learning . Springer, 2004, pp. 169–207
Original
2017
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…