Fetching the paper…
Reading the bibliography…
We empirically show that the test error of deep networks can be estimated by simply training the same architecture on the same training set but with a different run of Stochastic Gradient Descent (SGD), and measuring the disagreement rate between the two networks on unlabeled test data.
Generalization in deep networks: The role of distance from initialization
Vaishnavh Nagarajan and J Zico Kolter · 1901
Earlier work this paper cites.
Deterministic pac-bayesian generalization bounds for deep networks via generalizing noise-resilience
Vaishnavh Nagarajan and J Zico Kolter · 1905
Earlier work this paper cites.
Towards task and architecture-independent generalization gap predictors
Scott Yak, Javier Gonzalvo, and Hanna Mazzawi · 1906
Earlier work this paper cites.
Verification of probabilistic predictions: A brief review
Allan H Murphy and Edward S Epstein · 1967
Earlier work this paper cites.
Chervonenkis: On the uniform convergence of relative frequencies of events to their probabilities
Vladimir Naumovich Vapnik · 1971
Earlier work this paper cites.
The well-calibrated bayesian
A Philip Dawid · 1982
Earlier work this paper cites.
A theory of the learnable
Leslie G Valiant · 1984
Earlier work this paper cites.
Bagging predictors
Leo Breiman · 1996
Earlier work this paper cites.
Obtaining calibrated probability estimates from decision trees and naive bayesian classifiers
Bianca Zadrozny and Charles Elkan · 2001
Earlier work this paper cites.
Predicting neural network accuracy from weights
Thomas Unterthiner, Daniel Keysers, Sylvain Gelly, Olivier Bousquet, and Ilya O. Tolstikhin · 2002
Earlier work this paper cites.
Co-validation: Using model disagreement on unlabeled data to validate classification algorithms
Omid Madani, David M. Pennock, and Gary William Flake · 2004
Earlier work this paper cites.
Moment multicalibration for uncertainty estimation
Christopher Jung, Changhwa Lee, Mallesh M. Pai, Aaron Roth, and Rakesh Vohra · 2008
Earlier work this paper cites.
Imagenet: A large-scale hierarchical image database
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei · 2009
Earlier work this paper cites.
Learning multiple layers of features from tiny images
Alex Krizhevsky, Geoffrey Hinton, et al · 2009
Earlier work this paper cites.
Distributional generalization: A new kind of generalization
Preetum Nakkiran and Yamini Bansal · 2009
Earlier work this paper cites.
Unsupervised supervised learning I: estimating classification and regression errors without labels
Pinar Donmez, Guy Lebanon, and Krishnakumar Balasubramanian · 2010
Earlier work this paper cites.
Reading digits in natural images with unsupervised feature learning
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng · 2011
Earlier work this paper cites.
Towards understanding ensemble, knowledge distillation and self-distillation in deep learning
Zeyuan Allen-Zhu and Yuanzhi Li · 2012
Earlier work this paper cites.
Neurips 2020 competition: Predicting generalization in deep learning
Yiding Jiang, Pierre Foret, Scott Yak, Daniel M Roy, Hossein Mobahi, Gintare Karolina Dziugaite, Samy Bengio, Suriya Gunasekar, Isabelle Guyon, and Behnam Neyshabur · 2012
Earlier work this paper cites.
Representation based complexity measures for predicting generalization in deep learning
Parth Natekar and Manik Sharma · 2012
Earlier work this paper cites.
Min Lin, Qiang Chen, and Shuicheng Yan · 2013
Earlier work this paper cites.
In search of the real inductive bias: On the role of implicit regularization in deep learning
Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro · 2014
Earlier work this paper cites.
Estimating the accuracies of multiple classifiers without labeled data
Ariel Jaffe, Boaz Nadler, and Yuval Kluger · 2015
Earlier work this paper cites.
Obtaining well calibrated probabilities using bayesian binning
Mahdi Pakdaman Naeini, Gregory F. Cooper, and Milos Hauskrecht · 2015
Cited alongside, same era.
Deep residual learning for image recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun · 2016
Cited alongside, same era.
Launch and iterate: Reducing prediction churn
Mahdi Milani Fard, Quentin Cormier, Kevin Canini, and Maya Gupta · 2016
Cited alongside, same era.
Unsupervised risk estimation using only conditional independence structure
Jacob Steinhardt and Percy Liang · 2016
Cited alongside, same era.
A closer look at memorization in deep networks
Devansh Arpit, Stanislaw Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S. Kanwal, Tegan Maharaj, Asja Fischer, Aaron C. Courville, Yoshua Bengio, and Simon Lacoste-Julien · 2017
Cited alongside, same era.
Spectrally-normalized margin bounds for neural networks
Peter L. Bartlett, Dylan J. Foster, and Matus J. Telgarsky · 2017
The implicit fairness criterion of unconstrained learning
Lydia T. Liu, Max Simchowitz, and Moritz Hardt · 2019
Later among the works it cites.
Measuring calibration in deep learning
Jeremy Nixon, Michael W Dusenberry, Linchuan Zhang, Ghassen Jerfel, and Dustin Tran · 2019
Later among the works it cites.
Identifying and understanding deep learning phenomena
Hanie Sedghi, Samy Bengio, Kenji Hata, Aleksander Madry, Ari Morcos, Behnam Neyshabur, Maithra Raghu, Ali Rahimi, Ludwig Schmidt, and Ying Xiao · 2019
Later among the works it cites.
Evaluating model calibration in classification
Juozas Vaicenavicius, David Widmann, Carl R. Andersson, Fredrik Lindsten, Jacob Roll, and Thomas B. Schön · 2019
Later among the works it cites.
Calibration tests in multi-class classification: A unifying framework
David Widmann, Fredrik Lindsten, and Dave Zachariah · 2019
Later among the works it cites.
Estimating generalization under distribution shifts via domain-invariant representations
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Cited alongside, same era.
Gintare Karolina Dziugaite and Daniel M Roy · 2017
Cited alongside, same era.
On calibration of modern neural networks
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger · 2017
Cited alongside, same era.
Simple and scalable predictive uncertainty estimation using deep ensembles
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell · 2017
Cited alongside, same era.
Deeper, broader and artier domain generalization
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M. Hospedales · 2017
Cited alongside, same era.
Exploring generalization in deep learning
Behnam Neyshabur, Srinadh Bhojanapalli, David McAllester, and Nati Srebro · 2017
Cited alongside, same era.
Estimating accuracy from unlabeled data: A probabilistic logic approach
Emmanouil A. Platanios, Hoifung Poon, Tom M. Mitchell, and Eric Horvitz · 2017
Cited alongside, same era.
Ching-Yao Chuang, Antonio Torralba, and Stefanie Jegelka · 2020
Later among the works it cites.
In search of robust measures of generalization
Gintare Karolina Dziugaite, Alexandre Drouin, Brady Neal, Nitarshan Rajkumar, Ethan Caballero, Linbo Wang, Ioannis Mitliagkas, and Daniel M Roy · 2020
Later among the works it cites.
Distribution-free binary classification: prediction sets, confidence intervals and calibration
Chirag Gupta, Aleksandr Podkopaev, and Aaditya Ramdas · 2020
Later among the works it cites.
Deep double descent: Where bigger models and more data hurt
Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever · 2020
Later among the works it cites.
In defense of uniform convergence: Generalization via derandomization with an application to interpolating predictors
Jeffrey Negrea, Gintare Karolina Dziugaite, and Daniel Roy · 2020
Later among the works it cites.
Why are bootstrapped deep ensembles not better?
Jeremy Nixon, Balaji Lakshminarayanan, and Dustin Tran · 2020
Later among the works it cites.
Learning to validate the predictions of black box classifiers on unseen data
Sebastian Schelter, Tammo Rukat, and Felix Bießmann · 2020
Later among the works it cites.
Sample complexity of uniform convergence for multicalibration
Eliran Shabat, Lee Cohen, and Yishay Mansour · 2020
Later among the works it cites.
On uniform convergence and low-norm interpolation learning
Lijia Zhou, Danica J. Sutherland, and Nati Srebro · 2020
Later among the works it cites.
Yu Bai, Song Mei, Huan Wang, and Caiming Xiong · 2021
Closest in time.
On the reproducibility of neural network predictions
Srinadh Bhojanapalli, Kimberly Wilber, Andreas Veit, Ankit Singh Rawat, Seungyeon Kim, Aditya Menon, and Sanjiv Kumar · 2021
Closest in time.
Detecting errors and estimating accuracy on unlabeled data with self-training ensembles
Jiefeng Chen, Frederick Liu, Besim Avci, Xi Wu, Yingyu Liang, and Somesh Jha · 2021
Closest in time.
RATT: leveraging unlabeled data to guarantee generalization
Saurabh Garg, Sivaraman Balakrishnan, J. Zico Kolter, and Zachary C. Lipton · 2021
Closest in time.
Early-stopped neural networks are consistent
Ziwei Ji, Justin D. Li, and Matus Telgarsky · 2021
Closest in time.
Churn reduction via distillation
Heinrich Jiang, Harikrishna Narasimhan, Dara Bahri, Andrew Cotter, and Afshin Rostamizadeh · 2021
Closest in time.
Jishnu Mukhoti, Andreas Kirsch, Joost van Amersfoort, Philip H. S. Torr, and Yarin Gal · 2021
Closest in time.
Should ensemble members be calibrated?
Xixin Wu and Mark Gales · 2021
Closest in time.