Built on

Uniform convergence may be unable to explain generalization in deep learning
Original
Vaishnavh Nagarajan and J. Zico Kolter · 1902
Earlier work this paper cites.
Generalization bounds for deep convolutional neural networks
Original
Philip M. Long and Hanie Sedghi · 1905
Earlier work this paper cites.
Data-dependent sample complexity of deep neural networks via lipschitz augmentation
Original
Colin Wei and Tengyu Ma · 1905
Earlier work this paper cites.
Compression based bound for non-compressed network: unified generalization error analysis of large compressible deep neural network
Original
Taiji Suzuki, Hiroshi Abe, and Tomoaki Nishimura · 1909
Earlier work this paper cites.
ℓ ∞ \ell_{\infty} vector contraction for rademacher complexity
Original
Dylan J. Foster and Alexander Rakhlin · 1911
Earlier work this paper cites.
Fantastic generalization measures and where to find them
Original
Yiding Jiang, Behnam Neyshabur, Hossein Mobahi, Dilip Krishnan, and Samy Bengio · 1912
Earlier work this paper cites.
Remarques sur un résultat non publié de b. maurey
Gilles Pisier · 1980
Earlier work this paper cites.
Neural Network Learning: Theoretical Foundations
Martin Anthony and Peter L. Bartlett · 1999
Earlier work this paper cites.

Similar

Model compression
Cristian Buciluŭ, Rich Caruana, and Alexandru Niculescu-Mizil · 2006
Cited alongside, same era.
Improved approximation algorithms for large matrices via random projections
Tamas Sarlos · 2006
Cited alongside, same era.
Pruning neural networks at initialization: Why are we missing the mark?
Original
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M. Roy, and Michael Carbin · 2009
Cited alongside, same era.
Sanity-checking pruning methods: Random tickets can win the jackpot
Original
Jingtong Su, Yihang Chen, Tianle Cai, Tianhao Wu, Ruiqi Gao, Liwei Wang, and Jason D. Lee · 2009
Cited alongside, same era.
Boosting: Foundations and Algorithms
Robert E. Schapire and Yoav Freund · 2012
Cited alongside, same era.

Then

Dawnbench: An end-to-end deep learning benchmark and competition
Cody A. Coleman, Deepak Narayanan, Daniel Kang, Tian Zhao, Jian Zhang, Luigi Nardi, Peter Bailis, Kunle Olukotun, Chris Ré, and Matei Zaharia · 2017
Later among the works it cites.
Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data
Original
Gintare Karolina Dziugaite and Daniel M. Roy · 2017
Later among the works it cites.
Uniform convergence rates for kernel density estimation
Heinrich Jiang · 2017
Later among the works it cites.
Stronger generalization bounds for deep nets via a compression approach
Original
Sanjeev Arora, Rong Ge, Behnam Neyshabur, and Yi Zhang · 2018
Later among the works it cites.
Size-independent sample complexity of neural networks
Noah Golowich, Alexander Rakhlin, and Ohad Shamir · 2018
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…

2021

Generalization bounds via distillation

Hsu, Daniel, Ji, Ziwei, Telgarsky, Matus et al.

Understand

This paper theoretically investigates the following empirical phenomenon: given a high-complexity network with poor generalization bounds, one can distill it into a network with nearly identical predictions but low complexity and vastly smaller generalization bounds.

The main contribution is an analysis showing that the original network inherits this good generalization bound from its distillation, assuming the use of well-behaved data augmentation.
This bound is presented both in an abstract and in a concrete form, the latter complemented by a reduction technique to handle modern computation graphs featuring convolutional layers, fully-connected layers, and skip connections, to name a few.
To round out the story, a (looser) classical uniform convergence analysis of compression is also presented, as well as a variety of experiments on cifar and mnist demonstrating similar generalization performance between the original network and its distillation.

Built on

Uniform convergence may be unable to explain generalization in deep learning
Original
Vaishnavh Nagarajan and J. Zico Kolter · 1902
Earlier work this paper cites.
Generalization bounds for deep convolutional neural networks
Original
Philip M. Long and Hanie Sedghi · 1905
Earlier work this paper cites.
Data-dependent sample complexity of deep neural networks via lipschitz augmentation
Original
Colin Wei and Tengyu Ma · 1905
Earlier work this paper cites.
Compression based bound for non-compressed network: unified generalization error analysis of large compressible deep neural network
Original
Taiji Suzuki, Hiroshi Abe, and Tomoaki Nishimura · 1909
Earlier work this paper cites.
ℓ ∞ \ell_{\infty} vector contraction for rademacher complexity
Original
Dylan J. Foster and Alexander Rakhlin · 1911
Earlier work this paper cites.
Fantastic generalization measures and where to find them
Original
Yiding Jiang, Behnam Neyshabur, Hossein Mobahi, Dilip Krishnan, and Samy Bengio · 1912
Earlier work this paper cites.
Remarques sur un résultat non publié de b. maurey
Gilles Pisier · 1980
Earlier work this paper cites.
Neural Network Learning: Theoretical Foundations
Martin Anthony and Peter L. Bartlett · 1999
Earlier work this paper cites.

Similar

Model compression
Cristian Buciluŭ, Rich Caruana, and Alexandru Niculescu-Mizil · 2006
Cited alongside, same era.
Improved approximation algorithms for large matrices via random projections
Tamas Sarlos · 2006
Cited alongside, same era.
Pruning neural networks at initialization: Why are we missing the mark?
Original
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M. Roy, and Michael Carbin · 2009
Cited alongside, same era.
Sanity-checking pruning methods: Random tickets can win the jackpot
Original
Jingtong Su, Yihang Chen, Tianle Cai, Tianhao Wu, Ruiqi Gao, Liwei Wang, and Jason D. Lee · 2009
Cited alongside, same era.
Boosting: Foundations and Algorithms
Robert E. Schapire and Yoav Freund · 2012
Cited alongside, same era.

Then

Dawnbench: An end-to-end deep learning benchmark and competition
Cody A. Coleman, Deepak Narayanan, Daniel Kang, Tian Zhao, Jian Zhang, Luigi Nardi, Peter Bailis, Kunle Olukotun, Chris Ré, and Matei Zaharia · 2017
Later among the works it cites.
Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data
Original
Gintare Karolina Dziugaite and Daniel M. Roy · 2017
Later among the works it cites.
Uniform convergence rates for kernel density estimation
Heinrich Jiang · 2017
Later among the works it cites.
Stronger generalization bounds for deep nets via a compression approach
Original
Sanjeev Arora, Rong Ge, Behnam Neyshabur, and Yi Zhang · 2018
Later among the works it cites.
Size-independent sample complexity of neural networks
Noah Golowich, Alexander Rakhlin, and Ohad Shamir · 2018
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Understanding Machine Learning: From Theory to Algorithms

Shai Shalev-Shwartz and Shai Ben-David · 2014

Cited alongside, same era.

Distilling the knowledge in a neural network

Original

Geoffrey Hinton, Oriol Vinyals, and Jeff Dean · 2015

Cited alongside, same era.

Understanding deep learning requires rethinking generalization

Original

Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals · 2016

Cited alongside, same era.

Spectrally-normalized margin bounds for neural networks

Peter L Bartlett, Dylan J Foster, and Matus J Telgarsky

Cited in the paper.

Nearly-tight vc-dimension and pseudodimension bounds for piecewise linear neural networks

Original

Peter L. Bartlett, Nick Harvey, Chris Liaw, and Abbas Mehrabian

Cited in the paper.

Predicting the generalization gap in deep networks with margin distributions

Original

Yiding Jiang, Dilip Krishnan, Hossein Mobahi, and Samy Bengio

Cited in the paper.

The lottery ticket hypothesis: Finding sparse, trainable neural networks

Original

Jonathan Frankle and Michael Carbin · 2019

Later among the works it cites.

In search of robust measures of generalization

Gintare Karolina Dziugaite, Alexandre Drouin, Brady Neal, Nitarshan Rajkumar, Ethan Caballero, Linbo Wang, Ioannis Mitliagkas, and Daniel M. Roy · 2020

Later among the works it cites.