Fetching the paper…

Reading the bibliography…

2013

A PAC-Bayesian Tutorial with A Dropout Bound

McAllester, David

Original
PDF
Post on X

Understand

This tutorial gives a concise overview of existing PAC-Bayesian theory focusing on three generalization bounds.

The first is an Occam bound which handles rules with finite precision parameters and which states that generalization loss is near training loss when the number of bits needed to write the rule is small compared to the sample size.
The second is a PAC-Bayesian bound providing a generalization guarantee for posterior distributions rather than for individual rules.
The PAC-Bayesian bound naturally handles infinite precision rule parameters, $L_2$ regularization, {\em provides a bound for dropout training}, and defines a natural notion of a single distinguished PAC-Bayesian posterior distribution.

A PAC-Bayesian Tutorial with A Dropout Bound · Around

Built on

Fast probabilistic algorithms for hamiltonian circuits and matchings
Dana Angluin and Leslie G Valiant · 1977
Earlier work this paper cites.
A theory of the learnable
Original
Lelsie Valiant · 1984
Earlier work this paper cites.
Bagging predictors
Leo Breiman · 1996
Earlier work this paper cites.
Microchoice bounds and self bounding learning algorithms
John Langford and Avrim Blum · 1999
Earlier work this paper cites.
Pac-bayesian model averaging
David A. McAllester · 1999
Earlier work this paper cites.

Similar

Pac-bayesian generalisation error bounds for gaussian process classification
Matthias Seeger · 2003
Cited alongside, same era.
A note on the pac-bayesian theorem
Andreas Maurer · 2004
Cited alongside, same era.
Use of variance estimation in the multi-armed bandit problem
Jean-Yves Audibert, Rémi Munos, Csaba Szepesvari, et al · 2006
Cited alongside, same era.
Tutorial on practical prediction theory for classification
John Langford · 2006
Cited alongside, same era.

Then

Pac-bayesian supervised classification: the thermodynamics of statistical learning
Original
Olivier Catoni · 2007
Later among the works it cites.
Pac-bayesian learning of linear classifiers
Pascal Germain, Alexandre Lacasse, François Laviolette, and Mario Marchand · 2009
Later among the works it cites.
Distribution-dependent pac-bayes priors
Guy Lever, François Laviolette, and John Shawe-Taylor · 2010
Later among the works it cites.
New types of deep neural network learning for speech recognition and related applications: An overview
Li Deng, Geoffrey Hinton, and Brian Kingsbury · 2013
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…