2020

Wide Neural Networks with Bottlenecks are Deep Gaussian Processes

Agrawal, Devanshu, Papamarkou, Theodore, Hinkle, Jacob

Understand

There has recently been much work on the "wide limit" of neural networks, where Bayesian neural networks (BNNs) are shown to converge to a Gaussian process (GP) as all hidden layers are sent to infinite width.

However, these results do not apply to architectures that require one or more of the hidden layers to remain narrow.
In this paper, we consider the wide limit of BNNs where some hidden layers, called "bottlenecks", are held at finite width.
The result is a composition of GPs that we term a "bottleneck neural network Gaussian process" (bottleneck NNGP).

Built on

The irises of the Gaspe Peninsula
Edgar Anderson · 1935
Earlier work this paper cites.
The use of multiple measurements in taxonomic problems
R. A. Fisher · 1936
Earlier work this paper cites.
Central limit theorems for interchangeable processes
J. R. Blum, H. Chernoff, M. Rosenblatt, and H. Teicher · 1958
Earlier work this paper cites.
The perceptron: A probabilistic model for information storage and organization in the brain
F Rosenblat · 1958
Earlier work this paper cites.
Probability and potentials , volume 1318
Paul André Meyer · 1966
Earlier work this paper cites.
Hedonic housing prices and the demand for clean air
David Harrison and Daniel L Rubinfeld · 1978
Earlier work this paper cites.
Uniform asymptotic normality of the maximum likelihood estimator
TJ Sweeting · 1980
Earlier work this paper cites.
Continuous weak convergence and stochastic equicontinuity results for integrated processes with an application to the estimation of a regression model
Pentti Saikkonen · 1993
Earlier work this paper cites.
Probability and measure
Patrick Billingsley · 1995
Earlier work this paper cites.
Priors for infinite networks
Radford M Neal · 1996
Earlier work this paper cites.
Computing with infinite networks
Christopher KI Williams · 1997
Earlier work this paper cites.
Convergence of probability measures
Patrick Billingsley · 1999
Earlier work this paper cites.
Gaussian process latent variable models for visualisation of high dimensional data
Neil D Lawrence · 2004
Earlier work this paper cites.
Reducing the dimensionality of data with neural networks
Geoffrey E Hinton and Ruslan R Salakhutdinov · 2006
Earlier work this paper cites.
Gaussian processes for machine learning
Carl E. Rasmussen and Christopher K. I. Williams · 2006
Earlier work this paper cites.
Multi-task Gaussian process prediction
Edwin V Bonilla, Kian M Chai, and Christopher Williams · 2008
Earlier work this paper cites.
Random features for large-scale kernel machines
Ali Rahimi and Benjamin Recht · 2008
Earlier work this paper cites.
Sparse convolved gaussian processes for multi-output regression
Mauricio Alvarez and Neil D Lawrence · 2009
Earlier work this paper cites.
Kernel methods for deep learning
Youngmin Cho and Lawrence K Saul · 2009
Earlier work this paper cites.
Variational learning of inducing variables in sparse Gaussian processes
Michalis Titsias · 2009
Earlier work this paper cites.

Similar

Bayesian gaussian process latent variable model
Michalis Titsias and Neil D Lawrence · 2010
Cited alongside, same era.
Imagenet classification with deep convolutional neural networks
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton · 2012
Cited alongside, same era.
Deep Gaussian processes
Andreas Damianou and Neil Lawrence · 2013
Cited alongside, same era.
Gaussian processes for big data
Original
James Hensman, Nicolo Fusi, and Neil D Lawrence · 2013
Cited alongside, same era.
Distributed representations of words and phrases and their compositionality
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean · 2013
Cited alongside, same era.
Avoiding pathologies in very deep networks
David Duvenaud, Oren Rippel, Ryan Adams, and Zoubin Ghahramani · 2014
Cited alongside, same era.
Nested variational compression in deep Gaussian processes
Original
James Hensman and Neil D Lawrence · 2014
Cited alongside, same era.
Auto-encoding variational Bayes
Diederik Kingma and Max Welling · 2014
Cited alongside, same era.
Adam: A method for stochastic optimization
Original
Diederik P Kingma and Jimmy Ba · 2014
Cited alongside, same era.
Learned vs. hand-crafted features for pedestrian gender recognition
Grigory Antipov, Sid-Ahmed Berrani, Natacha Ruchaud, and Jean-Luc Dugelay · 2015
Cited alongside, same era.
Variational auto-encoded deep Gaussian processes
Original
Zhenwen Dai, Andreas Damianou, Javier González, and Neil Lawrence · 2015
Cited alongside, same era.
Dropout as a Bayesian approximation: representing model uncertainty in deep learning
Original
Y. Gal and Z. Ghahramani · 2015
Cited alongside, same era.

Then

Deep information propagation
Original
Samuel S Schoenholz, Justin Gilmer, Surya Ganguli, and Jascha Sohl-Dickstein · 2016
Later among the works it cites.
Sequential inference for deep Gaussian process
Yali Wang, Marcus Brubaker, Brahim Chaib-Draa, and Raquel Urtasun · 2016
Later among the works it cites.
Deep multi-task gaussian processes for survival analysis with competing risks
Ahmed M Alaa and Mihaela van der Schaar · 2017
Later among the works it cites.
Random feature expansions for deep gaussian processes
Kurt Cutajar, Edwin V Bonilla, Pietro Michiardi, and Maurizio Filippone · 2017
Later among the works it cites.
Deep neural networks as Gaussian processes
Original
Jaehoon Lee, Yasaman Bahri, Roman Novak, Samuel S Schoenholz, Jeffrey Pennington, and Jascha Sohl-Dickstein · 2017
Later among the works it cites.
Text feature extraction based on deep learning: a review
Hong Liang, Xiao Sun, Yunlei Sun, and Yuan Gao · 2017
Later among the works it cites.
An overview of multi-task learning in deep neural networks
Original
Sebastian Ruder · 2017
Later among the works it cites.
Doubly stochastic variational inference for deep Gaussian processes
Hugh Salimbeni and Marc Deisenroth · 2017
Later among the works it cites.
Mean field residual networks: on the edge of chaos
Ge Yang and Samuel Schoenholz · 2017
Later among the works it cites.
Gaussian process behaviour in wide deep neural networks
Alexander G. de Matthews, Jiri Hron, Mark Rowland, Richard E. Turner, and Zoubin Ghahramani · 2018
Later among the works it cites.
How deep are deep Gaussian processes?
Matthew M Dunlop, Mark A Girolami, Andrew M Stuart, and Aretha L Teckentrup · 2018
Later among the works it cites.
Gaussian process conditional density estimation
Vincent Dutordoir, Hugh Salimbeni, James Hensman, and Marc Deisenroth · 2018
Later among the works it cites.
Neural tangent kernel: convergence and generalization in neural networks
Arthur Jacot, Franck Gabriel, and Clément Hongler · 2018
Later among the works it cites.
On exact computation with an infinitely wide neural net
Sanjeev Arora, Simon S Du, Wei Hu, Zhiyuan Li, Russ R Salakhutdinov, and Ruosong Wang · 2019
Later among the works it cites.
Deep convolutional networks as shallow Gaussian processes
Adria Garriga-Alonso, Carl Edward Rasmussen, and Laurence Aitchison · 2019
Later among the works it cites.
Wide neural networks of any depth evolve as linear models under gradient descent
Original
Jaehoon Lee, Lechao Xiao, Samuel S Schoenholz, Yasaman Bahri, Jascha Sohl-Dickstein, and Jeffrey Pennington · 2019
Later among the works it cites.
On random deep weight-tied autoencoders: Exact asymptotic analysis, phase transitions, and implications to training
Ping Li and Phan-Minh Nguyen · 2019
Later among the works it cites.
Interpretable deep Gaussian processes with moments
Original
Chi-Ken Lu, Scott Cheng-Hsin Yang, Xiaoran Hao, and Patrick Shafto · 2019
Later among the works it cites.
Bayesian deep convolutional networks with many channels are Gaussian processes
Roman Novak, Lechao Xiao, Yasaman Bahri, Jaehoon Lee, Greg Yang, Daniel A. Abolafia, Jeffrey Pennington, and Jascha Sohl-dickstein · 2019
Later among the works it cites.
Scaling limits of wide neural networks with weight sharing: Gaussian process behavior, gradient independence, and neural tangent kernel derivation
Original
Greg Yang · 2019
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…