2018

A Gaussian Process perspective on Convolutional Neural Networks

Borovykh, Anastasia

Understand

In this paper we cast the well-known convolutional neural network in a Gaussian process perspective.

In this way we hope to gain additional insights into the performance of convolutional networks, in particular understand under what circumstances they tend to perform well and what assumptions are implicitly made in the network.
While for fully-connected networks the properties of convergence to Gaussian processes have been studied extensively, little is known about situations in which the output from a convolutional network approaches a multivariate normal distribution.

Computing with infinite networks
Williams, C. K. (1997) · 1997
Earlier work this paper cites.
Bayesian Gaussian processes for regression and classification
Gibbs, M. N. (1998) · 1998
Earlier work this paper cites.
A Lyapunov-type bound in ℝ d \mathbb{R}^{d}
Bentkus, V. (2005) · 2005
Earlier work this paper cites.
Gaussian processes for machine learning
Rasmussen, C. E. and Williams, C. K. (2006) · 2006
Earlier work this paper cites.
Using deep belief nets to learn covariance kernels for gaussian processes
Hinton, G. E. and Salakhutdinov, R. R. (2008) · 2008
Earlier work this paper cites.
Kernel methods for deep learning
Cho, Y. and Saul, L. K. (2009) · 2009
Earlier work this paper cites.
Additive Gaussian processes
Duvenaud, D. K., Nickisch, H., and Rasmussen, C. E. (2011) · 2011
Earlier work this paper cites.
Gaussian process regression networks
Original
Wilson, A. G., Knowles, D. A., and Ghahramani, Z. (2011) · 2011
Earlier work this paper cites.
Additive covariance kernels for high-dimensional Gaussian process modeling
Durrande, N., Ginsbourger, D., and Roustant, O. (2012) · 2012
Earlier work this paper cites.
A kernel two-sample test
Gretton, A., Borgwardt, K. M., Rasch, M. J., Schölkopf, B., and Smola, A. (2012) · 2012
Earlier work this paper cites.

ImageNet Classification with Deep Convolutional Neural Networks
Krizhevsky, A., Sutskever, I., and Hinton, G. E. (2012) · 2012
Cited alongside, same era.
Bayesian learning for neural networks
Neal, R. M. (2012) · 2012
Cited alongside, same era.
Deep gaussian processes
Damianou, A. C. and Lawrence, N. D. (2013) · 2013
Cited alongside, same era.
Gaussian process kernels for pattern discovery and extrapolation
Wilson, A. and Adams, R. (2013) · 2013
Cited alongside, same era.
Large-scale video classification with convolutional neural networks
Karpathy, A., Toderici, G., Shetty, S., Leung, T., Sukthankar, R., and Fei-Fei, L. (2014) · 2014
Cited alongside, same era.
Convolutional neural networks for sentence classification
Original
Kim, Y. (2014) · 2014
Cited alongside, same era.
Steps toward deep kernel methods from infinite neural networks
Original
Hazan, T. and Jaakkola, T. (2015) · 2015
Cited alongside, same era.
Manifold gaussian processes for regression
Calandra, R., Peters, J., Rasmussen, C. E., and Deisenroth, M. P. (2016) · 2016
Cited alongside, same era.
Simple and scalable predictive uncertainty estimation using deep ensembles
Original
Lakshminarayanan, B., Pritzel, A., and Blundell, C. (2016) · 2016
Cited alongside, same era.

WaveNet: A Generative Model for Raw Audio
van den Oord, A., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner, N., Senior, A., and Kavukcuoglu, K. (2016) · 2016
Later among the works it cites.
Stochastic variational deep kernel learning
Wilson, A. G., Hu, Z., Salakhutdinov, R. R., and Xing, E. P. (2016) · 2016
Later among the works it cites.
Conditional time series forecasting with convolutional neural networks
Original
Borovykh, A., Bohte, S., and Oosterlee, C. W. (2017) · 2017
Later among the works it cites.
Adverial examples, uncertainty and transfer testing robustness in gaussian process hybrid deep networks
Bradshaw, J., Matthews, A. G. d. G., and Ghahramani, Z. (2017) · 2017
Later among the works it cites.
Stochastic gradient descent as approximate bayesian inference
Original
Mandt, S., Hoffman, M. D., and Blei, D. M. (2017) · 2017
Later among the works it cites.
Convolutional gaussian processes
van der Wilk, M., Rasmussen, C. E., and Hensman, J. (2017) · 2017
Later among the works it cites.
Convolutional sequence modeling revisited
Bai, S., Kolter, J. Z., and Koltun, V. (2018) · 2018
Closest in time.
Deep neural networks as gaussian processes
Lee, J., Bahri, Y., Novak, R., Schoenholz, S. S., Pennington, J., and Sohl-Dickstein, J. (2018) · 2018
Closest in time.
Gaussian process behaviour in wide deep neural networks
Matthews, A. G., Hron, J., Rowland, M., Turner, R. E., and Ghahramani, Z. (2018) · 2018
Closest in time.

alphaXiv searches the wider corpus for related work and actual follow-ups.

alphaXiv is searching for related work…

Reading the bibliography…