Towards NNGP-guided Neural Architecture Search

Understand

The predictions of wide Bayesian neural networks are described by a Gaussian process, known as the Neural Network Gaussian Process (NNGP).

• Analytic forms for NNGP kernels are known for many models, but computing the exact kernel for convolutional architectures is prohibitively expensive.
• One can obtain effective approximations of these kernels through Monte-Carlo estimation using finite networks at initialization.
• Monte-Carlo NNGP inference is orders-of-magnitude cheaper in FLOPs compared to gradient descent training when the dataset size is small.