Variational Gaussian Dropout is not Bayesian

Understand

Gaussian multiplicative noise is commonly used as a stochastic regularisation technique in training of deterministic neural networks.

• A recent paper reinterpreted the technique as a specific algorithm for approximate inference in Bayesian neural networks; several extensions ensued.
• We show that the log-uniform prior used in all the above publications does not generally induce a proper posterior, and thus Bayesian inference in such models is ill-posed.
• Independent of the log-uniform prior, the correlated weight noise approximation has further issues leading to either infinite objective or high risk of overfitting.