A Structured Variational Auto-encoder for Learning Deep Hierarchies of Sparse Features

Understand

In this note we present a generative model of natural images consisting of a deep hierarchy of layers of latent random variables, each of which follows a new type of distribution that we call rectified Gaussian.

• These rectified Gaussian units allow spike-and-slab type sparsity, while retaining the differentiability necessary for efficient stochastic gradient variational inference.
• To learn the parameters of the new model, we approximate the posterior of the latent variables with a variational auto-encoder.
• Rather than making the usual mean-field assumption however, the encoder parameterizes a new type of structured variational approximation that retains the prior dependencies of the generative model.