Embrace the Gap: VAEs Perform Independent Mechanism Analysis

Understand

Variational autoencoders (VAEs) are a popular framework for modeling complex data distributions; they can be efficiently trained via variational inference by maximizing the evidence lower bound (ELBO), at the expense of a gap to the exact (log-)marginal likelihood.

• While VAEs are commonly used for representation learning, it is unclear why ELBO maximization would yield useful representations, since unregularized maximum likelihood estimation cannot invert the data-generating process.
• Yet, VAEs often succeed at this task.
• We seek to elucidate this apparent paradox by studying nonlinear VAEs in the limit of near-deterministic decoders.