Rate-Regularization and Generalization in VAEs

Understand

Variational autoencoders optimize an objective that combines a reconstruction loss (the distortion) and a KL term (the rate).

• The rate is an upper bound on the mutual information, which is often interpreted as a regularizer that controls the degree of compression.
• We here examine whether inclusion of the rate also acts as an inductive bias that improves generalization.
• We perform rate-distortion analyses that control the strength of the rate term, the network capacity, and the difficulty of the generalization problem.