Information bottleneck through variational glasses

Understand

Information bottleneck (IB) principle [1] has become an important element in information-theoretic analysis of deep models.

• Many state-of-the-art generative models of both Variational Autoencoder (VAE) [2; 3] and Generative Adversarial Networks (GAN) [4] families use various bounds on mutual information terms to introduce certain regularization constraints [5; 6; 7; 8; 9; 10].
• Accordingly, the main difference between these models consists in add regularization constraints and targeted objectives.
• In this work, we will consider the IB framework for three classes of models that include supervised, unsupervised and adversarial generative models.