Layer-wise Learning of Stochastic Neural Networks with Information Bottleneck

Understand

Information Bottleneck (IB) is a generalization of rate-distortion theory that naturally incorporates compression and relevance trade-offs for learning.

• Though the original IB has been extensively studied, there has not been much understanding of multiple bottlenecks which better fit in the context of neural networks.
• In this work, we propose Information Multi-Bottlenecks (IMBs) as an extension of IB to multiple bottlenecks which has a direct application to training neural networks by considering layers as multiple bottlenecks and weights as parameterized encoders and decoders.
• We show that the multiple optimality of IMB is not simultaneously achievable for stochastic encoders.