The Dual Information Bottleneck

Understand

The Information Bottleneck (IB) framework is a general characterization of optimal representations obtained using a principled approach for balancing accuracy and complexity.

• Here we present a new framework, the Dual Information Bottleneck (dualIB), which resolves some of the known drawbacks of the IB.
• We provide a theoretical analysis of the dualIB framework; (i) solving for the structure of its solutions (ii) unraveling its superiority in optimizing the mean prediction error exponent and (iii) demonstrating its ability to preserve exponential forms of the original distribution.
• To approach large scale problems, we present a novel variational formulation of the dualIB for Deep Neural Networks.