Compressing Neural Networks using the Variational Information Bottleneck

Understand

Neural networks can be compressed to reduce memory and computational requirements, or to increase accuracy by facilitating the use of a larger base architecture.

• In this paper we focus on pruning individual neurons, which can simultaneously trim model size, FLOPs, and run-time memory.
• To improve upon the performance of existing compression algorithms we utilize the information bottleneck principle instantiated via a tractable variational bound.
• Minimization of this information theoretic bound reduces the redundancy between adjacent layers by aggregating useful information into a subset of neurons that can be preserved.