Resolution and Relevance Trade-offs in Deep Learning

Understand

Deep learning has been successfully applied to various tasks, but its underlying mechanism remains unclear.

• Neural networks associate similar inputs in the visible layer to the same state of hidden variables in deep layers.
• The fraction of inputs that are associated to the same state is a natural measure of similarity and is simply related to the cost in bits required to represent these inputs.
• The degeneracy of states with the same information cost provides instead a natural measure of noise and is simply related the entropy of the frequency of states, that we call relevance.