On the Expressive Power of Deep Learning: A Tensor Analysis

Understand

It has long been conjectured that hypotheses spaces suitable for data that is compositional in nature, such as text or images, may be more efficiently represented with deep hierarchical networks than with shallow ones.

• Despite the vast empirical evidence supporting this belief, theoretical justifications to date are limited.
• In particular, they do not account for the locality, sharing and pooling constructs of convolutional networks, the most successful deep learning architecture to date.
• In this work we derive a deep network architecture based on arithmetic circuits that inherently employs locality, sharing and pooling.