Weight Sharing is Crucial to Succesful Optimization

Understand

Exploiting the great expressive power of Deep Neural Network architectures, relies on the ability to train them.

• While current theoretical work provides, mostly, results showing the hardness of this task, empirical evidence usually differs from this line, with success stories in abundance.
• A strong position among empirically successful architectures is captured by networks where extensive weight sharing is used, either by Convolutional or Recurrent layers.
• Additionally, characterizing specific aspects of different tasks, making them "harder" or "easier", is an interesting direction explored both theoretically and empirically.