GradNets: Dynamic Interpolation Between Neural Architectures

Understand

In machine learning, there is a fundamental trade-off between ease of optimization and expressive power.

• Neural Networks, in particular, have enormous expressive power and yet are notoriously challenging to train.
• The nature of that optimization challenge changes over the course of learning.
• Traditionally in deep learning, one makes a static trade-off between the needs of early and late optimization.