Neural networks with differentiable structure

Understand

While gradient descent has proven highly successful in learning connection weights for neural networks, the actual structure of these networks is usually determined by hand, or by other optimization algorithms.

• Here we describe a simple method to make network structure differentiable, and therefore accessible to gradient descent.
• We test this method on recurrent neural networks applied to simple sequence prediction problems.
• Starting with initial networks containing only one node, the method automatically builds networks that successfully solve the tasks.