Adding Gradient Noise Improves Learning for Very Deep Networks

Understand

Deep feedforward and recurrent networks have achieved impressive results in many perception and language processing applications.

• This success is partially attributed to architectural innovations such as convolutional and long short-term memory networks.
• The main motivation for these architectural innovations is that they capture better domain knowledge, and importantly are easier to optimize than more basic architectures.
• Recently, more complex architectures such as Neural Turing Machines and Memory Networks have been proposed for tasks including question answering and general computation, creating a new set of optimization challenges.