Continual Backprop: Stochastic Gradient Descent with Persistent Randomness

Understand

The Backprop algorithm for learning in neural networks utilizes two mechanisms: first, stochastic gradient descent and second, initialization with small random weights, where the latter is essential to the effectiveness of the former.

• We show that in continual learning setups, Backprop performs well initially, but over time its performance degrades.
• Stochastic gradient descent alone is insufficient to learn continually; the initial randomness enables only initial learning but not continual learning.
• To the best of our knowledge, ours is the first result showing this degradation in Backprop's ability to learn.