Overcoming catastrophic forgetting problem by weight consolidation and long-term memory

Understand

Sequential learning of multiple tasks in artificial neural networks using gradient descent leads to catastrophic forgetting, whereby previously learned knowledge is erased during learning of new, disjoint knowledge.

• Here, we propose a new approach to sequential learning which leverages the recent discovery of adversarial examples.
• We use adversarial subspaces from previous tasks to enable learning of new tasks with less interference.
• We apply our method to sequentially learning to classify digits 0, 1, 2 (task 1), 4, 5, 6, (task 2), and 7, 8, 9 (task 3) in MNIST (disjoint MNIST task).