Towards Training Recurrent Neural Networks for Lifelong Learning

Understand

Catastrophic forgetting and capacity saturation are the central challenges of any parametric lifelong learning system.

• In this work, we study these challenges in the context of sequential supervised learning with an emphasis on recurrent neural networks.
• To evaluate the models in the lifelong learning setting, we propose a curriculum-based, simple, and intuitive benchmark where the models are trained on tasks with increasing levels of difficulty.
• To measure the impact of catastrophic forgetting, the model is tested on all the previous tasks as it completes any task.