Adaptive Compression-based Lifelong Learning

Understand

The problem of a deep learning model losing performance on a previously learned task when fine-tuned to a new one is a phenomenon known as Catastrophic forgetting.

• There are two major ways to mitigate this problem: either preserving activations of the initial network during training with a new task; or restricting the new network activations to remain close to the initial ones.
• The latter approach falls under the denomination of lifelong learning, where the model is updated in a way that it performs well on both old and new tasks, without having access to the old task's training samples anymore.
• Recently, approaches like pruning networks for freeing network capacity during sequential learning of tasks have been gaining in popularity.