How do Quadratic Regularizers Prevent Catastrophic Forgetting: The Role of Interpolation

Understand

Catastrophic forgetting undermines the effectiveness of deep neural networks (DNNs) in scenarios such as continual learning and lifelong learning.

• While several methods have been proposed to tackle this problem, there is limited work explaining why these methods work well.
• This paper has the goal of better explaining a popularly used technique for avoiding catastrophic forgetting: quadratic regularization.
• We show that quadratic regularizers prevent forgetting of past tasks by interpolating current and previous values of model parameters at every training iteration.