Sparsity and Heterogeneous Dropout for Continual Learning in the Null Space of Neural Activations

Understand

Continual/lifelong learning from a non-stationary input data stream is a cornerstone of intelligence.

• Despite their phenomenal performance in a wide variety of applications, deep neural networks are prone to forgetting their previously learned information upon learning new ones.
• This phenomenon is called "catastrophic forgetting" and is deeply rooted in the stability-plasticity dilemma.
• Overcoming catastrophic forgetting in deep neural networks has become an active field of research in recent years.