Overcoming Catastrophic Forgetting beyond Continual Learning: Balanced Training for Neural Machine Translation

Understand

Neural networks tend to gradually forget the previously learned knowledge when learning multiple tasks sequentially from dynamic data distributions.

• This problem is called \textit{catastrophic forgetting}, which is a fundamental challenge in the continual learning of neural networks.
• In this work, we observe that catastrophic forgetting not only occurs in continual learning but also affects the traditional static training.
• Neural networks, especially neural machine translation models, suffer from catastrophic forgetting even if they learn from a static training set.