Neural Supervised Domain Adaptation by Augmenting Pre-trained Models with Random Units

Understand

Neural Transfer Learning (TL) is becoming ubiquitous in Natural Language Processing (NLP), thanks to its high performance on many tasks, especially in low-resourced scenarios.

• Notably, TL is widely used for neural domain adaptation to transfer valuable knowledge from high-resource to low-resource domains.
• In the standard fine-tuning scheme of TL, a model is initially pre-trained on a source domain and subsequently fine-tuned on a target domain and, therefore, source and target domains are trained using the same architecture.
• In this paper, we show through interpretation methods that such scheme, despite its efficiency, is suffering from a main limitation.