How transfer learning impacts linguistic knowledge in deep NLP models?

Understand

Transfer learning from pre-trained neural language models towards downstream tasks has been a predominant theme in NLP recently.

• Several researchers have shown that deep NLP models learn non-trivial amount of linguistic knowledge, captured at different layers of the model.
• We investigate how fine-tuning towards downstream NLP tasks impacts the learned linguistic knowledge.
• We carry out a study across popular pre-trained models BERT, RoBERTa and XLNet using layer and neuron-level diagnostic classifiers.