Identifying the Limits of Cross-Domain Knowledge Transfer for Pretrained Models

Understand

There is growing evidence that pretrained language models improve task-specific fine-tuning not just for the languages seen in pretraining, but also for new languages and even non-linguistic data.

• What is the nature of this surprising cross-domain transfer? We offer a partial answer via a systematic exploration of how much transfer occurs when models are denied any information about word identity via random scrambling.
• In four classification tasks and two sequence labeling tasks, we evaluate baseline models, LSTMs using GloVe embeddings, and BERT.
• We find that only BERT shows high rates of transfer into our scrambled domains, and for classification but not sequence labeling tasks.