Analyzing the Forgetting Problem in the Pretrain-Finetuning of Dialogue Response Models

Understand

In this work, we study how the finetuning stage in the pretrain-finetune framework changes the behavior of a pretrained neural language generator.

• We focus on the transformer encoder-decoder model for the open-domain dialogue response generation task.
• Our major finding is that after standard finetuning, the model forgets some of the important language generation skills acquired during large-scale pretraining.
• We demonstrate the forgetting phenomenon through a set of detailed behavior analysis from the perspectives of knowledge transfer, context sensitivity, and function space projection.