Understanding Knowledge Distillation in Non-autoregressive Machine Translation

Understand

Non-autoregressive machine translation (NAT) systems predict a sequence of output tokens in parallel, achieving substantial improvements in generation speed compared to autoregressive models.

• Existing NAT models usually rely on the technique of knowledge distillation, which creates the training data from a pretrained autoregressive model for better performance.
• Knowledge distillation is empirically useful, leading to large gains in accuracy for NAT models, but the reason for this success has, as of yet, been unclear.
• In this paper, we first design systematic experiments to investigate why knowledge distillation is crucial to NAT training.