Noisy Self-Knowledge Distillation for Text Summarization

Understand

In this paper we apply self-knowledge distillation to text summarization which we argue can alleviate problems with maximum-likelihood training on single reference and noisy datasets.

• Instead of relying on one-hot annotation labels, our student summarization model is trained with guidance from a teacher which generates smoothed labels to help regularize training.
• Furthermore, to better model uncertainty during training, we introduce multiple noise signals for both teacher and student models.
• We demonstrate experimentally on three benchmarks that our framework boosts the performance of both pretrained and non-pretrained summarizers achieving state-of-the-art results.