PLATO: Pre-trained Dialogue Generation Model with Discrete Latent Variable

Understand

Pre-training models have been proved effective for a wide range of natural language processing tasks.

• Inspired by this, we propose a novel dialogue generation pre-training framework to support various kinds of conversations, including chit-chat, knowledge grounded dialogues, and conversational question answering.
• In this framework, we adopt flexible attention mechanisms to fully leverage the bi-directional context and the uni-directional characteristic of language generation.
• We also introduce discrete latent variables to tackle the inherent one-to-many mapping problem in response generation.