Ensemble Distillation for Neural Machine Translation

Understand

Knowledge distillation describes a method for training a student network to perform better by learning from a stronger teacher network.

• Translating a sentence with an Neural Machine Translation (NMT) engine is time expensive and having a smaller model speeds up this process.
• We demonstrate how to transfer the translation quality of an ensemble and an oracle BLEU teacher network into a single NMT system.
• Further, we present translation improvements from a teacher network that has the same architecture and dimensions of the student network.