Two-stage Textual Knowledge Distillation for End-to-End Spoken Language Understanding

Understand

End-to-end approaches open a new way for more accurate and efficient spoken language understanding (SLU) systems by alleviating the drawbacks of traditional pipeline systems.

• Previous works exploit textual information for an SLU model via pre-training with automatic speech recognition or fine-tuning with knowledge distillation.
• To utilize textual information more effectively, this work proposes a two-stage textual knowledge distillation method that matches utterance-level representations and predicted logits of two modalities during pre-training and fine-tuning, sequentially.
• We use vq-wav2vec BERT as a speech encoder because it captures general and rich features.