CoDERT: Distilling Encoder Representations with Co-learning for Transducer-based Speech Recognition

Understand

We propose a simple yet effective method to compress an RNN-Transducer (RNN-T) through the well-known knowledge distillation paradigm.

• We show that the transducer's encoder outputs naturally have a high entropy and contain rich information about acoustically similar word-piece confusions.
• This rich information is suppressed when combined with the lower entropy decoder outputs to produce the joint network logits.
• Consequently, we introduce an auxiliary loss to distill the encoder logits from a teacher transducer's encoder, and explore training strategies where this encoder distillation works effectively.