E2E Segmentation in a Two-Pass Cascaded Encoder ASR Model

Understand

We explore unifying a neural segmenter with two-pass cascaded encoder ASR into a single model.

• A key challenge is allowing the segmenter (which runs in real-time, synchronously with the decoder) to finalize the 2nd pass (which runs 900 ms behind real-time) without introducing user-perceived latency or deletion errors during inference.
• We propose a design where the neural segmenter is integrated with the causal 1st pass decoder to emit a end-of-segment (EOS) signal in real-time.
• The EOS signal is then used to finalize the non-causal 2nd pass.