Self-supervised Contrastive Cross-Modality Representation Learning for Spoken Question Answering

Understand

Spoken question answering (SQA) requires fine-grained understanding of both spoken documents and questions for the optimal answer prediction.

• In this paper, we propose novel training schemes for spoken question answering with a self-supervised training stage and a contrastive representation learning stage.
• In the self-supervised stage, we propose three auxiliary self-supervised tasks, including utterance restoration, utterance insertion, and question discrimination, and jointly train the model to capture consistency and coherence among speech documents without any additional data or annotations.
• We then propose to learn noise-invariant utterance representations in a contrastive objective by adopting multiple augmentation strategies, including span deletion and span substitution.