Deep Audio-Visual Singing Voice Transcription based on Self-Supervised Learning Models

Understand

Singing voice transcription converts recorded singing audio to musical notation.

• Sound contamination (such as accompaniment) and lack of annotated data make singing voice transcription an extremely difficult task.
• We take two approaches to tackle the above challenges: 1) introducing multimodal learning for singing voice transcription together with a new multimodal singing dataset, N20EMv2, enhancing noise robustness by utilizing video information (lip movements to predict the onset/offset of notes), and 2) adapting self-supervised learning models from the speech domain to the singing voice transcription task, significantly reducing annotated data requirements while preserving pretrained features.
• We build a self-supervised learning based audio-only singing voice transcription system, which not only outperforms current state-of-the-art technologies as a strong baseline, but also generalizes well to out-of-domain singing data.