A Unified Audio-Visual Learning Framework for Localization, Separation, and Recognition

Understand

The ability to accurately recognize, localize and separate sound sources is fundamental to any audio-visual perception task.

• Historically, these abilities were tackled separately, with several methods developed independently for each task.
• However, given the interconnected nature of source localization, separation, and recognition, independent models are likely to yield suboptimal performance as they fail to capture the interdependence between these tasks.
• To address this problem, we propose a unified audio-visual learning framework (dubbed OneAVM) that integrates audio and visual cues for joint localization, separation, and recognition.