Learning Representations from Audio-Visual Spatial Alignment

Understand

We introduce a novel self-supervised pretext task for learning representations from audio-visual content.

• Prior work on audio-visual representation learning leverages correspondences at the video level.
• Approaches based on audio-visual correspondence (AVC) predict whether audio and video clips originate from the same or different video instances.
• Audio-visual temporal synchronization (AVTS) further discriminates negative pairs originated from the same video instance but at different moments in time.