Cooperative Learning of Audio and Video Models from Self-Supervised Synchronization

Understand

There is a natural correlation between the visual and auditive elements of a video.

• In this work we leverage this connection to learn general and effective models for both audio and video analysis from self-supervised temporal synchronization.
• We demonstrate that a calibrated curriculum learning scheme, a careful choice of negative examples, and the use of a contrastive loss are critical ingredients to obtain powerful multi-sensory representations from models optimized to discern temporal synchronization of audio-video pairs.
• Without further finetuning, the resulting audio features achieve performance superior or comparable to the state-of-the-art on established audio classification benchmarks (DCASE2014 and ESC-50).