Learning Explicit and Implicit Latent Common Spaces for Audio-Visual Cross-Modal Retrieval

Understand

Learning common subspace is prevalent way in cross-modal retrieval to solve the problem of data from different modalities having inconsistent distributions and representations that cannot be directly compared.

• Previous cross-modal retrieval methods focus on projecting the cross-modal data into a common space by learning the correlation between them to bridge the modality gap.
• However, the rich semantic information in the video and the heterogeneous nature of audio-visual data leads to more serious heterogeneous gaps intuitively, which may lead to the loss of key semantic content of video with single clue by the previous methods when eliminating the modality gap, while the semantics of the categories may undermine the properties of the original features.
• In this work, we aim to learn effective audio-visual representations to support audio-visual cross-modal retrieval (AVCMR).