CMCGAN: A Uniform Framework for Cross-Modal Visual-Audio Mutual Generation

Understand

Visual and audio modalities are two symbiotic modalities underlying videos, which contain both common and complementary information.

• If they can be mined and fused sufficiently, performances of related video tasks can be significantly enhanced.
• However, due to the environmental interference or sensor fault, sometimes, only one modality exists while the other is abandoned or missing.
• By recovering the missing modality from the existing one based on the common information shared between them and the prior information of the specific modality, great bonus will be gained for various vision tasks.