Deep Cross-Modal Audio-Visual Generation

Understand

Cross-modal audio-visual perception has been a long-lasting topic in psychology and neurology, and various studies have discovered strong correlations in human perception of auditory and visual stimuli.

• Despite works in computational multimodal modeling, the problem of cross-modal audio-visual generation has not been systematically studied in the literature.
• In this paper, we make the first attempt to solve this cross-modal generation problem leveraging the power of deep generative adversarial training.
• Specifically, we use conditional generative adversarial networks to achieve cross-modal audio-visual generation of musical performances.