Sound-to-Imagination: An Exploratory Study on Unsupervised Crossmodal Translation Using Diverse Audiovisual Data

Understand

The motivation of our research is to explore the possibilities of automatic sound-to-image (S2I) translation for enabling a human receiver to visually infer the occurrence of sound related events.

• We expect the computer to 'imagine' the scene from the captured sound, generating original images that picture the sound emitting source.
• Previous studies on similar topics opted for simplified approaches using data with low content diversity and/or sound class supervision.
• Differently, we propose to perform unsupervised S2I translation using thousands of distinct and unknown scenes, with slightly pre-cleaned data, just enough to guarantee aural-visual semantic coherence.