Read, Watch and Scream! Sound Generation from Text and Video

Understand

Despite the impressive progress of multimodal generative models, video-to-audio generation still suffers from limited performance and limits the flexibility to prioritize sound synthesis for specific objects within the scene.

• Conversely, text-to-audio generation methods generate high-quality audio but pose challenges in ensuring comprehensive scene depiction and time-varying control.
• To tackle these challenges, we propose a novel video-and-text-to-audio generation method, called \ours, where video serves as a conditional control for a text-to-audio generation model.
• Especially, our method estimates the structural information of sound (namely, energy) from the video while receiving key content cues from a user prompt.