VinTAGe: Joint Video and Text Conditioning for Holistic Audio Generation

Understand

Recent advances in audio generation have focused on text-to-audio (T2A) and video-to-audio (V2A) tasks.

• However, T2A or V2A methods cannot generate holistic sounds (onscreen and off-screen).
• This is because T2A cannot generate sounds aligning with onscreen objects, while V2A cannot generate semantically complete (offscreen sounds missing).
• In this work, we address the task of holistic audio generation: given a video and a text prompt, we aim to generate both onscreen and offscreen sounds that are temporally synchronized with the video and semantically aligned with text and video.