Video-Foley: Two-Stage Video-To-Sound Generation via Temporal Event Condition For Foley Sound

Understand

Foley sound synthesis is crucial for multimedia production, enhancing user experience by synchronizing audio and video both temporally and semantically.

• Recent studies on automating this labor-intensive process through video-to-sound generation face significant challenges.
• Systems lacking explicit temporal features suffer from poor alignment and controllability, while timestamp-based models require costly and subjective human annotation.
• We propose Video-Foley, a video-to-sound system using Root Mean Square (RMS) as an intuitive condition with semantic timbre prompts (audio or text).