AADiff: Audio-Aligned Video Synthesis with Text-to-Image Diffusion

Understand

Recent advances in diffusion models have showcased promising results in the text-to-video (T2V) synthesis task.

• However, as these T2V models solely employ text as the guidance, they tend to struggle in modeling detailed temporal dynamics.
• In this paper, we introduce a novel T2V framework that additionally employ audio signals to control the temporal dynamics, empowering an off-the-shelf T2I diffusion to generate audio-aligned videos.
• We propose audio-based regional editing and signal smoothing to strike a good balance between the two contradicting desiderata of video synthesis, i.e., temporal flexibility and coherence.