SAVGBench: Benchmarking Spatially Aligned Audio-Video Generation

Understand

This work addresses the lack of multimodal generative models capable of producing high-quality videos with spatially aligned audio.

• While recent advancements in generative models have been successful in video generation, they often overlook the spatial alignment between audio and visuals, which is essential for immersive experiences.
• To tackle this problem, we establish a new research direction in benchmarking the Spatially Aligned Audio-Video Generation (SAVG) task.
• We introduce a spatially aligned audio-visual dataset, whose audio and video data are curated based on whether sound events are onscreen or not.