Temporally Aligned Audio for Video with Autoregression

Understand

We introduce V-AURA, the first autoregressive model to achieve high temporal alignment and relevance in video-to-audio generation.

• V-AURA uses a high-framerate visual feature extractor and a cross-modal audio-visual feature fusion strategy to capture fine-grained visual motion events and ensure precise temporal alignment.
• Additionally, we propose VisualSound, a benchmark dataset with high audio-visual relevance.
• VisualSound is based on VGGSound, a video dataset consisting of in-the-wild samples extracted from YouTube.