Fetching the paper…

Reading the bibliography…

2024

STA-V2A: Video-to-Audio Generation with Semantic and Temporal Alignment

Ren, Yong, Li, Chenxing, Xu, Manjie et al.

Original
PDF
Post on X

Understand

Visual and auditory perception are two crucial ways humans experience the world.

Text-to-video generation has made remarkable progress over the past year, but the absence of harmonious audio in generated video limits its broader applications.
In this paper, we propose Semantic and Temporal Aligned Video-to-Audio (STA-V2A), an approach that enhances audio generation from videos by extracting both local temporal and global semantic video features and combining these refined video features with text as cross-modal guidance.
To address the issue of information redundancy in videos, we propose an onset prediction pretext task for local temporal feature extraction and an attentive pooling module for global semantic feature extraction.

STA-V2A: Video-to-Audio Generation with Semantic and Temporal Alignment · Around

Built on

D. P. Kingma and M. Welling, “Auto-encoding variational bayes,” arXiv preprint arXiv:1312.6114 , 2013
Original
2013
Earlier work this paper cites.
S. Böck and G. Widmer, “Maximum filter vibrato suppression for onset detection,” in Proc. of the 16th Int. Conf. on Digital Audio Effects (DAFx). Maynooth, Ireland (Sept 2013) , vol. 7. Citeseer, 2013, p. 4
2013
Earlier work this paper cites.
J. Kong, J. Kim, and J. Bae, “Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis,” Advances in neural information processing systems , vol. 33, pp. 17 022–17 033, 2020
2020
Earlier work this paper cites.
H. Chen, W. Xie, A. Vedaldi, and A. Zisserman, “Vggsound: A large-scale audio-visual dataset,” in ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2020, pp. 721–725
2020
Earlier work this paper cites.
L. Xue, N. Constant, A. Roberts, M. Kale, R. Al-Rfou, A. Siddhant, A. Barua, and C. Raffel, “mt5: A massively multilingual pre-trained text-to-text transformer,” arXiv preprint arXiv:2010.11934 , 2020
Original
2020
Earlier work this paper cites.
V. Iashin and E. Rahtu, “Taming visually guided sound generation,” in BMVA , 2021
2021
Earlier work this paper cites.
A. Ali, I. Schwartz, T. Hazan, and L. Wolf, “Video and text matching with conditioned embeddings,” in Proceedings of the IEEE/CVF winter conference on applications of computer vision , 2022, pp. 1565–1574
2022
Earlier work this paper cites.
J. Ho and T. Salimans, “Classifier-free diffusion guidance,” arXiv preprint arXiv:2207.12598 , 2022
Original
2022
Earlier work this paper cites.
H. Liu, Z. Chen, Y. Yuan, X. Mei, X. Liu, D. Mandic, W. Wang, and M. D. Plumbley, “Audioldm: text-to-audio generation with latent diffusion models,” in Proceedings of the 40th International Conference on Machine Learning , 2023, pp. 21 450–21 474
2023
Earlier work this paper cites.
D. Ghosal, N. Majumder, A. Mehrish, and S. Poria, “Text-to-audio generation using instruction guided latent diffusion model,” in Proceedings of the 31st ACM International Conference on Multimedia , 2023, pp. 3590–3598
2023
Earlier work this paper cites.
L. Ruan, Y. Ma, H. Yang, H. He, B. Liu, J. Fu, N. J. Yuan, Q. Jin, and B. Guo, “Mm-diffusion: Learning multi-modal diffusion models for joint audio and video generation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 10 219–10 228
2023
Earlier work this paper cites.

Similar

R. Sheffer and Y. Adi, “I hear your true colors: Image guided audio generation,” in ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2023, pp. 1–5
2023
Cited alongside, same era.
L. Zhang, A. Rao, and M. Agrawala, “Adding conditional control to text-to-image diffusion models,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023, pp. 3836–3847
2023
Cited alongside, same era.
D. Zavadski, J.-F. Feiden, and C. Rother, “Controlnet-xs: Designing an efficient and effective architecture for controlling text-to-image diffusion models,” arXiv preprint arXiv:2312.06573 , 2023
Original
2023
Cited alongside, same era.
L. Sun, X. Xu, M. Wu, and W. Xie, “A large-scale dataset for audio-language representation learning,” arXiv preprint arXiv:2309.11500 , 2023
Original
2023
Cited alongside, same era.

Then

S. Pascual, C. Yeh, I. Tsiamas, and J. Serrà, “Masked generative video-to-audio transformers with enhanced synchronicity,” arXiv preprint arXiv:2407.10387 , 2024
Original
2024
Closest in time.
Y. Wang, W. Guo, R. Huang, J. Huang, Z. Wang, F. You, R. Li, and Z. Zhao, “Frieren: Efficient video-to-audio generation with rectified flow matching,” arXiv preprint arXiv:2406.00320 , 2024
Original
2024
Closest in time.
S. Mo, J. Shi, and Y. Tian, “Text-to-audio generation synchronized with videos,” arXiv preprint arXiv:2403.07938 , 2024
Original
2024
Closest in time.
Y. Zhang, Y. Gu, Y. Zeng, Z. Xing, Y. Wang, Z. Wu, and K. Chen, “Foleycrafter: Bring silent videos to life with lifelike and synchronized sounds,” arXiv preprint arXiv:2407.01494 , 2024
Original
2024
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Y. Wu, K. Chen, T. Zhang, Y. Hui, T. Berg-Kirkpatrick, and S. Dubnov, “Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation,” in ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2023, pp. 1–5

2023

Cited alongside, same era.

H. Liu, Y. Yuan, X. Liu, X. Mei, Q. Kong, Q. Tian, Y. Wang, W. Wang, Y. Wang, and M. D. Plumbley, “Audioldm 2: Learning holistic audio generation with self-supervised pretraining,” IEEE/ACM Transactions on Audio, Speech, and Language Processing , 2024

2024

Cited alongside, same era.

Y. Xing, Y. He, Z. Tian, X. Wang, and Q. Chen, “Seeing and hearing: Open-domain visual-audio generation with diffusion latent aligners,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024, pp. 7151–7161

2024

Cited alongside, same era.

Y. Mao, X. Shen, J. Zhang, Z. Qin, J. Zhou, M. Xiang, Y. Zhong, and Y. Dai, “Tavgbench: Benchmarking text to audible-video generation,” arXiv preprint arXiv:2404.14381 , 2024

Original

2024

Cited alongside, same era.

A. Hayakawa, M. Ishii, T. Shibuya, and Y. Mitsufuji, “Discriminator-guided cooperative diffusion for joint audio and video generation,” arXiv preprint arXiv:2405.17842 , 2024

2024

Cited alongside, same era.

S. Luo, C. Yan, C. Hu, and H. Zhao, “Diff-foley: Synchronized video-to-audio synthesis with latent diffusion models,” Advances in Neural Information Processing Systems , vol. 36, 2024

2024

Cited alongside, same era.

M. Xu, C. Li, Y. Ren, R. Chen, Y. Gu, W. Liang, and D. Yu, “Video-to-audio generation with hidden alignment,” arXiv preprint arXiv:2407.07464 , 2024

2024

Cited alongside, same era.

M. Comunità, R. F. Gramaccioni, E. Postolache, E. Rodolà, D. Comminiello, and J. D. Reiss, “Syncfusion: Multimodal onset-synchronized video-to-audio foley synthesis,” in ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2024, pp. 936–940

2024

Closest in time.

G. Yariv, I. Gat, S. Benaim, L. Wolf, I. Schwartz, and Y. Adi, “Diverse and aligned audio-to-video generation via text-to-video model adaptation,” in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 38, 2024, pp. 6639–6647

2024

Closest in time.

S. Zhao, D. Chen, Y.-C. Chen, J. Bao, S. Hao, L. Yuan, and K.-Y. K. Wong, “Uni-controlnet: All-in-one control to text-to-image diffusion models,” Advances in Neural Information Processing Systems , vol. 36, 2024

2024

Closest in time.

W. Xuan, Y. Xu, S. Zhao, C. Wang, J. Liu, B. Du, and D. Tao, “When controlnet meets inexplicit masks: A case study of controlnet on its contour-following ability,” arXiv preprint arXiv:2403.00467 , 2024

2024

Closest in time.

S. Deshmukh, D. Alharthi, B. Elizalde, H. Gamper, M. A. Ismail, R. Singh, B. Raj, and H. Wang, “Pam: Prompting audio-language models for audio quality assessment,” arXiv preprint arXiv:2402.00282 , 2024

2024

Closest in time.

I. Schwartz, S. Yu, T. Hazan, and A. G. Schwing, “Factor graph attention,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2019, pp. 2039–2048

2048

Closest in time.