Fetching the paper…

Reading the bibliography…

2025

Vchitect-2.0: Parallel Transformer for Scaling Up Video Diffusion Models

Fan, Weichen, Si, Chenyang, Song, Junhao et al.

Original
PDF
Post on X

Understand

We present Vchitect-2.0, a parallel transformer architecture designed to scale up video diffusion models for large-scale text-to-video generation.

The overall Vchitect-2.0 system has several key designs.
(1) By introducing a novel Multimodal Diffusion Block, our approach achieves consistent alignment between text descriptions and generated video frames, while maintaining temporal coherence across sequences.
(2) To overcome memory and computational bottlenecks, we propose a Memory-efficient Training framework that incorporates hybrid parallelism and other memory reduction techniques, enabling efficient training of long video sequences on distributed systems.

Vchitect-2.0: Parallel Transformer for Scaling Up Video Diffusion Models · Around

Built on

Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro, “Megatron-lm: Training multi-billion parameter language models using model parallelism,”
Original
1909
Earlier work this paper cites.
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He, “ZeRO: Memory optimization towards training a trillion parameter models,”
Original
1910
Earlier work this paper cites.
Shen Li, Yanli Zhao, Rohan Varma, Omkar Salpekar, Pieter Noordhuis, Teng Li, Adam Paszke, Jeff Smith, Brian Vaughan, Pritam Damania, and Soumith Chintala, “Pytorch distributed: Experiences on accelerating data parallel training,” 2020. [Online]. Available:
Original
2006
Earlier work this paper cites.
J. Sohl-Dickstein, E. A. Weiss, N. Maheswaranathan, and S. Ganguli, “Deep unsupervised learning using nonequilibrium thermodynamics,”
Original
2015
Earlier work this paper cites.
Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin, “Training deep nets with sublinear memory cost,” in
Original
2016
Earlier work this paper cites.
P. Micikevicius, S. Narang, J. Alben, G. F. Diamos, E. Elsen, D. García, B. Ginsburg, M. Houston, O. Kuchaiev, G. Venkatesh, and H. Wu, “Mixed precision training,”
Original
2017
Earlier work this paper cites.
Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Mia Xu Chen, Dehao Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V. Le, Yonghui Wu, and Zhifeng Chen, “GPipe: Efficient training of giant neural networks using pipeline parallelism,”
Original
2018
Earlier work this paper cites.
Aaron Harlap, Deepak Narayanan, Amar Phanishayee, Vivek Seshadri, Nikhil Devanur, Greg Ganger, and Phil Gibbons, “Pipedream: Generalized pipeline parallelism for dnn training,”
Original
2018
Earlier work this paper cites.
J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,”
2020
Earlier work this paper cites.
J. Song, C. Meng, and S. Ermon, “Denoising diffusion implicit models,”
Original
2020
Earlier work this paper cites.
J. L. Chien-Chin Huang, Gu Jin, “Swapadvisor: Pushing deep learning beyond the gpu memory limit via smart swapping,”
2020
Earlier work this paper cites.
Z. Teed and J. Deng, “Raft: Recurrent all-pairs field transforms for optical flow,” in
2020
Earlier work this paper cites.
P. Esser, R. Rombach, and B. Ommer, “Taming transformers for high-resolution image synthesis,” in
2021
Earlier work this paper cites.
A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. Voss, A. Radford, M. Chen, and I. Sutskever, “Zero-shot text-to-image generation,” in
2021
Earlier work this paper cites.
M. Ding, Z. Yang, W. Hong, W. Zheng, C. Zhou, D. Yin, J. Lin, X. Zou, Z. Shao, H. Yang
2021
Earlier work this paper cites.
J. Lin, R. Men, A. Yang, C. Zhou, M. Ding, Y. Zhang, P. Wang, A. Wang, L. Jiang, X. Jia
Original
2021
Earlier work this paper cites.
P. Dhariwal and A. Nichol, “Diffusion models beat gans on image synthesis,”
Original
2021
Earlier work this paper cites.
R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High-resolution image synthesis with latent diffusion models,” 2021
2021
Earlier work this paper cites.
Jianfei Chen, Lianmin Zheng, Zhewei Yao, Dequan Wang, Ion Stoica, Michael W. Mahoney, and Joseph E. Gonzalez, “Actnn: Reducing training memory footprint via 2-bit activation compressed training,”
Original
2021
Earlier work this paper cites.
M. Bain, A. Nagrani, G. Varol, and A. Zisserman, “Frozen in time: A joint video and image encoder for end-to-end retrieval,” in
2021
Earlier work this paper cites.
B. Koonce and B. Koonce, “Efficientnet,”
2021
Earlier work this paper cites.

Similar

J. Yu, Y. Xu, J. Y. Koh, T. Luong, G. Baid, Z. Wang, V. Vasudevan, A. Ku, Y. Yang, B. K. Ayan
Original
2022
Cited alongside, same era.
R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High-resolution image synthesis with latent diffusion models,” in
2022
Cited alongside, same era.
U. Singer, A. Polyak, T. Hayes, X. Yin, J. An, S. Zhang, Q. Hu, H. Yang, O. Ashual, O. Gafni
Original
2022
Cited alongside, same era.
J. Ho, W. Chan, C. Saharia, J. Whang, R. Gao, A. Gritsenko, D. P. Kingma, B. Poole, M. Norouzi, D. J. Fleet
Original
2022
Cited alongside, same era.
D. Zhou, W. Wang, H. Yan, W. Lv, Y. Zhu, and J. Feng, “Magicvideo: Efficient video generation with latent diffusion models,”
Original

Then

Y. Zhao, A. Gu, R. Varma, L. Luo, C. chin Huang, M. Xu, L. Wright, H. Shojanazeri, M. Ott, S. Shleifer, A. Desmaison, C. Balioglu, B. Nguyen, G. Chauhan, Y. Hao, and S. Li, “Pytorch fsdp: Experiences on scaling fully sharded data parallel,” pp. 3848–3860, 2023. [Online]. Available:
2023
Later among the works it cites.
Xiuyu Li, Yijiang Liu, Long Lian, Huanrui Yang, Zhen Dong, Daniel Kang, Shanghang Zhang, and Kurt Keutzer, “QDiffusion: Quantizing diffusion models,”
Original
2023
Later among the works it cites.
Y. Wang, Y. He, Y. Li, K. Li, J. Yu, X. Ma, X. Li, G. Chen, X. Chen, Y. Wang
2023
Later among the works it cites.
K. Li, Y. He, Y. Wang, Y. Li, W. Wang, P. Luo, Y. Wang, L. Wang, and Y. Qiao, “Videochat: Chat-centric video understanding,”
Original
2023
Later among the works it cites.
S. A. Jacobs, M. Tanaka, C. Zhang, M. Zhang, L. Song, S. Rajbhandari, and Y. He, “Deepspeed ulysses: System optimizations for enabling training of extreme long sequence transformer models,”

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

2022

Cited alongside, same era.

J. Ho, W. Chan, C. Saharia, J. Whang, R. Gao, A. Gritsenko, D. P. Kingma, B. Poole, M. Norouzi, D. J. Fleet, and T. Salimans, “Imagen video: High definition video generation with diffusion models,” 2022

2022

Cited alongside, same era.

J. Ho, T. Salimans, A. Gritsenko, W. Chan, M. Norouzi, and D. J. Fleet, “Video diffusion models,” 2022

2022

Cited alongside, same era.

T. Dao, D. Y. Fu, S. Ermon, A. Rudra, and C. R’e, “Flashattention: Fast and memory-efficient exact attention with io-awareness,”

2022

Cited alongside, same era.

W. Peebles and S. Xie, “Scalable diffusion models with transformers,” in

2023

Cited alongside, same era.

J. Chen, J. Yu, C. Ge, L. Yao, E. Xie, Y. Wu, Z. Wang, J. Kwok, P. Luo, H. Lu

2023

Cited alongside, same era.

J. An, S. Zhang, H. Yang, S. Gupta, J.-B. Huang, J. Luo, and X. Yin, “Latent-shift: Latent diffusion with temporal shift for efficient text-to-video generation,”

2023

Cited alongside, same era.

A. Blattmann, R. Rombach, H. Ling, T. Dockhorn, S. W. Kim, S. Fidler, and K. Kreis, “Align your latents: High-resolution video synthesis with latent diffusion models,”

2023

Cited alongside, same era.

2023

Later among the works it cites.

T.-S. Chen, A. Siarohin, W. Menapace, E. Deyneka, H. wei Chao, B. E. Jeon, Y. Fang, H.-Y. Lee, J. Ren, M.-H. Yang, and S. Tulyakov, “Panda-70m: Captioning 70m videos with multiple cross-modality teachers,” 2024

2024

Later among the works it cites.

H. Chen, Y. Zhang, X. Cun, M. Xia, X. Wang, C. Weng, and Y. Shan, “Videocrafter2: Overcoming data limitations for high-quality video diffusion models,” 2024

2024

Later among the works it cites.

NVIDIA, “Context parallelism overview,” 2024, accessed: 2024-11-15. [Online]. Available:

2024

Later among the works it cites.

J. Fang and S. Zhao, “Usp: A unified sequence parallelism approach for long context generative ai,”

2024

Later among the works it cites.

D. Gu, P. Sun, Q. Hu, T. Huang, X. Chen, Y. Xiong, G. Wang, Q. Chen, S. Zhao, J. Fang, Y. Wen, T. Zhang, X. Jin, and X. Liu, “Loongtrain: Efficient training of long-sequence llms with head-context parallelism,”

2024

Later among the works it cites.

P. Esser, S. Kulal, A. Blattmann, R. Entezari, J. Müller, H. Saini, Y. Levi, D. Lorenz, A. Sauer, F. Boesel

2024

Later among the works it cites.

Z. Huang, Y. He, J. Yu, F. Zhang, C. Si, Y. Jiang, Y. Zhang, T. Wu, Q. Jin, N. Chanpaisit

2024

Later among the works it cites.

Y. Zhang, B. Li, h. Liu, Y. j. Lee, L. Gui, D. Fu, J. Feng, Z. Liu, and C. Li, “Llava-next: A strong zero-shot video understanding model,” April 2024. [Online]. Available:

2024

Later among the works it cites.

K. Li, Y. Wang, Y. He, Y. Li, Y. Wang, Y. Liu, Z. Wang, J. Xu, G. Chen, P. Luo

2024

Later among the works it cites.

X. Zhao, S. Cheng, Z. Zheng, Z. Yang, Z. Liu, and Y. You, “Dsp: Dynamic sequence parallelism for multi-dimensional transformers,”

2024

Later among the works it cites.

Z. Huang, F. Zhang, X. Xu, Y. He, J. Yu, Z. Dong, Q. Ma, N. Chanpaisit, C. Si, Y. Jiang, Y. Wang, X. Chen, Y.-C. Chen, L. Wang, D. Lin, Y. Qiao, and Z. Liu, “Vbench++: Comprehensive and versatile benchmark suite for video generative models,”

2024

Later among the works it cites.

J. He, T. Xue, D. Liu, X. Lin, P. Gao, D. Lin, Y. Qiao, W. Ouyang, and Z. Liu, “Venhancer: Generative space-time enhancement for video generation,”

2024

Later among the works it cites.

K. AI, “Kling ai official website,” 2024, accessed: 2024-12-10. [Online]. Available:

2024

Later among the works it cites.

Z. Yang, J. Teng, W. Zheng, M. Ding, S. Huang, J. Xu, Y. Yang, W. Hong, X. Zhang, G. Feng

2024

Later among the works it cites.

P.-Y. Lab and T. A. etc., “Open-sora-plan,” Apr. 2024. [Online]. Available:

2024

Later among the works it cites.

X. Ju, Y. Gao, Z. Zhang, Z. Yuan, X. Wang, A. Zeng, Y. Xiong, Q. Xu, and Y. Shan, “Miradata: A large-scale video dataset with long durations and structured captions,” 2024. [Online]. Available:

2024

Later among the works it cites.