DSV: Exploiting Dynamic Sparsity to Accelerate Large-Scale Video DiT Training

UCF101: A dataset of 101 human actions classes from videos in the wild

Original

K Soomro. 2012 · 2012

Earlier work this paper cites.

Training deep nets with sublinear memory cost

Original

Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. 2016 · 2016

Earlier work this paper cites.

Attention is all you need. In NIPS

A Waswani, N Shazeer, N Parmar, J Uszkoreit, L Jones, A Gomez, L Kaiser, and I Polosukhin. 2017 · 2017

Earlier work this paper cites.

Towards accurate generative models of video: A new metric & challenges

Original

Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. 2018 · 2018

Earlier work this paper cites.

Information aggregation for multi-head attention with routing-by-agreement

Original

Jian Li, Baosong Yang, Zi-Yi Dou, Xing Wang, Michael R Lyu, and Zhaopeng Tu. 2019 · 2019

Earlier work this paper cites.

Analyzing the structure of attention in a transformer language model

Original

Jesse Vig and Yonatan Belinkov. 2019 · 2019

Earlier work this paper cites.

An image is worth 16x16 words: Transformers for image recognition at scale

Original

Alexey Dosovitskiy. 2020 · 2020

Earlier work this paper cites.

Denoising diffusion probabilistic models. In Proc. NeurIPS

Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020 · 2020

Earlier work this paper cites.

Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval. In Proc. IEEE/CVF ICCV

Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. 2021 · 2021

Earlier work this paper cites.

Flashattention: Fast and memory-efficient exact attention with io-awareness. In Advances in Neural Information Processing Systems

Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. 2022 · 2022

Earlier work this paper cites.

Classifier-free diffusion guidance

Original

Jonathan Ho and Tim Salimans. 2022 · 2022

Earlier work this paper cites.

Cogvideo: Large-scale pretraining for text-to-video generation via transformers

Original

Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. 2022 · 2022

Earlier work this paper cites.

Flow matching for generative modeling

Original

Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. 2022 · 2022

Earlier work this paper cites.

Pixartalpha : Fast training of diffusion transformer for photorealistic text-to-image synthesis

Original

Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, et al · 2023

Earlier work this paper cites.

Flashattention-2: Faster attention with better parallelism and work partitioning

Original

Tri Dao. 2023 · 2023

Earlier work this paper cites.

Neighborhood attention transformer. In Proc. IEEE/CVF CVPR

Ali Hassani, Steven Walton, Jiachen Li, Shen Li, and Humphrey Shi. 2023 · 2023

Earlier work this paper cites.

Deepspeed ulysses: System optimizations for enabling training of extreme long sequence transformer models

Original

Sam Ade Jacobs, Masahiro Tanaka, Chengming Zhang, Minjia Zhang, Shuaiwen Leon Song, Samyam Rajbhandari, and Yuxiong He. 2023 · 2023

Earlier work this paper cites.

Ring attention with blockwise transformers for near-infinite context

Original

Hao Liu, Matei Zaharia, and Pieter Abbeel. 2023 · 2023

Earlier work this paper cites.

GPT-4 Technical Report

Original

OpenAI. 2023 · 2023

Cited alongside, same era.

Scalable diffusion models with transformers. In Proc. IEEE/CVF ICCV

William Peebles and Saining Xie. 2023 · 2023

Cited alongside, same era.

Efficient streaming language models with attention sinks

Original

Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. 2023 · 2023

Cited alongside, same era.

H2o: Heavy-hitter oracle for efficient generative inference of large language models. In Proc. NeurIPS

Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, et al · 2023

Cited alongside, same era.

Pytorch fsdp: experiences on scaling fully sharded data parallel

Original

Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, et al · 2023

Vbench: Comprehensive benchmark suite for video generative models. In Proc. IEEE/CVF CVPR

Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, et al · 2024

Later among the works it cites.

Minference 1.0: Accelerating pre-filling for long-context llms via dynamic sparse attention

Original

Huiqiang Jiang, Yucheng Li, Chengruidong Zhang, Qianhui Wu, Xufang Luo, Surin Ahn, Zhenhua Han, Amir H Abdi, Dongsheng Li, Chin-Yew Lin, et al · 2024

Later among the works it cites.

Moh: Multi-head attention as mixture-of-head attention

Original

Peng Jin, Bo Zhu, Li Yuan, and Shuicheng Yan. 2024 · 2024

Later among the works it cites.

Adaptive caching for faster video generation with diffusion transformers

Original

Kumara Kahatapitiya, Haozhe Liu, Sen He, Ding Liu, Menglin Jia, Michael S Ryoo, and Tian Xie. 2024 · 2024

Later among the works it cites.