Fetching the paper…

Reading the bibliography…

2025

Long-Context Autoregressive Video Modeling with Next-Frame Prediction

Gu, Yuchao, Mao, Weijia, Shou, Mike Zheng

Original
PDF
Post on X

Understand

Long-context video modeling is essential for enabling generative models to function as world simulators, as they must maintain temporal coherence over extended time spans.

However, most existing models are trained on short clips, limiting their ability to capture long-range dependencies, even with test-time extrapolation.
While training directly on long videos is a natural solution, the rapid growth of vision tokens makes it computationally prohibitive.
To support exploring efficient long-context video modeling, we first establish a strong autoregressive baseline called Frame AutoRegressive (FAR).

Long-Context Autoregressive Video Modeling with Next-Frame Prediction · Around

Built on

K. Soomro, A. R. Zamir, and M. Shah, “Ucf101: A dataset of 101 human actions classes from videos in the wild,” arXiv preprint arXiv:1212.0402 , 2012
Original
2012
Earlier work this paper cites.
F. Ebert, C. Finn, A. X. Lee, and S. Levine, “Self-supervised visual planning with temporal skip connections.” CoRL , vol. 12, no. 16, p. 23, 2017
2017
Earlier work this paper cites.
T. Unterthiner, S. Van Steenkiste, K. Kurach, R. Marinier, M. Michalski, and S. Gelly, “Fvd: A new metric for video generation,” 2019
2019
Earlier work this paper cites.
O. Press, N. A. Smith, and M. Lewis, “Train short, test long: Attention with linear biases enables input length extrapolation,” arXiv preprint arXiv:2108.12409 , 2021
Original
2021
Earlier work this paper cites.
M. Babaeizadeh, M. T. Saffar, S. Nair, S. Levine, C. Finn, and D. Erhan, “Fitvid: Overfitting in pixel-level video prediction,” arXiv preprint arXiv:2106.13195 , 2021
Original
2021
Earlier work this paper cites.
V. Saxena, J. Ba, and D. Hafner, “Clockwork variational autoencoders,” Advances in Neural Information Processing Systems , vol. 34, pp. 29 246–29 257, 2021
2021
Earlier work this paper cites.
W. Hong, M. Ding, W. Zheng, X. Liu, and J. Tang, “Cogvideo: Large-scale pretraining for text-to-video generation via transformers,” arXiv preprint arXiv:2205.15868 , 2022
Original
2022
Earlier work this paper cites.
C. Hawthorne, A. Jaegle, C. Cangea, S. Borgeaud, C. Nash, M. Malinowski, S. Dieleman, O. Vinyals, M. Botvinick, I. Simon et al. , “General-purpose, long-context autoregressive modeling with perceiver ar,” in International Conference on Machine Learning . PMLR, 2022, pp. 8535–8558
2022
Earlier work this paper cites.
W. Harvey, S. Naderiparizi, V. Masrani, C. Weilbach, and F. Wood, “Flexible diffusion modeling of long videos,” Advances in Neural Information Processing Systems , vol. 35, pp. 27 953–27 965, 2022
2022
Earlier work this paper cites.
X. Liu, C. Gong, and Q. Liu, “Flow straight and fast: Learning to generate and transfer data with rectified flow,” arXiv preprint arXiv:2209.03003 , 2022
Original
2022
Earlier work this paper cites.
Y. Lipman, R. T. Chen, H. Ben-Hamu, M. Nickel, and M. Le, “Flow matching for generative modeling,” arXiv preprint arXiv:2210.02747 , 2022
Original
2022
Earlier work this paper cites.
M. S. Albergo and E. Vanden-Eijnden, “Building normalizing flows with stochastic interpolants,” arXiv preprint arXiv:2209.15571 , 2022
Original
2022
Earlier work this paper cites.
S. Ge, T. Hayes, H. Yang, X. Yin, G. Pang, D. Jacobs, J.-B. Huang, and D. Parikh, “Long video generation with time-agnostic vqgan and time-sensitive transformer,” in European Conference on Computer Vision . Springer, 2022, pp. 102–118
2022
Earlier work this paper cites.
Y. He, T. Yang, Y. Zhang, Y. Shan, and Q. Chen, “Latent video diffusion models for high-fidelity long video generation,” arXiv preprint arXiv:2211.13221 , 2022
Original
2022
Earlier work this paper cites.
V. Voleti, A. Jolicoeur-Martineau, and C. Pal, “Mcvd-masked conditional video diffusion for prediction, generation, and interpolation,” Advances in neural information processing systems , vol. 35, pp. 23 371–23 385, 2022
2022
Earlier work this paper cites.
T. Höppe, A. Mehrjou, S. Bauer, D. Nielsen, and A. Dittadi, “Diffusion models for video prediction and infilling,” arXiv preprint arXiv:2206.07696 , 2022
Original
2022
Earlier work this paper cites.
Y. Guo, C. Yang, A. Rao, Z. Liang, Y. Wang, Y. Qiao, M. Agrawala, D. Lin, and B. Dai, “Animatediff: Animate your personalized text-to-image diffusion models without specific tuning,” arXiv preprint arXiv:2307.04725 , 2023
Original
2023
Earlier work this paper cites.
F.-Y. Wang, W. Chen, G. Song, H.-J. Ye, Y. Liu, and H. Li, “Gen-l-video: Multi-text to long video generation via temporal co-denoising,” arXiv preprint arXiv:2305.18264 , 2023
Original
2023
Earlier work this paper cites.
L. Yu, J. Lezama, N. B. Gundavarapu, L. Versari, K. Sohn, D. Minnen, Y. Cheng, V. Birodkar, A. Gupta, X. Gu et al. , “Language model beats diffusion–tokenizer is key to visual generation,” arXiv preprint arXiv:2310.05737 , 2023
Original
2023
Earlier work this paper cites.
D. Kondratyuk, L. Yu, X. Gu, J. Lezama, J. Huang, G. Schindler, R. Hornung, V. Birodkar, J. Yan, M.-C. Chiu et al. , “Videopoet: A large language model for zero-shot video generation,” arXiv preprint arXiv:2312.14125 , 2023
Original
2023
Earlier work this paper cites.

Similar

T. Wu, Z. Fan, X. Liu, H.-T. Zheng, Y. Gong, J. Jiao, J. Li, J. Guo, N. Duan, W. Chen et al. , “Ar-diffusion: Auto-regressive diffusion model for text generation,” Advances in Neural Information Processing Systems , vol. 36, pp. 39 957–39 974, 2023
2023
Cited alongside, same era.
B. Peng, J. Quesnelle, H. Fan, and E. Shippole, “Yarn: Efficient context window extension of large language models,” arXiv preprint arXiv:2309.00071 , 2023
Original
2023
Cited alongside, same era.
bloc97, “NTK-Aware Scaled RoPE allows LLaMA models to have extended (8k+) context size without any fine-tuning and minimal perplexity degradation.” 2023. [Online]. Available: https://www.reddit.com/r/LocalLLaMA/comments/14lz7j5/ntkaware_scaled_rope_allows_llama_models_to_have/
2023
Cited alongside, same era.
S. Chen, S. Wong, L. Chen, and Y. Tian, “Extending context window of large language models via positional interpolation,” arXiv preprint arXiv:2306.15595 , 2023
Original
2023

Then

L. Barrault, P.-A. Duquenne, M. Elbayad, A. Kozhevnikov, B. Alastruey, P. Andrews, M. Coria, G. Couairon, M. R. Costa-jussà, D. Dale et al. , “Large concept models: Language modeling in a sentence representation space,” arXiv e-prints , pp. arXiv–2412, 2024
2024
Later among the works it cites.
J. Hu, S. Hu, Y. Song, Y. Huang, M. Wang, H. Zhou, Z. Liu, W.-Y. Ma, and M. Sun, “Acdit: Interpolating autoregressive conditional modeling and diffusion transformer,” arXiv preprint arXiv:2412.07720 , 2024
Original
2024
Later among the works it cites.
D. Valevski, Y. Leviathan, M. Arar, and S. Fruchter, “Diffusion models are real-time game engines,” arXiv preprint arXiv:2408.14837 , 2024
Original
2024
Later among the works it cites.
J. Bruce, M. D. Dennis, A. Edwards, J. Parker-Holder, Y. Shi, E. Hughes, M. Lai, A. Mavalankar, R. Steigerwald, C. Apps et al. , “Genie: Generative interactive environments,” in Forty-first International Conference on Machine Learning , 2024
2024

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Cited alongside, same era.

Y. Chen, S. Qian, H. Tang, X. Lai, Z. Liu, S. Han, and J. Jia, “Longlora: Efficient fine-tuning of long-context large language models,” arXiv preprint arXiv:2309.12307 , 2023

2023

Cited alongside, same era.

W. Yan, D. Hafner, S. James, and P. Abbeel, “Temporally consistent transformers for video generation,” in International Conference on Machine Learning . PMLR, 2023, pp. 39 062–39 098

2023

Cited alongside, same era.

W. Peebles and S. Xie, “Scalable diffusion models with transformers,” in Proceedings of the IEEE/CVF international conference on computer vision , 2023, pp. 4195–4205

2023

Cited alongside, same era.

H. Ni, C. Shi, K. Li, S. X. Huang, and M. R. Min, “Conditional image-to-video generation with latent flow diffusion models,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2023, pp. 18 444–18 455

2023

Cited alongside, same era.

K. Mei and V. Patel, “Vidm: Video implicit diffusion models,” in Proceedings of the AAAI conference on artificial intelligence , vol. 37, no. 8, 2023, pp. 9117–9125

2023

Cited alongside, same era.

T. Brooks, B. Peebles, C. Holmes, W. DePue, Y. Guo, L. Jing, D. Schnurr, J. Taylor, T. Luhman, E. Luhman, C. Ng, R. Wang, and A. Ramesh, “Video generation models as world simulators,” 2024. [Online]. Available: https://openai.com/research/video-generation-models-as-world-simulators

2024

Cited alongside, same era.

Y. Lu, Y. Liang, L. Zhu, and Y. Yang, “Freelong: Training-free long video generation with spectralblend temporal attention,” arXiv preprint arXiv:2407.19918 , 2024

2024

Cited alongside, same era.

Y. Jin, Z. Sun, N. Li, K. Xu, H. Jiang, N. Zhuang, Q. Huang, Y. Song, Y. Mu, and Z. Lin, “Pyramidal flow matching for efficient video generative modeling,” arXiv preprint arXiv:2410.05954 , 2024

Original

2024

Cited alongside, same era.

Later among the works it cites.

J. Parker-Holder, P. Ball, J. Bruce, V. Dasagi, K. Holsheimer, C. Kaplanis, A. Moufarek, G. Scully, J. Shar, J. Shi, S. Spencer, J. Yung, M. Dennis, S. Kenjeyev, S. Long, V. Mnih, H. Chan, M. Gazeau, B. Li, F. Pardo, L. Wang, L. Zhang, F. Besse, T. Harley, A. Mitenkova, J. Wang, J. Clune, D. Hassabis, R. Hadsell, A. Bolton, S. Singh, and T. Rocktäschel, “Genie 2: A large-scale foundation world model,” 2024. [Online]. Available: https://deepmind.google/discover/blog/genie-2-a-large-scale-foundation-world-model/

2024

Later among the works it cites.

N. Ma, M. Goldstein, M. S. Albergo, N. M. Boffi, E. Vanden-Eijnden, and S. Xie, “Sit: Exploring flow and diffusion-based generative models with scalable interpolant transformers,” in European Conference on Computer Vision . Springer, 2024, pp. 23–40

2024

Later among the works it cites.

X. Ma, Y. Wang, G. Jia, X. Chen, Z. Liu, Y.-F. Li, C. Chen, and Y. Qiao, “Latte: Latent diffusion transformer for video generation,” arXiv preprint arXiv:2401.03048 , 2024

2024

Later among the works it cites.

P. Esser, S. Kulal, A. Blattmann, R. Entezari, J. Müller, H. Saini, Y. Levi, D. Lorenz, A. Sauer, F. Boesel et al. , “Scaling rectified flow transformers for high-resolution image synthesis,” in Forty-first international conference on machine learning , 2024

2024

Later among the works it cites.

J. Su, M. Ahmed, Y. Lu, S. Pan, W. Bo, and Y. Liu, “Roformer: Enhanced transformer with rotary position embedding,” Neurocomputing , vol. 568, p. 127063, 2024

2024

Later among the works it cites.

J. Chen, H. Cai, J. Chen, E. Xie, S. Yang, H. Tang, M. Li, Y. Lu, and S. Han, “Deep compression autoencoder for efficient high-resolution diffusion models,” arXiv preprint arXiv:2410.10733 , 2024

2024

Later among the works it cites.

J. Wang, Y. Jiang, Z. Yuan, B. Peng, Z. Wu, and Y.-G. Jiang, “Omnitokenizer: A joint image-video tokenizer for visual generation,” arXiv preprint arXiv:2406.09399 , 2024

2024

Later among the works it cites.

Z. Zhang, J. Hu, W. Cheng, D. Paudel, and J. Yang, “Extdm: Distribution extrapolation diffusion model for video prediction,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024, pp. 19 310–19 320

2024

Later among the works it cites.

A. Wang, B. Ai, B. Wen, C. Mao, C.-W. Xie, D. Chen, F. Yu, H. Zhao, J. Yang, J. Zeng et al. , “Wan: Open and advanced large-scale video generative models,” arXiv preprint arXiv:2503.20314 , 2025

2025

Closest in time.

N. Agarwal, A. Ali, M. Bala, Y. Balaji, E. Barker, T. Cai, P. Chattopadhyay, Y. Chen, Y. Cui, Y. Ding et al. , “Cosmos world foundation model platform for physical ai,” arXiv preprint arXiv:2501.03575 , 2025

2025

Closest in time.

M. Zhao, G. He, Y. Chen, H. Zhu, C. Li, and J. Zhu, “Riflex: A free lunch for length extrapolation in video diffusion transformers,” arXiv preprint arXiv:2502.15894 , 2025

2025

Closest in time.

Y. Guo, C. Yang, Z. Yang, Z. Ma, Z. Lin, Z. Yang, D. Lin, and L. Jiang, “Long context tuning for video generation,” arXiv preprint arXiv:2503.10589 , 2025

2025

Closest in time.

K. Dalal, D. Koceja, G. Hussein, J. Xu, Y. Zhao, Y. Song, S. Han, K. C. Cheung, J. Kautz, C. Guestrin et al. , “One-minute video generation with test-time training,” arXiv preprint arXiv:2504.05298 , 2025

2025

Closest in time.

B. Chen, D. Martí Monsó, Y. Du, M. Simchowitz, R. Tedrake, and V. Sitzmann, “Diffusion forcing: Next-token prediction meets full-sequence diffusion,” Advances in Neural Information Processing Systems , vol. 37, pp. 24 081–24 125, 2025

2025

Closest in time.

T. Li, Y. Tian, H. Li, M. Deng, and K. He, “Autoregressive image generation without vector quantization,” Advances in Neural Information Processing Systems , vol. 37, pp. 56 424–56 445, 2025

2025

Closest in time.

D. Zhou, Q. Sun, Y. Peng, K. Yan, R. Dong, D. Wang, Z. Ge, N. Duan, X. Zhang, L. M. Ni et al. , “Taming teacher forcing for masked autoregressive video generation,” arXiv preprint arXiv:2501.12389 , 2025

2025

Closest in time.