Fetching the paper…

Reading the bibliography…

2025

A Survey of Interactive Generative Video

Yu, Jiwen, Qin, Yiran, Che, Haoxuan et al.

Original
PDF
Post on X

Understand

Interactive Generative Video (IGV) has emerged as a crucial technology in response to the growing demand for high-quality, interactive video content across various domains.

In this paper, we define IGV as a technology that combines generative capabilities to produce diverse high-quality video content with interactive features that enable user engagement through control signals and responsive feedback.
We survey the current landscape of IGV applications, focusing on three major domains: 1) gaming, where IGV enables infinite exploration in virtual worlds; 2) embodied AI, where IGV serves as a physics-aware environment synthesizer for training agents in multimodal interaction with dynamically evolving scenes; and 3) autonomous driving, where IGV provides closed-loop simulation capabilities for safety-critical testing and validation.
To guide future development, we propose a comprehensive framework that decomposes an ideal IGV system into five essential modules: Generation, Control, Memory, Dynamics, and Intelligence.

A Survey of Interactive Generative Video · Around

Built on

E. Santana and G. Hotz, “Learning a driving simulator,” arXiv preprint arXiv:1608.01230 , 2016
Original
2016
Earlier work this paper cites.
R. Goyal, S. Ebrahimi Kahou, V. Michalski, J. Materzynska, S. Westphal, H. Kim, V. Haenel, I. Fruend, P. Yianilos, M. Mueller-Freitag et al. , “The” something something” video database for learning and evaluating visual common sense,” in Proceedings of the IEEE international conference on computer vision , 2017, pp. 5842–5850
2017
Earlier work this paper cites.
A. Chang, A. Dai, T. Funkhouser, M. Halber, M. Niessner, M. Savva, S. Song, A. Zeng, and Y. Zhang, “Matterport3d: Learning from rgb-d data in indoor environments,” arXiv preprint arXiv:1709.06158 , 2017
Original
2017
Earlier work this paper cites.
A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, and V. Koltun, “Carla: An open urban driving simulator,” in Conference on robot learning . PMLR, 2017, pp. 1–16
2017
Earlier work this paper cites.
E. Kolve, R. Mottaghi, W. Han, E. VanderBilt, L. Weihs, A. Herrasti, M. Deitke, K. Ehsani, D. Gordon, Y. Zhu et al. , “Ai2-thor: An interactive 3d environment for visual ai,” arXiv preprint arXiv:1712.05474 , 2017
Original
2017
Earlier work this paper cites.
A. Creswell, T. White, V. Dumoulin, K. Arulkumaran, B. Sengupta, and A. A. Bharath, “Generative adversarial networks: An overview,” IEEE signal processing magazine , vol. 35, no. 1, pp. 53–65, 2018
2018
Earlier work this paper cites.
S. Tulyakov, M.-Y. Liu, X. Yang, and J. Kautz, “Mocogan: Decomposing motion and content for video generation,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2018, pp. 1526–1535
2018
Earlier work this paper cites.
D. Ha and J. Schmidhuber, “Recurrent world models facilitate policy evolution,” Advances in Neural Information Processing Systems , vol. 31, 2018
2018
Earlier work this paper cites.
D. Damen, H. Doughty, G. M. Farinella, S. Fidler, A. Furnari, E. Kazakos, D. Moltisanti, J. Munro, T. Perrett, W. Price et al. , “Scaling egocentric vision: The epic-kitchens dataset,” in Proceedings of the European conference on computer vision (ECCV) , 2018, pp. 720–736
2018
Earlier work this paper cites.
F. Xia, A. R. Zamir, Z. He, A. Sax, J. Malik, and S. Savarese, “Gibson env: Real-world perception for embodied agents,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2018, pp. 9068–9079
2018
Earlier work this paper cites.
Y. Song and S. Ermon, “Generative modeling by estimating gradients of the data distribution,” Advances in neural information processing systems , 2019
2019
Earlier work this paper cites.
D. P. Kingma, M. Welling et al. , “An introduction to variational autoencoders,” Foundations and Trends® in Machine Learning , vol. 12, no. 4, pp. 307–392, 2019
2019
Earlier work this paper cites.
M. Savva, A. Kadian, O. Maksymets, Y. Zhao, E. Wijmans, B. Jain, J. Straub, J. Liu, V. Koltun, J. Malik et al. , “Habitat: A platform for embodied ai research,” in Proceedings of the IEEE/CVF international conference on computer vision , 2019, pp. 9339–9347
2019
Earlier work this paper cites.
W. H. Guss, B. Houghton, N. Topin, P. Wang, C. Codel, M. Veloso, and R. Salakhutdinov, “Minerl: A large-scale dataset of minecraft demonstrations,” arXiv preprint arXiv:1907.13440 , 2019
Original
2019
Earlier work this paper cites.
J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,” Advances in neural information processing systems , 2020
2020
Earlier work this paper cites.
S. W. Kim, Y. Zhou, J. Philion, A. Torralba, and S. Fidler, “Learning to simulate dynamic environments with gamegan,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2020, pp. 1231–1240
2020
Earlier work this paper cites.
J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,” Advances in neural information processing systems , vol. 33, pp. 6840–6851, 2020
2020
Earlier work this paper cites.
M. Chen, A. Radford, R. Child, J. Wu, H. Jun, D. Luan, and I. Sutskever, “Generative pretraining from pixels,” in International conference on machine learning . PMLR, 2020, pp. 1691–1703
2020
Earlier work this paper cites.
H. Caesar, V. Bankiti, A. H. Lang, S. Vora, V. E. Liong, Q. Xu, A. Krishnan, Y. Pan, G. Baldan, and O. Beijbom, “nuscenes: A multimodal dataset for autonomous driving,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2020, pp. 11 621–11 631
2020
Earlier work this paper cites.
P. Sun, H. Kretzschmar, X. Dotiwalla, A. Chouard, V. Patnaik, P. Tsui, J. Guo, Y. Zhou, Y. Chai, B. Caine et al. , “Scalability in perception for autonomous driving: Waymo open dataset,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2020, pp. 2446–2454
2020
Earlier work this paper cites.
S. James, Z. Ma, D. R. Arrojo, and A. J. Davison, “Rlbench: The robot learning benchmark & learning environment,” IEEE Robotics and Automation Letters , vol. 5, no. 2, pp. 3019–3026, 2020
2020
Earlier work this paper cites.
T. Yu, D. Quillen, Z. He, R. Julian, K. Hausman, C. Finn, and S. Levine, “Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning,” in Conference on robot learning . PMLR, 2020, pp. 1094–1100
2020
Earlier work this paper cites.
W. Yan, Y. Zhang, P. Abbeel, and A. Srinivas, “Videogpt: Video generation using vq-vae and transformers,” arXiv preprint arXiv:2104.10157 , 2021
Original
2021
Earlier work this paper cites.
W. Menapace, S. Lathuiliere, S. Tulyakov, A. Siarohin, and E. Ricci, “Playable video generation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2021, pp. 10 061–10 070
2021
Earlier work this paper cites.
S. W. Kim, J. Philion, A. Torralba, and S. Fidler, “Drivegan: Towards a controllable high-quality neural simulation,” in Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2021, pp. 5820–5829
2021
Earlier work this paper cites.
W. Yan, Y. Zhang, P. Abbeel, and A. Srinivas, “Videogpt: Video generation using vq-vae and transformers,” arXiv preprint arXiv:2104.10157 , 2021
Original
2021
Earlier work this paper cites.
C. Wu, L. Huang, Q. Zhang, B. Li, L. Ji, F. Yang, G. Sapiro, and N. Duan, “Godiva: Generating open-domain videos from natural descriptions,” arXiv preprint arXiv:2104.14806 , 2021
Original
2021
Earlier work this paper cites.
F. Ebert, Y. Yang, K. Schmeckpeper, B. Bucher, G. Georgakis, K. Daniilidis, C. Finn, and S. Levine, “Bridge data: Boosting generalization of robotic skills with cross-domain datasets,” arXiv preprint arXiv:2109.13396 , 2021
Original
2021
Earlier work this paper cites.
C. Schuhmann, R. Vencu, R. Beaumont, R. Kaczmarczyk, C. Mullis, A. Katta, T. Coombes, J. Jitsev, and A. Komatsuzaki, “Laion-400m: Open dataset of clip-filtered 400 million image-text pairs,” arXiv preprint arXiv:2111.02114 , 2021
Original
2021
Earlier work this paper cites.
S. K. Ramakrishnan, A. Gokaslan, E. Wijmans, O. Maksymets, A. Clegg, J. Turner, E. Undersander, W. Galuba, A. Westbury, A. X. Chang et al. , “Habitat-matterport 3d dataset (hm3d): 1000 large-scale 3d environments for embodied ai,” arXiv preprint arXiv:2109.08238 , 2021
Original
2021
Earlier work this paper cites.
D. Shah, B. Eysenbach, G. Kahn, N. Rhinehart, and S. Levine, “Rapid exploration for open-world navigation with latent goal models,” arXiv preprint arXiv:2104.05859 , 2021
Original
2021
Earlier work this paper cites.
E. Xing, A. Gupta, S. Powers, and V. Dean, “Kitchenshift: Evaluating zero-shot generalization of imitation-based policy learning under domain shifts,” in NeurIPS 2021 Workshop on Distribution Shifts: Connecting Methods and Applications , 2021
2021
Earlier work this paper cites.
Y. Lipman, R. T. Chen, H. Ben-Hamu, M. Nickel, and M. Le, “Flow matching for generative modeling,” arXiv preprint arXiv:2210.02747 , 2022
Original
2022
Earlier work this paper cites.
X. Liu, C. Gong, and Q. Liu, “Flow straight and fast: Learning to generate and transfer data with rectified flow,” arXiv preprint arXiv:2209.03003 , 2022
Original
2022
Earlier work this paper cites.
W. Menapace, S. Lathuilière, A. Siarohin, C. Theobalt, S. Tulyakov, V. Golyanik, and E. Ricci, “Playable environments: Video manipulation in space and time,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 3584–3593
2022
Earlier work this paper cites.
X. Liu, C. Gong, and Q. Liu, “Flow straight and fast: Learning to generate and transfer data with rectified flow,” arXiv preprint arXiv:2209.03003 , 2022
Original
2022
Earlier work this paper cites.
Y. Lipman, R. T. Chen, H. Ben-Hamu, M. Nickel, and M. Le, “Flow matching for generative modeling,” arXiv preprint arXiv:2210.02747 , 2022
Original
2022
Earlier work this paper cites.
J. Ho, T. Salimans, A. Gritsenko, W. Chan, M. Norouzi, and D. J. Fleet, “Video diffusion models,” Advances in Neural Information Processing Systems , vol. 35, pp. 8633–8646, 2022
2022
Earlier work this paper cites.
W. Harvey, S. Naderiparizi, V. Masrani, C. Weilbach, and F. Wood, “Flexible diffusion modeling of long videos,” Advances in Neural Information Processing Systems , vol. 35, pp. 27 953–27 965, 2022
2022
Earlier work this paper cites.
Y. He, T. Yang, Y. Zhang, Y. Shan, and Q. Chen, “Latent video diffusion models for high-fidelity long video generation,” arXiv preprint arXiv:2211.13221 , 2022
Original
2022
Earlier work this paper cites.
J. Yu, Y. Xu, J. Y. Koh, T. Luong, G. Baid, Z. Wang, V. Vasudevan, A. Ku, Y. Yang, B. K. Ayan et al. , “Scaling autoregressive models for content-rich text-to-image generation,” arXiv preprint arXiv:2206.10789 , vol. 2, no. 3, p. 5, 2022
Original
2022
Earlier work this paper cites.
W. Hong, M. Ding, W. Zheng, X. Liu, and J. Tang, “Cogvideo: Large-scale pretraining for text-to-video generation via transformers,” arXiv preprint arXiv:2205.15868 , 2022
Original
2022
Earlier work this paper cites.
S. Ge, T. Hayes, H. Yang, X. Yin, G. Pang, D. Jacobs, J.-B. Huang, and D. Parikh, “Long video generation with time-agnostic vqgan and time-sensitive transformer,” in European Conference on Computer Vision . Springer, 2022, pp. 102–118
2022
Earlier work this paper cites.
S. Reed, K. Zolna, E. Parisotto, S. G. Colmenarejo, A. Novikov, G. Barth-Maron, M. Gimenez, Y. Sulsky, J. Kay, J. T. Springenberg et al. , “A generalist agent,” arXiv preprint arXiv:2205.06175 , 2022
Original
2022
Earlier work this paper cites.
O. Mees, L. Hermann, E. Rosete-Beas, and W. Burgard, “Calvin: A benchmark for language-conditioned policy learning for long-horizon robot manipulation tasks,” IEEE Robotics and Automation Letters , vol. 7, no. 3, pp. 7327–7334, 2022
2022
Earlier work this paper cites.
W. Menapace, S. Lathuilière, A. Siarohin, C. Theobalt, S. Tulyakov, V. Golyanik, and E. Ricci, “Playable environments: Video manipulation in space and time,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 3584–3593
2022
Earlier work this paper cites.
K. Grauman, A. Westbury, E. Byrne, Z. Chavis, A. Furnari, R. Girdhar, J. Hamburger, H. Jiang, M. Liu, X. Liu et al. , “Ego4d: Around the world in 3,000 hours of egocentric video,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2022, pp. 18 995–19 012
2022
Earlier work this paper cites.
A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, J. Hsu et al. , “Rt-1: Robotics transformer for real-world control at scale,” arXiv preprint arXiv:2212.06817 , 2022
Original
2022
Earlier work this paper cites.
H. Karnan, A. Nair, X. Xiao, G. Warnell, S. Pirk, A. Toshev, J. Hart, J. Biswas, and P. Stone, “Socially compliant navigation dataset (scand): A large-scale dataset of demonstrations for social navigation,” IEEE Robotics and Automation Letters , vol. 7, no. 4, pp. 11 807–11 814, 2022
2022
Earlier work this paper cites.
S. Triest, M. Sivaprakasam, S. J. Wang, W. Wang, A. M. Johnson, and S. Scherer, “Tartandrive: A large-scale dataset for learning off-road dynamics models,” in 2022 International Conference on Robotics and Automation (ICRA) . IEEE, 2022, pp. 2546–2552
2022
Earlier work this paper cites.
J. Mao, T. Lozano-Pérez, J. Tenenbaum, and L. Kaelbling, “Pdsketch: Integrated domain programming, learning, and planning,” Advances in Neural Information Processing Systems , vol. 35, pp. 36 972–36 984, 2022
2022
Earlier work this paper cites.
M. Shridhar, L. Manuelli, and D. Fox, “Cliport: What and where pathways for robotic manipulation,” in Conference on robot learning . PMLR, 2022, pp. 894–906
2022
Earlier work this paper cites.
Y. Jiang, A. Gupta, Z. Zhang, G. Wang, Y. Dou, Y. Chen, L. Fei-Fei, A. Anandkumar, Y. Zhu, and L. Fan, “Vima: General robot manipulation with multimodal prompts,” arXiv preprint arXiv:2210.03094 , vol. 2, no. 3, p. 6, 2022
Original
2022
Earlier work this paper cites.
D. Kondratyuk, L. Yu, X. Gu, J. Lezama, J. Huang, G. Schindler, R. Hornung, V. Birodkar, J. Yan, M.-C. Chiu et al. , “Videopoet: A large language model for zero-shot video generation,” arXiv preprint arXiv:2312.14125 , 2023
Original
2023
Earlier work this paper cites.
P. BAAI, “Plan4mc: Skill reinforcement learning and planning for open-world minecraft tasks,” arXiv preprint arXiv:2303.16563 , 2023
Original
2023
Earlier work this paper cites.
Y. Du, M. Yang, P. Florence, F. Xia, A. Wahid, B. Ichter, P. Sermanet, T. Yu, P. Abbeel, J. B. Tenenbaum et al. , “Video language planning,” arXiv preprint arXiv:2310.10625 , 2023
Original
2023
Earlier work this paper cites.
A. Ajay, S. Han, Y. Du, S. Li, A. Gupta, T. Jaakkola, J. Tenenbaum, L. Kaelbling, A. Srivastava, and P. Agrawal, “Compositional foundation models for hierarchical planning,” Advances in Neural Information Processing Systems , vol. 36, pp. 22 304–22 325, 2023
2023
Earlier work this paper cites.

Similar

Y. Du, S. Yang, B. Dai, H. Dai, O. Nachum, J. Tenenbaum, D. Schuurmans, and P. Abbeel, “Learning universal policies via text-guided video generation,” Advances in neural information processing systems , vol. 36, pp. 9156–9172, 2023
2023
Cited alongside, same era.
M. Yang, Y. Du, K. Ghasemipour, J. Tompson, D. Schuurmans, and P. Abbeel, “Learning interactive real-world simulators,” arXiv preprint arXiv:2310.06114 , 2023
Original
2023
Cited alongside, same era.
A. Hu, L. Russell, H. Yeo, Z. Murez, G. Fedoseev, A. Kendall, J. Shotton, and G. Corrado, “Gaia-1: A generative world model for autonomous driving,” arXiv preprint arXiv:2309.17080 , 2023
Original
2023
Cited alongside, same era.
F. Jia, W. Mao, Y. Liu, Y. Zhao, Y. Wen, C. Zhang, X. Zhang, and T. Wang, “Adriver-i: A general world model for autonomous driving,” arXiv preprint arXiv:2311.13549 , 2023
Original
2023
Cited alongside, same era.

Then

X. Wang, X. Zhang, Z. Luo, Q. Sun, Y. Cui, J. Wang, F. Zhang, Y. Wang, Z. Li, Q. Yu et al. , “Emu3: Next-token prediction is all you need,” arXiv preprint arXiv:2409.18869 , 2024
Original
2024
Later among the works it cites.
Y. Wang, T. Xiong, D. Zhou, Z. Lin, Y. Zhao, B. Kang, J. Feng, and X. Liu, “Loong: Generating minute-level long videos with autoregressive language models,” arXiv preprint arXiv:2410.02757 , 2024
Original
2024
Later among the works it cites.
T. Yin, Q. Zhang, R. Zhang, W. T. Freeman, F. Durand, E. Shechtman, and X. Huang, “From slow bidirectional to fast autoregressive video diffusion models,” arXiv preprint arXiv:2412.07772 , vol. 2, 2024
Original
2024
Later among the works it cites.
Q. Sun, Y. Cui, X. Zhang, F. Zhang, Q. Yu, Y. Wang, Y. Rao, J. Liu, T. Huang, and X. Wang, “Generative multimodal models are in-context learners,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024, pp. 14 398–14 409
2024

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

A. Blattmann, T. Dockhorn, S. Kulal, D. Mendelevitch, M. Kilian, D. Lorenz, Y. Levi, Z. English, V. Voleti, A. Letts et al. , “Stable video diffusion: Scaling latent video diffusion models to large datasets,” arXiv preprint arXiv:2311.15127 , 2023

Original

2023

Cited alongside, same era.

Z. Luo, D. Chen, Y. Zhang, Y. Huang, L. Wang, Y. Shen, D. Zhao, J. Zhou, and T. Tan, “Videofusion: Decomposed diffusion models for high-quality video generation,” arXiv preprint arXiv:2303.08320 , 2023

2023

Cited alongside, same era.

H. Chen, M. Xia, Y. He, Y. Zhang, X. Cun, S. Yang, J. Xing, Y. Liu, Q. Chen, X. Wang et al. , “Videocrafter1: Open diffusion models for high-quality video generation,” arXiv preprint arXiv:2310.19512 , 2023

2023

Cited alongside, same era.

D. Kondratyuk, L. Yu, X. Gu, J. Lezama, J. Huang, G. Schindler, R. Hornung, V. Birodkar, J. Yan, M.-C. Chiu et al. , “Videopoet: A large language model for zero-shot video generation,” arXiv preprint arXiv:2312.14125 , 2023

2023

Cited alongside, same era.

X. Wang, S. Zhang, H. Zhang, Y. Liu, Y. Zhang, C. Gao, and N. Sang, “Videolcm: Video latent consistency model,” arXiv preprint arXiv:2312.09109 , 2023

2023

Cited alongside, same era.

S. Deshmukh, B. Elizalde, R. Singh, and H. Wang, “Pengi: An audio language model for audio tasks,” in Advances in Neural Information Processing Systems , 2023

2023

Cited alongside, same era.

B. Jiang, X. Chen, W. Liu, J. Yu, G. Yu, and T. Chen, “Motiongpt: Human motion as a foreign language,” Advances in Neural Information Processing Systems , vol. 36, pp. 20 067–20 079, 2023

2023

Cited alongside, same era.

B. Liu, Y. Zhu, C. Gao, Y. Feng, Q. Liu, Y. Zhu, and P. Stone, “Libero: Benchmarking knowledge transfer for lifelong robot learning,” Advances in Neural Information Processing Systems , vol. 36, pp. 44 776–44 791, 2023

2023

Cited alongside, same era.

Later among the works it cites.

Y. Ge, S. Zhao, J. Zhu, Y. Ge, K. Yi, L. Song, C. Li, X. Ding, and Y. Shan, “Seed-x: Multimodal models with unified multi-granularity comprehension and generation,” arXiv preprint arXiv:2404.14396 , 2024

2024

Later among the works it cites.

C. Zhou, L. Yu, A. Babu, K. Tirumala, M. Yasunaga, L. Shamis, J. Kahn, X. Ma, L. Zettlemoyer, and O. Levy, “Transfusion: Predict the next token and diffuse images with one multi-modal model,” arXiv preprint arXiv:2408.11039 , 2024

2024

Later among the works it cites.

J. Xie, W. Mao, Z. Bai, D. J. Zhang, W. Wang, K. Q. Lin, Y. Gu, Z. Chen, Z. Yang, and M. Z. Shou, “Show-o: One single transformer to unify multimodal understanding and generation,” arXiv preprint arXiv:2408.12528 , 2024

2024

Later among the works it cites.

T. Li, Y. Tian, H. Li, M. Deng, and K. He, “Autoregressive image generation without vector quantization,” arXiv preprint arXiv:2406.11838 , 2024

2024

Later among the works it cites.

H. Deng, T. Pan, H. Diao, Z. Luo, Y. Cui, H. Lu, S. Shan, Y. Qi, and X. Wang, “Autoregressive video generation without vector quantization,” arXiv preprint arXiv:2412.14169 , 2024

2024

Later among the works it cites.

T. Yin, Q. Zhang, R. Zhang, W. T. Freeman, F. Durand, E. Shechtman, and X. Huang, “From slow bidirectional to fast causal video generators,” arXiv preprint arXiv:2412.07772 , 2024

2024

Later among the works it cites.

D. Ruhe, J. Heek, T. Salimans, and E. Hoogeboom, “Rolling diffusion models,” arXiv preprint arXiv:2402.09470 , 2024

2024

Later among the works it cites.

R. Feng, H. Zhang, Z. Yang, J. Xiao, Z. Shu, Z. Liu, A. Zheng, Y. Huang, Y. Liu, and H. Zhang, “The matrix: Infinite-horizon world generation with real-time moving control,” arXiv preprint arXiv:2412.03568 , 2024

2024

Later among the works it cites.

Decart, J. Quevedo, Q. McIntyre, S. Campbell, X. Chen, and R. Wachen, “Oasis: A universe in a transformer,” 2024. [Online]. Available: https://oasis-model.github.io/

2024

Later among the works it cites.

H. Deng, T. Pan, H. Diao, Z. Luo, Y. Cui, H. Lu, S. Shan, Y. Qi, and X. Wang, “Autoregressive video generation without vector quantization,” arXiv preprint arXiv:2412.14169 , 2024

2024

Later among the works it cites.

J. Chen, H. Cai, J. Chen, E. Xie, S. Yang, H. Tang, M. Li, Y. Lu, and S. Han, “Deep compression autoencoder for efficient high-resolution diffusion models,” arXiv preprint arXiv:2410.10733 , 2024

2024

Later among the works it cites.

L. Yang, B. Kang, Z. Huang, X. Xu, J. Feng, and H. Zhao, “Depth anything: Unleashing the power of large-scale unlabeled data,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024, pp. 10 371–10 381

2024

Later among the works it cites.

Y. Bai, X. Geng, K. Mangalam, A. Bar, A. L. Yuille, T. Darrell, J. Malik, and A. A. Efros, “Sequential modeling enables scalable learning for large vision models,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024, pp. 22 861–22 872

2024

Later among the works it cites.

H. Che, X. He, Q. Liu, C. Jin, and H. Chen, “Gamegen-x: Interactive open-world game video generation,” arXiv preprint arXiv:2411.00769 , 2024

2024

Later among the works it cites.

Z. Wang, Z. Yuan, X. Wang, Y. Li, T. Chen, M. Xia, P. Luo, and Y. Shan, “Motionctrl: A unified and flexible motion controller for video generation,” in ACM SIGGRAPH 2024 Conference Papers , 2024

2024

Later among the works it cites.

H.-X. Yu, H. Duan, J. Hur, K. Sargent, M. Rubinstein, W. T. Freeman, F. Cole, D. Sun, N. Snavely, J. Wu et al. , “Wonderjourney: Going from anywhere to everywhere,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024, pp. 6658–6667

2024

Later among the works it cites.

W. Yu, J. Xing, L. Yuan, W. Hu, X. Li, Z. Huang, X. Gao, T.-T. Wong, Y. Shan, and Y. Tian, “Viewcrafter: Taming video diffusion models for high-fidelity novel view synthesis,” arXiv preprint arXiv:2409.02048 , 2024

2024

Later among the works it cites.

B. Ma, H. Gao, H. Deng, Z. Luo, T. Huang, L. Tang, and X. Wang, “You see it, you got it: Learning 3d creation on pose-free videos at scale,” arXiv preprint arXiv:2412.06699 , 2024

2024

Later among the works it cites.

A. Montanaro, L. Savant Aira, E. Aiello, D. Valsesia, and E. Magli, “Motioncraft: Physics-based zero-shot video generation,” Advances in Neural Information Processing Systems , vol. 37, pp. 123 155–123 181, 2024

2024

Later among the works it cites.

S. Liu, Z. Ren, S. Gupta, and S. Wang, “Physgen: Rigid-body physics-grounded image-to-video generation,” in European Conference on Computer Vision . Springer, 2024, pp. 360–378

2024

Later among the works it cites.

T. Zhang, H.-X. Yu, R. Wu, B. Y. Feng, C. Zheng, N. Snavely, J. Wu, and W. T. Freeman, “Physdreamer: Physics-based interaction with 3d objects via video generation,” in European Conference on Computer Vision . Springer, 2024, pp. 388–406

2024

Later among the works it cites.

W. Menapace, A. Siarohin, S. Lathuilière, P. Achlioptas, V. Golyanik, S. Tulyakov, and E. Ricci, “Promptable game models: Text-guided game simulation via masked diffusion models,” ACM Transactions on Graphics , vol. 43, no. 2, pp. 1–16, 2024

2024

Later among the works it cites.

V. Protocol, “Video game generation: A practical study using mario,” 2024, preprint. [Online]. Available: https://github.com/Virtual-Protocol/mario-videogamegen/blob/main/static/pdfs/VideoGameGen.pdf

2024

Later among the works it cites.

N. Kazemi, N. Savov, D. Paudel, and L. Van Gool, “Learning generative interactive environments by trained agent exploration,” arXiv preprint arXiv:2409.06445 , 2024

2024

Later among the works it cites.

M. Yang, J. Li, Z. Fang, S. Chen, Y. Yu, Q. Fu, W. Yang, and D. Ye, “Playable game generation,” arXiv preprint arXiv:2412.00887 , 2024

2024

Later among the works it cites.

T. Lu, T. Shu, J. Xiao, L. Ye, J. Wang, C. Peng, C. Wei, D. Khashabi, R. Chellappa, A. Yuille et al. , “Genex: Generating an explorable world,” arXiv preprint arXiv:2412.09624 , 2024

2024

Later among the works it cites.

A. Bar, G. Zhou, D. Tran, T. Darrell, and Y. LeCun, “Navigation world models,” arXiv preprint arXiv:2412.03572 , 2024

2024

Later among the works it cites.

E. Zhou, Y. Qin, Z. Yin, Y. Huang, R. Zhang, L. Sheng, Y. Qiao, and J. Shao, “Minedreamer: Learning to follow instructions via chain-of-imagination for simulated-world control,” arXiv preprint arXiv:2403.12037 , 2024

2024

Later among the works it cites.

C. Chi, Z. Xu, C. Pan, E. Cousineau, B. Burchfiel, S. Feng, R. Tedrake, and S. Song, “Universal manipulation interface: In-the-wild robot teaching without in-the-wild robots,” arXiv preprint arXiv:2402.10329 , 2024

2024

Later among the works it cites.

W. Huang, C. Wang, Y. Li, R. Zhang, and L. Fei-Fei, “Rekep: Spatio-temporal reasoning of relational keypoint constraints for robotic manipulation,” arXiv preprint arXiv:2409.01652 , 2024

2024

Later among the works it cites.

J. Yang, S. Gao, Y. Qiu, L. Chen, T. Li, B. Dai, K. Chitta, P. Wu, J. Zeng, P. Luo et al. , “Generalized predictive model for autonomous driving,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024, pp. 14 662–14 672

2024

Later among the works it cites.

T. Seawead, C. Yang, Z. Lin, Y. Zhao, S. Lin, Z. Ma, H. Guo, H. Chen, L. Qi, S. Wang et al. , “Seaweed-7b: Cost-effective training of video generation foundation model,” arXiv preprint arXiv:2504.08685 , 2025

2025

Closest in time.

A. Wang, B. Ai, B. Wen, C. Mao, C.-W. Xie, D. Chen, F. Yu, H. Zhao, J. Yang, J. Zeng et al. , “Wan: Open and advanced large-scale video generative models,” arXiv preprint arXiv:2503.20314 , 2025

2025

Closest in time.

J. Yu, Y. Qin, X. Wang, P. Wan, D. Zhang, and X. Liu, “Gamefactory: Creating new games with generative interactive videos,” 2025

2025

Closest in time.

DeepMind, “Genie 2: A large-scale foundation world model,” 2025, accessed: 2025-01-17. [Online]. Available: https://deepmind.google/discover/blog/genie-2-a-large-scale-foundation-world-model/

2025

Closest in time.

A. Kanervisto, D. Bignell, L. Y. Wen, M. Grayson, R. Georgescu, S. Valcarcel Macua, S. Z. Tan, T. Rashid, T. Pearce, Y. Cao et al. , “World and human action models towards gameplay ideation,” Nature , vol. 638, no. 8051, pp. 656–663, 2025

2025

Closest in time.

J. Guo, Y. Ye, T. He, H. Wu, Y. Jiang, T. Pearce, and J. Bian, “Mineworld: a real-time and open-source interactive world model on minecraft,” arXiv preprint arXiv:2504.08388 , 2025

2025

Closest in time.

J. Chen, Y. Zhao, Y. Huang, L. Cui, L. Dong, T. Lv, Q. Chen, and F. Wei, “Model as a game: On numerical and spatial consistency for generative games,” arXiv preprint arXiv:2503.21172 , 2025

2025

Closest in time.

S. Gao, S. Zhou, Y. Du, J. Zhang, and C. Gan, “Adaworld: Learning adaptable world models with latent actions,” arXiv preprint arXiv:2503.18938 , 2025

2025

Closest in time.

Y. Qin, A. Sun, Y. Hong, B. Wang, and R. Zhang, “Navigatediff: Visual predictors are zero-shot navigation assistants,” arXiv preprint arXiv:2502.13894 , 2025

2025

Closest in time.

NVIDIA, “Cosmos world foundation model platform for physical ai,” arXiv preprint arXiv:2501.03575 , 2025

2025

Closest in time.

S. Li, Y. Gao, D. Sadigh, and S. Song, “Unified video action model,” arXiv preprint arXiv:2503.00200 , 2025

2025

Closest in time.

G. Zhao, X. Wang, Z. Zhu, X. Chen, G. Huang, X. Bao, and X. Wang, “Drivedreamer-2: Llm-enhanced world models for diverse driving video generation,” in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 39, no. 10, 2025, pp. 10 412–10 420

2025

Closest in time.

L. Russell, A. Hu, L. Bertoni, G. Fedoseev, J. Shotton, E. Arani, and G. Corrado, “Gaia-2: A controllable multi-view generative world model for autonomous driving,” arXiv preprint arXiv:2503.20523 , 2025

2025

Closest in time.

Y. Ji, Z. Zhu, Z. Zhu, K. Xiong, M. Lu, Z. Li, L. Zhou, H. Sun, B. Wang, and T. Lu, “Cogen: 3d consistent video generation via adaptive conditioning for autonomous driving,” arXiv preprint arXiv:2503.22231 , 2025

2025

Closest in time.

J. Ni, Y. Guo, Y. Liu, R. Chen, L. Lu, and Z. Wu, “Maskgwm: A generalizable driving world model with video mask reconstruction,” arXiv preprint arXiv:2502.11663 , 2025

2025

Closest in time.

J. Yu, Y. Qin, H. Che, Q. Liu, X. Wang, P. Wan, D. Zhang, and X. Liu, “Position: Interactive generative video as next-generation game engine,” arXiv preprint arXiv:2503.17359 , 2025

2025

Closest in time.

S. Chen, C. Ge, Y. Zhang, Y. Zhang, F. Zhu, H. Yang, H. Hao, H. Wu, Z. Lai, Y. Hu et al. , “Goku: Flow based video generative foundation models,” arXiv preprint arXiv:2502.04896 , 2025

2025

Closest in time.

M. Sun, W. Wang, G. Li, J. Liu, J. Sun, W. Feng, S. Lao, S. Zhou, Q. He, and J. Liu, “Ar-diffusion: Asynchronous video generation with auto-regressive diffusion,” arXiv preprint arXiv:2503.07418 , 2025

2025

Closest in time.

K. Song, B. Chen, M. Simchowitz, Y. Du, R. Tedrake, and V. Sitzmann, “History-guided video diffusion,” arXiv preprint arXiv:2502.06764 , 2025

2025

Closest in time.

B.-K. Kim, H.-K. Song, T. Castells, and S. Choi, “Bk-sdm: A lightweight, fast, and cheap version of stable diffusion,” in European Conference on Computer Vision . Springer, 2025, pp. 381–399

2025

Closest in time.

Y. Qin, L. Kang, X. Song, Z. Yin, X. Liu, X. Liu, R. Zhang, and L. Bai, “Robofactory: Exploring embodied agent collaboration with compositional constraints,” arXiv preprint arXiv:2503.16408 , 2025

2025

Closest in time.