Can VLMs Predict Future States? Bootstrapping World Models from Inverse Dynamics

A short note on the kinetics-700 human action dataset

Original

Joao Carreira, Eric Noland, Chloe Hillier, and Andrew Zisserman. 2019 · 1907

Earlier work this paper cites.

Learning to predict by the methods of temporal differences

Richard S Sutton. 1988 · 1988

Earlier work this paper cites.

BLEU: a method for automatic evaluation of machine translation

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002 · 2002

Earlier work this paper cites.

ROUGE: A package for automatic evaluation of summaries

Chin-Yew Lin. 2004 · 2004

Earlier work this paper cites.

UCF101: A dataset of 101 human actions classes from videos in the wild

Original

Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. 2012 · 2012

Earlier work this paper cites.

The kinetics human action video dataset

Original

Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, and 1 others. 2017 · 2017

Earlier work this paper cites.

Scaling egocentric vision: The EPIC-KITCHENS dataset

Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and 1 others. 2018 · 2018

Earlier work this paper cites.

Recurrent world models facilitate policy evolution

David Ha and Jürgen Schmidhuber. 2018 · 2018

Earlier work this paper cites.

Moments in time dataset: one million videos for event understanding

Mathew Monfort, Alex Andonian, Bolei Zhou, Kandan Ramakrishnan, Sarah Adel Bargal, Tom Yan, Lisa Brown, Quanfu Fan, Dan Gutfruend, Carl Vondrick, and 1 others. 2019 · 2019

Earlier work this paper cites.

Transformers: State-of-the-art natural language processing

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, and 3 others. 2020 · 2020

Earlier work this paper cites.

BERTScore: Evaluating text generation with bert

Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2020 · 2020

Earlier work this paper cites.

Can language models encode perceptual structure without grounding? A case study in color

Mostafa Abdou, Artur Kulmizev, Daniel Hershcovich, Stella Frank, Ellie Pavlick, and Anders Søgaard. 2021 · 2021

Earlier work this paper cites.

Decision transformer: Reinforcement learning via sequence modeling

Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. 2021 · 2021

Earlier work this paper cites.

Video PreTraining (VPT): Learning to act by watching unlabeled online videos

Bowen Baker, Ilge Akkaya, Peter Zhokov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampedro, and Jeff Clune. 2022 · 2022

Earlier work this paper cites.

Kubric: A scalable dataset generator

Klaus Greff, Francois Belletti, Lucas Beyer, Carl Doersch, Yilun Du, Daniel Duckworth, David J Fleet, Dan Gnanapragasam, Florian Golemo, Charles Herrmann, and 1 others. 2022 · 2022

Earlier work this paper cites.

LoRA: Low-rank adaptation of large language models

Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, and 1 others. 2022 · 2022

Earlier work this paper cites.

Inner Monologue: Embodied reasoning through planning with language models

Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, and 1 others. 2022 · 2022

Earlier work this paper cites.

Transformers are sample-efficient world models

Vincent Micheli, Eloi Alonso, and François Fleuret. 2022 · 2022

Cited alongside, same era.

Do as I can, not as I say: Grounding language in robotic affordances

Anthony Brohan, Yevgen Chebotar, Chelsea Finn, Karol Hausman, Alexander Herzog, Daniel Ho, Julian Ibarz, Alex Irpan, Eric Jang, Ryan Julian, and 1 others. 2023 · 2023

Cited alongside, same era.

InstructPix2Pix: Learning to follow image editing instructions

Tim Brooks, Aleksander Holynski, and Alexei A Efros. 2023 · 2023

Cited alongside, same era.

Transformer-based world models are happy with 100k interactions

Original

Jan Robine, Marc Höftmann, Tobias Uelwer, and Stefan Harmeling. 2023 · 2023

Cited alongside, same era.

Learning interactive real-world simulators

Original

Mengjiao Yang, Yilun Du, Kamyar Ghasemipour, Jonathan Tompson, Dale Schuurmans, and Pieter Abbeel. 2023 · 2023

Cited alongside, same era.

World model on million-length video and language with blockwise RingAttention

Original

Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel. 2024 · 2024

Later among the works it cites.

WorldSimBench: Towards video generation models as world simulators

Original

Yiran Qin, Zhelun Shi, Jiwen Yu, Xijun Wang, Enshen Zhou, Lijun Li, Zhenfei Yin, Xihui Liu, Lu Sheng, Jing Shao, and 1 others. 2024 · 2024

Later among the works it cites.

Are large language model temporally grounded?

Yifu Qiu, Zheng Zhao, Yftah Ziser, Anna Korhonen, Edoardo Ponti, and Shay B Cohen. 2024 · 2024

Later among the works it cites.

Scaling LLM test-time compute optimally can be more effective than scaling model parameters

Original

Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. 2024 · 2024

Later among the works it cites.

VideoAgent: Self-improving video generation