TraveLER: A Modular Multi-LMM Agent Framework for Video Question-Answering

Language models are few-shot learners

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020 · 1901

Earlier work this paper cites.

Learning object detection from captions via textual scene attributes

Original

Achiya Jerbi, Roei Herzig, Jonathan Berant, Gal Chechik, and Amir Globerson. 2020 · 2009

Earlier work this paper cites.

Neural module networks

Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. 2016 · 2016

Earlier work this paper cites.

A dataset and exploration of models for understanding video data through fill-in-the-blank question-answering

Tegan Maharaj, Nicolas Ballas, Anna Rohrbach, Aaron Courville, and Christopher Pal. 2017 · 2017

Earlier work this paper cites.

Unifying the video and question attentions for open-ended video question answering

Hongyang Xue, Zhou Zhao, and Deng Cai. 2017 · 2017

Earlier work this paper cites.

Leveraging video descriptions to learn video question answering

Kuo-Hao Zeng, Tseng-Hung Chen, Ching-Yao Chuang, Yuan-Hong Liao, Juan Carlos Niebles, and Min Sun. 2017 · 2017

Earlier work this paper cites.

Uncovering the temporal context for video question answering

Linchao Zhu, Zhongwen Xu, Yi Yang, and Alexander G. Hauptmann. 2017 · 2017

Earlier work this paper cites.

Object level visual reasoning in videos

Fabien Baradel, Natalia Neverova, Christian Wolf, Julien Mille, and Greg Mori. 2018 · 2018

Earlier work this paper cites.

Relational inductive biases, deep learning, and graph networks

Original

Peter Battaglia, Jessica Blake Chandler Hamrick, Victor Bapst, Alvaro Sanchez, Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, Caglar Gulcehre, Francis Song, Andy Ballard, Justin Gilmer, George E. Dahl, Ashish Vaswani, Kelsey Allen, Charles Nash, Victoria Jayne Langston, Chris Dyer, Nicolas Heess, Daan Wierstra, Pushmeet Kohli, Matt Botvinick, Oriol Vinyals, Yujia Li, and Razvan Pascanu. 2018 · 2018

Earlier work this paper cites.

Mapping images to scene graphs with permutation-invariant structured prediction

Roei Herzig, Moshiko Raboh, Gal Chechik, Jonathan Berant, and Amir Globerson. 2018 · 2018

Earlier work this paper cites.

Referring relationships

Ranjay Krishna, Ines Chami, Michael S. Bernstein, and Li Fei-Fei. 2018 · 2018

Earlier work this paper cites.

Spatio-temporal action graph networks

Roei Herzig, Elad Levi, Huijuan Xu, Hang Gao, Eli Brosh, Xiaolong Wang, Amir Globerson, and Trevor Darrell. 2019 · 2019

Earlier work this paper cites.

Multi-agent reinforcement learning based frame sampling for effective untrimmed video recognition

Wenhao Wu, Dongliang He, Xiao Tan, Shifeng Chen, and Shilei Wen. 2019 · 2019

Earlier work this paper cites.

Learning canonical representations for scene graph to image generation

Roei Herzig, Amir Bar, Huijuan Xu, Gal Chechik, Trevor Darrell, and Amir Globerson. 2020 · 2020

Earlier work this paper cites.

Compositional video synthesis with action graphs

Amir Bar, Roei Herzig, Xiaolong Wang, Anna Rohrbach, Gal Chechik, Trevor Darrell, and A. Globerson. 2021 · 2021

Earlier work this paper cites.

Star: A benchmark for situated reasoning in real-world videos

Bo Wu, Shoubin Yu, Zhenfang Chen, Joshua B Tenenbaum, and Chuang Gan. 2021 · 2021

Earlier work this paper cites.

Next-qa: Next phase of question-answering to explaining temporal actions

Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. 2021 · 2021

Earlier work this paper cites.

Just ask: Learning to answer questions from millions of narrated videos

Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid. 2021 · 2021

Earlier work this paper cites.

Flamingo: a visual language model for few-shot learning

Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. 2022 · 2022

Earlier work this paper cites.

Bringing image scene structure to video via frame-clip consistency of object tokens

Elad Ben Avraham, Roei Herzig, Karttikeya Mangalam, Amir Bar, Anna Rohrbach, Leonid Karlinsky, Trevor Darrell, and Amir Globerson. 2022 · 2022

Earlier work this paper cites.

Revisiting the “Video” in Video-Language Understanding

Shyamal Buch, Cristobal Eyzaguirre, Adrien Gaidon, Jiajun Wu, Li Fei-Fei, and Juan Carlos Niebles. 2022 · 2022

Earlier work this paper cites.

Scaling instruction-finetuned language models

Original

Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. 2022 · 2022

Cited alongside, same era.

Object-region video transformers

Roei Herzig, Elad Ben-Avraham, Karttikeya Mangalam, Amir Bar, Gal Chechik, Anna Rohrbach, Trevor Darrell, and Amir Globerson. 2022 · 2022

Cited alongside, same era.

From representation to reasoning: Towards both evidence and commonsense reasoning for video question-answering

Jiangtong Li, Li Niu, and Liqing Zhang. 2022 · 2022

Cited alongside, same era.

LGDN: Language-guided denoising network for video-language modeling

Haoyu Lu, Mingyu Ding, Nanyi Fei, Yuqi Huo, and Zhiwu Lu. 2022 · 2022

Cited alongside, same era.

Long-form video-language pre-training with multimodal temporal contrastive learning

Language models are causal knowledge extractors for zero-shot video question answering

Hung-Ting Su, Yulei Niu, Xudong Lin, Winston H. Hsu, and Shih-Fu Chang. 2023 · 2023

Later among the works it cites.

Modular visual question answering via code generation

Sanjay Subramanian, Medhini Narasimhan, Kushal Khangaonkar, Kevin Yang, Arsha Nagrani, Cordelia Schmid, Andy Zeng, Trevor Darrell, and Dan Klein. 2023 · 2023

Later among the works it cites.

Vipergpt: Visual inference via python execution for reasoning

Dídac Surís, Sachit Menon, and Carl Vondrick. 2023 · 2023

Later among the works it cites.

CLIP-vip: Adapting pre-trained image-text model to video-language alignment

Hongwei Xue, Yuchong Sun, Bei Liu, Jianlong Fu, Ruihua Song, Houqiang Li, and Jiebo Luo. 2023 · 2023

Later among the works it cites.

Vid2seq: Large-scale pretraining of a visual language model for dense video captioning

Antoine Yang, Arsha Nagrani, Paul Hongsuck Seo, Antoine Miech, Jordi Pont-Tuset, Ivan Laptev, Josef Sivic, and Cordelia Schmid. 2023 · 2023

Later among the works it cites.