2023

TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding

Ren, Shuhuai, Yao, Linli, Li, Shicheng et al.

Understand

This work proposes TimeChat, a time-sensitive multimodal large language model specifically designed for long video understanding.

Our model incorporates two key architectural contributions: (1) a timestamp-aware frame encoder that binds visual content with the timestamp of each frame, and (2) a sliding video Q-Former that produces a video token sequence of varying lengths to accommodate videos of various durations.
Additionally, we construct an instruction-tuning dataset, encompassing 6 tasks and a total of 125K instances, to further enhance TimeChat's instruction-following performance.
Experiment results across various video understanding tasks, such as dense captioning, temporal grounding, and highlight detection, demonstrate TimeChat's strong zero-shot temporal localization and reasoning capabilities.

Built on

Creating summaries from user videos
Michael Gygli, Helmut Grabner, Hayko Riemenschneider, and Luc Van Gool · 2014
Earlier work this paper cites.
Cider: Consensus-based image description evaluation
Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh · 2014
Earlier work this paper cites.
Tvsum: Summarizing web videos using titles
Yale Song, Jordi Vallmitjana, Amanda Stent, and Alejandro Jaimes · 2015
Earlier work this paper cites.
Video summarization with long short-term memory
Ke Zhang, Wei-Lun Chao, Fei Sha, and Kristen Grauman · 2016
Earlier work this paper cites.
Tall: Temporal activity localization via language query
J. Gao, Chen Sun, Zhenheng Yang, and Ramakant Nevatia · 2017
Earlier work this paper cites.
Localizing moments in video with natural language
Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan C. Russell · 2017
Earlier work this paper cites.
Dense-captioning events in videos
Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles · 2017
Earlier work this paper cites.
Towards automatic learning of procedures from web instructional videos
Luowei Zhou, Chenliang Xu, and Jason J. Corso · 2017
Earlier work this paper cites.
Coin: A large-scale dataset for comprehensive instructional video analysis
Yansong Tang, Dajun Ding, Yongming Rao, Yu Zheng, Danyang Zhang, Lili Zhao, Jiwen Lu, and Jie Zhou · 2019
Earlier work this paper cites.
An image is worth 16x16 words: Transformers for image recognition at scale
Original
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby · 2020
Earlier work this paper cites.
Soda: Story oriented dense video captioning evaluation framework
Soichiro Fujita, Tsutomu Hirao, Hidetaka Kamigaito, Manabu Okumura, and Masaaki Nagata · 2020
Earlier work this paper cites.
Multimodal pretraining for dense video captioning
Gabriel Huang, Bo Pang, Zhenhai Zhu, Clara Rivera, and Radu Soricut · 2020
Earlier work this paper cites.
Queryd: A video dataset with high-quality text and audio narrations
Andreea-Maria Oncescu, João F. Henriques, Yang Liu, Andrew Zisserman, and Samuel Albanie · 2020
Earlier work this paper cites.
Video summarization using deep neural networks: A survey
Evlampios Apostolidis, E. Adamantidou, Alexandros I. Metsai, Vasileios Mezaris, and I. Patras · 2021
Earlier work this paper cites.
Lora: Low-rank adaptation of large language models
Original
J. Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen · 2021
Earlier work this paper cites.
Qvhighlights: Detecting moments and highlights in videos via natural language queries
Original
Jie Lei, Tamara L. Berg, and Mohit Bansal · 2021
Earlier work this paper cites.
Learning transferable visual models from natural language supervision
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever · 2021
Earlier work this paper cites.

Similar

Learning relation alignment for calibrated cross-modal retrieval
Shuhuai Ren, Junyang Lin, Guangxiang Zhao, Rui Men, An Yang, Jingren Zhou, Xu Sun, and Hongxia Yang · 2021
Cited alongside, same era.
Learning to segment actions from visual and language instructions via differentiable weak sequence alignment
Yuhan Shen, Lu Wang, and Ehsan Elhamifar · 2021
Cited alongside, same era.
Videoclip: Contrastive pre-training for zero-shot video-text understanding
Hu Xu, Gargi Ghosh, Po-Yao Huang, Dmytro Okhonko, Armen Aghajanyan, and Florian Metze Luke Zettlemoyer Christoph Feichtenhofer · 2021
Cited alongside, same era.
Joint video summarization and moment localization by cross-task sample transfer
Hao Jiang · 2022
Cited alongside, same era.
Umt: Unified multi-modal transformers for joint video moment retrieval and highlight detection
Ye Liu, Siyuan Li, Yang Wu, Chang-Wen Chen, Ying Shan, and Xiaohu Qie · 2022
Cited alongside, same era.
Introducing chatgpt
OpenAI · 2022
Cited alongside, same era.
Training language models to follow instructions with human feedback
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al · 2022
Cited alongside, same era.
Robust speech recognition via large-scale weak supervision
Original
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever · 2022
Cited alongside, same era.
Multiinstruct: Improving multi-modal zero-shot learning via instruction tuning
Zhiyang Xu, Ying Shen, and Lifu Huang · 2022
Cited alongside, same era.
Merlot reserve: Neural script knowledge through vision and language and sound
Rowan Zellers, Jiasen Lu, Ximing Lu, Youngjae Yu, Yanpeng Zhao, Mohammadreza Salehi, Aditya Kusupati, Jack Hessel, Ali Farhadi, and Yejin Choi · 2022
Cited alongside, same era.
Unifying event detection and captioning as sequence generation via pre-training
Qi Zhang, Yuqing Song, and Qin Jin · 2022
Cited alongside, same era.
End-to-end dense video captioning as sequence generation
Original
Wanrong Zhu, Bo Pang, Ashish V Thapliyal, William Yang Wang, and Radu Soricut · 2022
Cited alongside, same era.

Then

Univtg: Towards unified video-language temporal grounding
Kevin Qinghong Lin, Pengchuan Zhang, Joya Chen, Shraman Pramanick, Difei Gao, Alex Jinpeng Wang, Rui Yan, and Mike Zheng Shou · 2023
Closest in time.
whisper-timestamped
Jérôme Louradour · 2023
Closest in time.
Towards generalisable video moment retrieval: Visual-dynamic injection to image-text pre-training
Dezhao Luo, Jiabo Huang, Shaogang Gong, Hailin Jin, and Yang Liu · 2023
Closest in time.
Video-chatgpt: Towards detailed video understanding via large vision and language models
Original
Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan · 2023
Closest in time.
Query - dependent video representation for moment retrieval and highlight detection
WonJun Moon, Sangeek Hyun, Sang shin Paldal-gu Suwon-city Park, Dongchan Park, and Jae-Pil Heo · 2023
Closest in time.
Gpt-4 technical report
Original
OpenAI · 2023
Closest in time.
TESTA: Temporal-spatial token aggregation for long-form video-language understanding
Shuhuai Ren, Sishuo Chen, Shicheng Li, Xu Sun, and Lu Hou · 2023
Closest in time.
Delving into the openness of CLIP
Shuhuai Ren, Lei Li, Xuancheng Ren, Guangxiang Zhao, and Xu Sun · 2023
Closest in time.
Moviechat: From dense token to sparse memory for long video understanding
Original
Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Xun Guo, Tianbo Ye, Yang Lu, Jenq-Neng Hwang, and Gaoang Wang · 2023
Closest in time.
Eva-clip: Improved training techniques for clip at scale
Original
Quan Sun, Yuxin Fang, Ledell Yu Wu, Xinlong Wang, and Yue Cao · 2023
Closest in time.
Vid2seq: Large-scale pretraining of a visual language model for dense video captioning
Antoine Yang, Arsha Nagrani, Paul Hongsuck Seo, Antoine Miech, Jordi Pont-Tuset, Ivan Laptev, Josef Sivic, and Cordelia Schmid · 2023
Closest in time.
Edit as you wish: Video description editing with multi-grained commands, 2023
Linli Yao, Yuanmeng Zhang, Ziheng Wang, Xinglin Hou, Tiezheng Ge, Yuning Jiang, and Qin Jin · 2023
Closest in time.
Movie101: A new movie understanding benchmark
Zihao Yue, Qi Zhang, Anwen Hu, Liang Zhang, Ziheng Wang, and Qin Jin · 2023
Closest in time.
Hierarchical video-moment retrieval and step-captioning
Abhaysinh Zala, Jaemin Cho, Satwik Kottur, Xilun Chen, Barlas Ouguz, Yasher Mehdad, and Mohit Bansal · 2023
Closest in time.
Video-llama: An instruction-tuned audio-visual language model for video understanding
Original
Hang Zhang, Xin Li, and Lidong Bing · 2023
Closest in time.
Minigpt-4: Enhancing vision-language understanding with advanced large language models
Original
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny · 2023
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…