LinVT: Empower Your Image-level Large Language Model to Understand Videos

Understand

Large Language Models (LLMs) have been widely used in various tasks, motivating us to develop an LLM-based assistant for videos.

• Instead of training from scratch, we propose a module to transform arbitrary well-trained image-based LLMs into video-LLMs (after being trained on video data).
• To better adapt image-LLMs for processing videos, we introduce two design principles: linear transformation to preserve the original visual-language alignment and representative information condensation from redundant video content.
• Guided by these principles, we propose a plug-and-play Linear Video Tokenizer(LinVT), which enables existing image-LLMs to understand videos.