Accelerating Multimodal Large Language Models by Searching Optimal Vision Token Reduction

Understand

Prevailing Multimodal Large Language Models (MLLMs) encode the input image(s) as vision tokens and feed them into the language backbone, similar to how Large Language Models (LLMs) process the text tokens.

• However, the number of vision tokens increases quadratically as the image resolutions, leading to huge computational costs.
• In this paper, we consider improving MLLM's efficiency from two scenarios, (I) Reducing computational cost without degrading the performance.
• (II) Improving the performance with given budgets.