When Do We Not Need Larger Vision Models?

Understand

Scaling up the size of vision models has been the de facto standard to obtain more powerful visual representations.

• In this work, we discuss the point beyond which larger vision models are not necessary.
• First, we demonstrate the power of Scaling on Scales (S$^2$), whereby a pre-trained and frozen smaller vision model (e.g., ViT-B or ViT-L), run over multiple image scales, can outperform larger models (e.g., ViT-H or ViT-G) on classification, segmentation, depth estimation, Multimodal LLM (MLLM) benchmarks, and robotic manipulation.
• Notably, S$^2$ achieves state-of-the-art performance in detailed understanding of MLLM on the V* benchmark, surpassing models such as GPT-4V.