Phantom of Latent for Large Language and Vision Models

Understand

The success of visual instruction tuning has accelerated the development of large language and vision models (LLVMs).

• Following the scaling laws of instruction-tuned large language models (LLMs), LLVMs either have further increased their sizes, reaching 26B, 34B, and even 80B parameters.
• While this increase in model size has yielded significant performance gains, it demands substantially more hardware resources for both training and inference.
• Consequently, there naturally exists a strong need for efficient LLVMs that achieve the performance of larger models while being smaller in size.