NVILA: Efficient Frontier Visual Language Models

Understand

Visual language models (VLMs) have made significant advances in accuracy in recent years.

• However, their efficiency has received much less attention.
• This paper introduces NVILA, a family of open VLMs designed to jointly optimize efficiency and accuracy.
• Building on top of VILA, we improve its model architecture by first scaling up the spatial and temporal resolutions, and then compressing visual tokens.