What is the Role of Small Models in the LLM Era: A Survey

Understand

Large Language Models (LLMs) have made significant progress in advancing artificial general intelligence (AGI), leading to the development of increasingly large models such as GPT-4 and LLaMA-405B.

• However, scaling up model sizes results in exponentially higher computational costs and energy consumption, making these models impractical for academic researchers and businesses with limited resources.
• At the same time, Small Models (SMs) are frequently used in practical settings, although their significance is currently underestimated.
• This raises important questions about the role of small models in the era of LLMs, a topic that has received limited attention in prior research.