Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Laws

Understand

Scaling laws describe the relationship between the size of language models and their capabilities.

• Unlike prior studies that evaluate a model's capability via loss or benchmarks, we estimate the number of knowledge bits a model stores.
• We focus on factual knowledge represented as tuples, such as (USA, capital, Washington D.C.) from a Wikipedia page.
• Through multiple controlled datasets, we establish that language models can and only can store 2 bits of knowledge per parameter, even when quantized to int8, and such knowledge can be flexibly extracted for downstream applications.