What Language Model to Train if You Have One Million GPU Hours?

Understand

The crystallization of modeling methods around the Transformer architecture has been a boon for practitioners.

• Simple, well-motivated architectural variations can transfer across tasks and scale, increasing the impact of modeling research.
• However, with the emergence of state-of-the-art 100B+ parameters models, large language models are increasingly expensive to accurately design and train.
• Notably, it can be difficult to evaluate how modeling decisions may impact emergent capabilities, given that these capabilities arise mainly from sheer scale alone.