OLMoE: Open Mixture-of-Experts Language Models

Understand

We introduce OLMoE, a fully open, state-of-the-art language model leveraging sparse Mixture-of-Experts (MoE).

• OLMoE-1B-7B has 7 billion (B) parameters but uses only 1B per input token.
• We pretrain it on 5 trillion tokens and further adapt it to create OLMoE-1B-7B-Instruct.
• Our models outperform all available models with similar active parameters, even surpassing larger ones like Llama2-13B-Chat and DeepSeekMoE-16B.