DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model

Understand

We present DeepSeek-V2, a strong Mixture-of-Experts (MoE) language model characterized by economical training and efficient inference.

• It comprises 236B total parameters, of which 21B are activated for each token, and supports a context length of 128K tokens.
• DeepSeek-V2 adopts innovative architectures including Multi-head Latent Attention (MLA) and DeepSeekMoE.
• MLA guarantees efficient inference through significantly compressing the Key-Value (KV) cache into a latent vector, while DeepSeekMoE enables training strong models at an economical cost through sparse computation.