DeepSeek

DeepSeek R1 employs a Mixture-of-Experts architecture where a router network selects which of the 58 expert layers to activate for each token, achieving frontier reasoning quality at a fraction of the FLOPs of dense equivalents. The model uses Multi-Head Latent Attention (MLA) which compresses the KV-cache by a factor of 10x compared to standard attention, reducing inference memory overhead significantly. Their training pipeline combines supervised pre-training on 14.8 trillion tokens with reinforcement learning via Group Relative Policy Optimization (GRPO) for reasoning tasks. DeepSeek V3 uses Multi-head Latent Attention and an auxiliary-loss-free load balancing strategy that eliminates the expert collapse problem common in MoE models.