How did DeepSeek V3 train a frontier model for just $5.6 million?

The answer

Released on December 26, 2024, DeepSeek V3 is a 671-billion-parameter Mixture-of-Experts model trained on 14.8 trillion tokens using just 2,048 NVIDIA H800 GPUs over 55 days, at a cost of approximately $5.6 million. For comparison, GPT-4's training reportedly cost over $100 million. Despite this 95% cost reduction, DeepSeek V3 achieved competitive performance with GPT-4 and Claude 3.5 Sonnet across major benchmarks. Andrej Karpathy praised the achievement, calling it a wake-up call for the industry. The model's efficiency came from architectural innovations including Multi-head Latent Attention and an auxiliary-loss-free load balancing strategy, proving that frontier AI doesn't necessarily require frontier budgets.