What is Mixture of Experts and how does it make giant AI models affordable?

The answer

Mixture of Experts (MoE) is an architecture where a model contains many specialized sub-networks ('experts'), but a gating mechanism routes each input to only a small subset of them. This means a model can have enormous total parameter counts while only activating a fraction per inference step, dramatically reducing compute costs. Google's Switch Transformer (2021) scaled to 1.6 trillion parameters while being practical to train because each token activated only one expert. The approach gained renewed attention with Mixtral 8x7B (Mistral AI, December 2023), which matched GPT-3.5 performance while using only 12.9B active parameters out of 46.7B total per forward pass.