Are AI scaling laws hitting a wall?
The recipe that made GPT-4 possible may be running out of ingredients, sparking an intense debate about AI's future trajectory.
The answer
DeepMind's 2022 Chinchilla paper established that optimal AI training requires roughly 20 tokens of data per model parameter. But this creates a dilemma: high-quality English text was projected to be exhausted by 2024 at current scaling rates. Meta's Llama 3 already deviated dramatically, training its 8B model on 15 trillion tokens - an 1,875:1 ratio - discovering that over-training smaller models yields surprisingly strong, cost-effective performance. This sparked what researchers call the 'Chinchilla Trap,' where following optimal scaling rules produces models too large and expensive to deploy. Meanwhile, Epoch AI estimates the indexed web contains around 500 trillion usable tokens.
Source: Epoch AI
- scaling laws
- Chinchilla
- data scarcity
More from AI & Machine Learning
- What radical question did Alan Turing ask in 1950 that launched the entire field of artificial intelligence?
- Can a machine fool you into thinking it's human - and does that mean it's truly intelligent?
- What happened at a small Dartmouth workshop in 1956 that gave artificial intelligence its name?
- How did a 1958 invention called the Perceptron convince people that machines could learn like brains?
- Could a 1966 chatbot really make people believe they were talking to a therapist?
- What was the first robot that could 'reason' about its own actions?