What did the Chinchilla paper reveal about how most AI labs were training their models wrong?

The answer

Published by DeepMind researchers Hoffmann et al. in March 2022, the Chinchilla paper ('Training Compute-Optimal Large Language Models') demonstrated that for a given compute budget, most existing LLMs were significantly oversized and undertrained. Their analysis showed that model size and training data should be scaled equally - meaning a model with 70 billion parameters should be trained on roughly 1.4 trillion tokens. Their 70B-parameter Chinchilla model outperformed the 280B-parameter Gopher on most benchmarks while using the same compute budget. This finding reshaped the industry, influencing the design of LLaMA, Gemini, and other subsequent models.