Do AI models just need to be bigger, or is there a smarter way to scale?

The answer

The scaling laws debate intensified after Kaplan et al. at OpenAI published empirical findings in January 2020 showing that model performance improves predictably as a power law with model size, dataset size, and compute budget. This 'bigger is better' philosophy drove the creation of ever-larger models from GPT-3 to PaLM (540B parameters). However, the 2022 Chinchilla paper challenged this by showing most large models were significantly undertrained relative to their size. The debate continues today with some researchers pursuing efficiency through architectural innovations, while others argue that scale remains the primary driver of capability.