Before GPT-3 made scaling feel inevitable, this OpenAI paper made it predictable. Kaplan and coauthors ran hundreds of training runs across model size, dataset size, and compute budget, and found that language model loss follows clean power laws across seven orders of magnitude. The implications rewired the field: given a fixed compute budget, there's an optimal model size and token count, and it's mostly not what practitioners had been assuming.

The walkthrough covers the three governing quantities, why architecture details matter less than you'd think, and how the paper's compute-optimal frontier ended up guiding everything from GPT-3 to Chinchilla to modern speech foundation models. There's also a note about a mistake the presenter caught on batch size dynamics — worth listening for if you care about the experimental setup. If you've ever had to justify a training budget or explain to a stakeholder why bigger really is better here, this is foundational reading that saves you a lot of hand-waving. Queue it up.