13. Scaling laws: why bigger worked

In 2020, OpenAI reported something unusual: language model loss falls as a power law in each of model size N, dataset size D, and compute C. On log-log axes, the relationship is a straight line, across many orders of magnitude.

L(N) ≈ L_∞ + (N_c / N)^α

The practical importance is not the equation but the predictability. Train a series of small models, fit the curve, extrapolate — and you can forecast the loss of a model 1000× larger before spending the money. Without that, no organisation would commit nine figures to a single run.

Read the fine print, though:

• The law predicts loss, not capability. The relationship between loss and whether a model can write correct SQL is far messier.
• Returns are strongly diminishing. Each equal step down in loss costs exponentially more compute.
• Power laws describe the range where they were fitted. They are not a guarantee about regions nobody has measured.

The 2020 work suggested spending most additional compute on parameters. The field did: GPT-3 was 175B parameters trained on 300B tokens.

In 2022, DeepMind's Chinchilla paper redid the experiments more carefully — varying model size and data together at fixed compute budgets — and found this was badly wrong. The result: for compute-optimal training, parameters and tokens should scale in roughly equal proportion, around 20 tokens per parameter.

By that standard GPT-3 was undertrained by more than an order of magnitude. Chinchilla itself, at 70B parameters trained on 1.4T tokens, outperformed the 280B-parameter Gopher trained on far less data, using the same compute.

The consequence was immediate and visible: the industry stopped racing on parameter count and started racing on data quantity and quality. Model sizes plateaued or fell while token counts exploded.

Training compute in FLOPs is well approximated by:

C ≈ 6 × N × D

where N is parameters and D is training tokens. The 6 breaks down as roughly 2 FLOPs per parameter for the forward pass and 4 for the backward pass.

Worked example — a 7B model on 2T tokens: 6 × 7×10⁹ × 2×10¹² = 8.4×10²² FLOPs. On 1000 H100s delivering roughly 160 teraFLOPs each after realistic utilisation losses, that is about 6,000 GPU-days, or a week of wall-clock time. The estimator below works the same arithmetic for any size you pick.

The corresponding inference rule is 2N FLOPs per generated token — forward pass only, no backward pass.

Chinchilla optimises one thing: loss for a fixed training budget. That is the wrong objective if you are going to serve the model to millions of users, because inference cost scales with parameters and gets paid over and over.

So the industry deliberately overtrains: take a smaller model and train it on far more than 20 tokens per parameter. Loss is slightly worse than a compute-optimal model at the same training cost, but the model is permanently cheaper and faster to serve. Llama 3 8B, trained on 15T tokens, is roughly 1,900 tokens per parameter — nearly a hundred times the Chinchilla ratio, and an entirely rational choice.

What is changing now:

• Data is becoming the binding constraint. High-quality text on the internet is finite, which drives interest in synthetic data, multimodal data and multiple epochs.
• Mixture-of-experts breaks the tie between parameter count and inference cost, changing the shape of the trade-off (Lesson 17).
• Inference-time compute is a new scaling axis: spending more compute per query — longer reasoning chains, sampling many candidates — buys accuracy without any change to the weights. Scaling that has its own, separate curves.