12. Pretraining: how the weights get their values

Training needs a single number to minimise. For next-token prediction it is cross-entropy loss:

loss = −log P(actual next token)

That is the whole thing. If the model gave the correct token probability 0.9, the loss is 0.105. If it gave it 0.1, the loss is 2.3. If it gave it 0.001, the loss is 6.9. The penalty grows sharply as the assigned probability approaches zero, which is exactly the incentive you want: being confidently wrong is punished far more than being uncertain.

Average this over every position in the batch and you have a number to descend.

Perplexity is exp(loss), and it has a nice reading: "the model is as confused as if it were choosing uniformly among this many options". Loss 2.3 means perplexity 10 — effectively a 10-way guess at each token. Good models on English land around loss 1.8–2.2.

You have a loss and billions of knobs. Which way do you turn each one?

The gradient answers this: for each weight, it is the partial derivative of the loss with respect to that weight — how much the loss would change if you nudged it up slightly. Step every weight a little in the opposite direction and the loss goes down.

w ← w − learning_rate × ∂loss/∂w

Backpropagation is how you get all those derivatives efficiently. Run the forward pass, keeping the intermediate activations. Then walk backwards through the computation graph, applying the chain rule layer by layer. The cost is roughly twice the forward pass, regardless of how many parameters there are — which is the fact that makes training large networks feasible at all.

Nobody uses plain gradient descent. Adam / AdamW is the standard: it keeps a running average of each weight's gradient (momentum) and of its squared gradient (scale), giving every parameter its own effective step size. The cost is memory — two extra numbers per parameter, which is why training a 7B model needs far more than 7B × 2 bytes of GPU memory.

• Batching. Gradients from a single sequence are noisy, so you average over many. Large models use batches of millions of tokens, assembled across thousands of GPUs.
• Learning rate schedule. Almost universally: linear warmup over the first few thousand steps (large steps on random weights destabilise training), then cosine decay toward near zero. The decay matters — models improve substantially in the final phase as the steps get small.
• Gradient clipping. Cap the gradient norm to survive occasional bad batches that would otherwise blow the weights apart.
• Mixed precision. Compute in bf16 for speed, keep a master copy of the weights in fp32 for numerical stability.
• Checkpointing. Runs last weeks to months across thousands of GPUs; hardware fails routinely, so state is saved constantly and runs resume from the last checkpoint.

One pass over the data is called an epoch, but frontier pretraining typically does roughly one epoch over an enormous corpus rather than many epochs over a small one — repeated data yields diminishing returns and eventually memorisation.

Pretraining corpora are on the order of trillions of tokens: filtered web crawl (the bulk), books, code repositories, Wikipedia, academic papers, curated Q&A.

Filtering is not a footnote — it is one of the highest-leverage parts of the whole process. Standard steps include quality classification (does this read like a reference document?), deduplication (near-duplicates cause memorisation and waste compute), removing machine-generated spam, language identification, toxicity and PII filtering, and decontamination: removing text that overlaps evaluation benchmarks, without which your reported scores measure memorisation rather than ability.

Data mixture matters too, and in non-obvious ways. Including a substantial fraction of code improves performance on reasoning tasks that have nothing to do with programming — a well-replicated result usually explained by code's strict, explicit logical structure.

Cost, for scale: a frontier pretraining run is thousands of GPUs for months, tens of millions of dollars, and a single unrecoverable failure can waste weeks. This is why pretraining happens rarely and fine-tuning happens constantly.