14. Post-training: SFT, RLHF and DPO

Ask a raw pretrained model "What is the capital of France?" and a very plausible continuation is:

What is the capital of France?
What is the largest city in Germany?
What is the currency of Japan?

It is not being unhelpful. It is doing its job perfectly: on the internet, a question is frequently followed by more questions — a quiz, an FAQ, an exercise list. Answering is only one of many likely continuations.

The base model contains the knowledge and the ability. What it lacks is the convention that a user turn should be followed by a helpful assistant turn. Post-training installs that convention, along with tone, refusal behaviour, formatting and much else.

SFT is ordinary next-token training on a curated dataset of conversations. Examples look like:

<|user|> Explain photosynthesis to a 10-year-old.
<|assistant|> Plants make their own food using sunlight…<|end|>

Loss is usually computed only on the assistant tokens — you want the model to learn to produce responses, not to produce user turns. Special role tokens delimit the turns and are what let the model track who is speaking. (This is also why prompt-injection attacks are structurally interesting: everything is ultimately one token stream, and the role boundaries are a learned convention rather than a hard barrier.)

Data volume is modest: tens of thousands of examples is often enough, and quality dominates quantity. The LIMA result — strong instruction-following from 1,000 carefully curated examples — is the canonical evidence that SFT is teaching a format the model already has the substance for, not teaching new knowledge.

SFT can only imitate demonstrations. But for many questions, writing an ideal answer is hard while comparing two answers is easy. RLHF exploits that asymmetry.

1. Collect comparisons. Sample two responses to the same prompt; a human labels which is better. Repeat for tens or hundreds of thousands of prompts.
2. Train a reward model. A separate network (often initialised from the LLM) learns to output a scalar score, trained so that preferred responses score higher. It is a learned, automatable stand-in for human judgement.
3. Optimise the policy. Use reinforcement learning — PPO — to update the LLM to produce responses the reward model scores highly, with a KL-divergence penalty that punishes drifting too far from the SFT model.

That KL penalty is essential. Without it the policy finds adversarial nonsense that the reward model happens to love — classic reward hacking. The reward model is an approximation of human preference, and any optimiser pushed hard enough will find where the approximation breaks.

PPO-based RLHF is fiddly: four models in memory at once — the policy being trained, a value network, the reward model and the frozen reference — plus unstable training and many hyperparameters. Direct Preference Optimisation showed that the RLHF objective can be rewritten as a simple classification loss directly on preference pairs — no reward model, no RL loop.

The intuition: increase the log-probability of the preferred response and decrease it for the rejected one, each measured relative to a frozen reference model. One model to train, standard supervised machinery, dramatically simpler. DPO and its variants (IPO, KTO, ORPO) are now the default for most open-model post-training, though carefully done PPO can still edge it out at the frontier.

Other things that live in post-training:

• Constitutional AI / RLAIF — a model critiques and revises its own outputs against a written set of principles, replacing much of the human labelling.
• RLVR — reinforcement learning from verifiable rewards: for maths and code you can check the answer automatically, giving a clean, unhackable reward signal. This is the engine behind current reasoning models.
• Reasoning training — rewarding long chains of thought that reach verified-correct answers, which produces models that spend far more tokens thinking before answering.