10. Sampling: turning probabilities into text
The model gives you a distribution. Choosing a token from it is a separate, tunable decision — and it changes output quality more than most people expect.
By the end of this lesson you can
- Explain greedy, temperature, top-k and top-p sampling precisely
- Predict how a distribution reshapes as temperature changes
- Choose sensible settings for factual versus creative tasks
- Explain repetition penalties, beam search and their failure modes
Greedy is not obviously right
The simplest rule is to always take the highest-probability token. It is deterministic and reproducible. It is also, for open-ended text, noticeably bad: greedy output falls into loops and repetitive, flat phrasing.
The reason is worth understanding. Human text is not the most probable text. At every position, natural writing takes a moderately likely option rather than the single most likely one; always choosing the mode produces something statistically unlike real language. Greedy decoding also has no way out of a repetition loop — if "the same the same" becomes locally most likely, it stays there forever.
For short factual answers or code, greedy (or temperature 0) is often exactly right. For anything longer or open-ended, you want randomness.
Temperature
Temperature divides the logits before the softmax:
P(i) = softmax(logit_i / T)
- T < 1 — logits spread apart, gaps amplify, the distribution sharpens toward the top token. More focused, more repetitive.
- T = 1 — the model's own distribution, untouched.
- T > 1 — logits compress toward each other, the distribution flattens. More varied, and past roughly 1.3 usually incoherent.
- T → 0 — becomes greedy decoding. Implementations special-case it rather than dividing by zero.
The important intuition: temperature does not add new information or make the model smarter or dumber. It reweights the same ranking. A token the model considers hopeless is still nearly hopeless at T = 1.5 — just slightly less so, which is exactly where incoherence comes from.
Temperature, live
Drag temperature and watch the same logits become a different distribution. Try 0.2 and 1.8.
Truncation: top-k and top-p
Even at moderate temperature, the tail of the distribution contains thousands of tokens that are individually near-zero but collectively hold real probability mass. Sample long enough and you will eventually draw one, derailing the text. Truncation removes the tail before sampling.
Top-k — keep the k highest-probability tokens, renormalise, sample. Simple, but k is fixed regardless of context: when the model is confident, k = 50 admits 49 bad options; when it is genuinely uncertain across 200 reasonable tokens, k = 50 cuts off good ones.
Top-p (nucleus) — sort by probability and keep the smallest set whose cumulative probability reaches p (typically 0.9–0.95). The set size adapts automatically: a couple of tokens when the model is sure, hundreds when it is not. This is the modern default.
Min-p — a newer variant: keep tokens whose probability is at least p × (probability of the top token). Robust at high temperatures, and increasingly available.
The usual ordering in an implementation is: apply penalties to logits → divide by temperature → truncate (top-k, then top-p) → renormalise → sample.
Top-k and top-p side by side
Set k and p and see which tokens survive under each rule. Move the temperature slider to change how confident the model is, and watch only the top-p set resize.
Penalties, beam search, and choosing settings
Repetition and frequency penalties subtract from the logits of tokens that already appeared. They reduce loops, but they are blunt instruments — they also penalise words that legitimately recur, such as a variable name in code or a subject's name in an essay. Keep them low or off for code.
Beam search keeps several candidate continuations alive and expands the best-scoring ones, aiming for a high-probability whole sequence rather than a high-probability next token. It is standard in translation, where there is one right answer, and poor for open-ended text, where it produces bland, generic output — again because likely text is not human text.
Sensible starting points:
| Task | Temperature | Notes |
|---|---|---|
| Factual Q&A, extraction, classification | 0 | Deterministic and reproducible |
| Code generation | 0 – 0.3 | No repetition penalty |
| General assistant prose | 0.7 | top-p 0.9–0.95 |
| Brainstorming, fiction | 0.9 – 1.1 | top-p 0.95, expect to discard some outputs |
Temperature 0 is not fully deterministic in practice. Floating-point non-associativity, batching and GPU kernel scheduling can change results run to run on a real serving stack. Do not build a system that depends on bit-identical outputs.
Lesson in one breath
Temperature divides logits before the softmax: below 1 sharpens, above 1 flattens. Top-k keeps the k best tokens, top-p keeps the smallest set whose probability sums to p. Truncation removes the long tail of bad tokens; temperature controls the shape of what remains.
Practice
Answers are checked in your browser and saved to this device. Get one wrong and you can retry as many times as you like.
What does raising the temperature from 0.7 to 1.4 do to the distribution?
The model gives probabilities [0.6, 0.25, 0.1, 0.03, 0.02]. Under top-p with p = 0.9, how many tokens remain in the candidate set?
Which are genuine drawbacks of pure greedy decoding? Select all.
Why is top-p usually preferred over top-k?
Two tokens have logits 2.0 and 1.0. At temperature 0.5, what is the probability of the higher one? Give two decimal places.
Order the operations a typical sampling implementation applies to raw logits.
Done with this lesson?
A lesson counts as complete once it is marked read and every exercise is solved.
Tip: press ← and → to move between lessons.