18. Evaluation, hallucination and what is still unsolved

• Perplexity. Cheap and objective, but only comparable between models sharing a tokenizer, and it correlates loosely with anything a user cares about.
• Multiple-choice benchmarks (MMLU and similar). Fast and automatic, but heavily contaminated — the questions are on the internet, therefore in the training data — and scoring is sensitive to formatting and option order. Saturated benchmarks tell you almost nothing at the frontier.
• Execution-based evaluation (HumanEval, SWE-bench). Run the generated code against tests. Objective and hard to fake, which is why code is unusually well-measured.
• Human preference arenas. Blind pairwise comparison at scale. Captures what people actually like, which includes a real bias toward length, formatting and confidence.
• LLM-as-judge. A strong model scores outputs against a rubric. Cheap and scalable, with documented biases: position bias, verbosity bias, and self-preference for its own outputs. Usable with care — randomise positions, use a rubric, calibrate against human labels.

Not a bug in the usual sense. Several structural causes stack up:

1. The objective rewards plausibility. Training optimises the probability of a plausible continuation. A fluent, well-formed falsehood is a good continuation by that measure. Nothing in pretraining distinguishes "true" from "reads like the truth".
2. There is no retrieval step. Facts are diffuse patterns in weights, not records. Nothing exists to return "not found".
3. Uncertainty has nowhere to go. The model has calibrated uncertainty internally — its probability distribution genuinely reflects it — but the sampled output is a token, and one token cannot carry a confidence interval unless the model was trained to say so.
4. Post-training can make it worse. Human raters prefer confident, complete answers over hedged ones. Preference training therefore rewards asserting over admitting ignorance.
5. Compounding. Once a fabricated detail is in the context, later tokens are conditioned on it and stay consistent with it.

Two useful nuances. First, models are often better calibrated than they appear — the information about uncertainty exists in the logits even when the text sounds certain. Second, prompting for uncertainty ("say if you are unsure") measurably helps, because it makes hedging a likely continuation rather than a disfavoured one.

• Grounding. Supply the source text and instruct the model to answer only from it. The single most effective intervention available.
• Citations. Require a quote or span reference for each claim, then verify the quote appears in the source. Cheap, mechanical, and catches a lot.
• Escape hatches. Explicitly permit "I don't know" and "the document does not say".
• Verification passes. A second call that checks the first against the sources. Not free, and not independent, but it catches a meaningful share of errors.
• Self-consistency. Sample several answers; agreement across them is a usable confidence signal, and disagreement is a strong warning.
• Constrained output. Schemas and enumerated options remove whole categories of invention.
• Verifiable domains. Where you can check — run the code, execute the query — check. Do not ask the model to be right; ask it to produce something testable.

None of these achieve elimination. Systems that must not state falsehoods need a human in the loop or a hard verification gate, not a better prompt.

Where the field actually stands, without hype in either direction:

• Reliability. Per-step accuracy is high; long chains still compound errors into failure. This is the main barrier to autonomous use.
• Interpretability. Sparse autoencoders and circuit analysis have made real progress, and we still cannot explain most of what a frontier model does on a given input.
• Continual learning. Models cannot absorb new knowledge without retraining; fine-tuning causes catastrophic forgetting. Context and retrieval are workarounds, not solutions.
• Genuine reasoning versus pattern matching. Performance often degrades on problems restructured to defeat memorised patterns. Where the line sits is contested and empirically unsettled.
• Evaluation itself. We are measurably bad at measuring these systems, and benchmarks decay as they leak into training data.
• Alignment. Preference training optimises for what raters approve of, which is not the same as what is true or good. Scalable oversight of systems that exceed human ability in a domain is unsolved.
• Data. High-quality human text is finite and the easy supply is largely consumed.