1. What a language model actually does

Start with a sentence with a hole in it:

The cat sat on the ___

You would probably write mat, couch or floor. You did not look those words up in a dictionary — you have seen enough English that, after "the", a piece of furniture or a surface is a likely follow-up. That guess is the entire job of a language model. It is, at heart, a very expensive fill-in-the-blank machine.

Written precisely, the whole task is:

given tokens t₁ … t_n, output P(t_n+1 | t₁ … t_n)

Do not let the notation intimidate you — you just did this exercise by hand. Read it as: "given everything written so far, produce the chance of each possible next piece." Two words to unpack:

• token — the "piece" of text the model works with. Often a word or a fragment of one ("under", "standing"); Lesson 2 is entirely about this.
• P(…) — a probability: a number from 0 to 1 for how likely something is. "Mat" might be 0.30, "couch" 0.12, "banana" 0.000001.

That is it. Not "answer the question", not "be helpful", not "understand". The model reads the text so far and produces a probability for every possible next token. If its vocabulary has 100,000 entries, the output is 100,000 numbers that sum to 1 — one chance per candidate, like odds on every horse in a race. The word for that full list of odds is a probability distribution, and you can see one live in the diagram below.

Everything you have seen an LLM do — write code, summarise a document, argue a position, translate Polish to Japanese — falls out of doing that one fill-in-the-blank job extremely well over an enormous amount of text.

Why would guessing the next word require anything more than memorised phrases? Because to predict the next word accurately across all of human writing, you are forced to pick up the things that determine the next word. Finish this and you need a fact: "The capital of France is ___". Finish this and you need arithmetic: "2 + 2 = ___". Finish a legal contract or a half-written program and you need to know how contracts and programs are built. The model was never trained to be helpful or correct — only to predict well. But prediction is so demanding that competence is what the model has to acquire in order to hit it. That is the whole trick, and it is worth pausing on.

Now see it with your eyes. Below is a small hand-built model over a toy vocabulary — a handful of prefixes, each with its odds over the possible next tokens. Click a prefix and read the bars like this:

• Each bar is one candidate for the next token and how likely the model thinks it is. The longest bar is the model's favourite guess.
• The prediction is never a single word. It is always the whole ranked, weighted list. The model is always uncertain, and the shape of that uncertainty is meaningful: a chart with one dominant bar means the prefix pinned the answer down; a chart of many similar bars means it did not.
• Compare "The capital of France is" (one bar towers over the rest — almost certainty) with "I went to the" (many plausible bars — genuine open-endedness).

The number under the chart, "entropy", is just a single score for how spread-out the bars are: low = the model is confident, high = it is guessing broadly. You do not need to compute it here; just notice it drops when the prefix constrains the answer.

You now know what happens in one step. But when you ask an LLM something, pages come out. Where does the rest come from? Not from a hidden script — from a loop, run over and over. Watch it happen one word at a time:

1. You give it the prompt: The cat sat
2. It predicts a distribution for the next token, and one token is picked from it: on
3. The picked token is glued onto the end of the input. The input is now The cat sat on — and the model treats its own output exactly like your input.
4. Predict again, pick again, glue again. Repeat until an "end of text" token comes out or a length limit is hit.

The technical name is autoregressive generation ("auto" + "regressing on itself": it feeds on itself). Press Generate next token below and watch the input grow by one word per click — the highlighted token is the newest one, about to rejoin the input.

Two consequences follow immediately, and they explain a lot of how LLMs behave in real life.

The model reads its own output. When it writes token 40, it is predicting from a sequence that includes tokens 1–39 — including the ones it wrote itself. That is why a model that starts an answer wrongly often keeps going wrongly, and confidently: its own bad first sentence is now evidence about what kind of text this is, and the model dutifully continues in that style. The flip side is the trick that makes LLMs genuinely better at hard questions: "let's think step by step" works because each step the model writes becomes input that makes the next step — and eventually the right answer — easier to predict.

Cost grows with length. Every single token costs another pass through the network, over a sequence that keeps getting longer. A 500-word answer is thousands of separate passes, one per token — which is why long answers take proportionally longer to type out. Lesson 11 covers the trick (the KV cache) that keeps this from getting out of hand, and why long chats cost more than short ones.

You now have a working mental model. Before building on it, remove three wrong ones. Most confusion about LLMs — and most failed attempts to use them well — comes from believing one of these.

It is not a database. The model does not look anything up when it answers. Facts, insofar as it has them, are smeared across billions of numbers ("weights") that were slowly adjusted during training — closer to a habit than a filing cabinet. This is why it can be confidently wrong: the goal was never "be correct", it was "produce the plausible-sounding continuation". A fluent falsehood scores just as well on that goal as a true sentence. Ask it about something rare and it will not say "I don't know" — it will keep writing, because continuing well is the job.

It has no memory between calls. Every request is a fresh calculation over exactly the text you (or your app) send. When ChatGPT seems to remember what you said ten messages ago, that is not the model remembering — the app quietly re-sends the whole conversation every time, and the model reads it again from scratch. The model itself is stateless: it calculates, answers, and forgets.

It does not plan ahead — structurally. The model emits one token at a time, and nothing in the machine reserves a plan for token 50. It has no outline, no intended ending. It behaves as if it plans because representations that predict the next token well also happen to encode where the sentence is heading. But the only thing that ever comes out is the next token, chosen from the current distribution.

One question you may reasonably still have: fine, it guesses the next token — but how, physically? There is no dictionary in there. How does a chunk of silicon turn "The cat sat on the" into a list of odds?

Here is the shape of the answer — an assembly line with four stations. You will build each station yourself over the next eight lessons, so do not try to absorb this now; just see the whole machine once so the lessons have somewhere to fit:

1. Tokenizer — chops your text into tokens (the "pieces" from earlier) and looks each one up in a fixed vocabulary list to get its number: an id. "The cat sat on the" becomes five or six integers. Nothing else in the machine ever sees letters. (Lesson 2)
2. Embedding table — turns each id into a long list of numbers (a vector) that captures how that token behaves — roughly, "what kind of thing is this and where does it usually appear". Meaning becomes direction in a giant space of numbers. (Lesson 3)
3. A stack of transformer blocks — the actual thinking. Each block lets every position in the sequence consult the other positions (that is attention — how the model knows that "it" in a sentence refers to the cat), then does some private number-crunching at each position. The same kind of block repeats 30–100+ times. (Lessons 5–9)
4. Output projection + softmax — after the last block, the numbers at the final position are converted into one probability per vocabulary entry: the bar chart you played with above. (Lesson 9)

Then the loop from the previous section picks a token, and the whole line runs again. That is the entire machine. Every part of it is just arithmetic on lists of numbers — there is no other machinery hiding inside, no dictionary, no search, no author. If any of the four stations above reads like gibberish, good: that is exactly what Lessons 2–9 are for, and they go one station at a time.