2. Tokens: how text becomes numbers

Lesson 1 ended with the assembly line, and its first station was the tokenizer. This lesson is that station. One sentence to start, because everything else follows from it: the model is a number machine — letters cannot enter it. Everything must be converted into numbers before any prediction happens, and the conversion has to be mechanical, because it runs on every text in every language, including texts nobody has ever seen.

So: chop the text into pieces, and give each piece a number. The only real question is how big a piece. Three options were tried; the first two fail.

Option 1: one piece per whole word. Sounds natural — words are the units we think in. Two problems. English has millions of word forms, plus names, typos, URLs and code, so the lookup list would be enormous — and still fail constantly, because any word not on the list is simply unrepresentable. And related words would share nothing: tokenization, tokenizations and Tokenizer would be three unrelated numbers, even though you can see they are the same root three ways. The model would have to learn each one from scratch.

Option 2: one piece per letter. Now the list is tiny — 26 letters plus digits and punctuation, or just 256 "bytes" (the numbers 0–255 that computers store every character as) — and nothing can ever be unknown, because every character is on the list. But look what happens to a short sentence:

Hello world → 10 pieces: H e l l o ␣ w o r l d

Recall the loop from Lesson 1: one pass through the whole machine per piece, and the model can only "see" a fixed-length stretch of pieces at a time — its context window. Ten passes instead of two, for the same sentence: five times the compute, and five times the space taken in the window. Too expensive.

Option 3: subwords — the compromise everyone landed on. Common things get one piece; rare things get built from shared fragments. "the" is one piece. "unhappiness" might be un + happi + ness. Nothing is ever unrepresentable (worst case: bytes), and sequences stay short. Real systems average roughly 4 characters per piece in English, and the list runs 32,000–256,000 entries.

One more thing to notice: the pieces are learned from text, not handed down by a linguist. What counts as "common" depends on which text you learned from — and that single fact is responsible for most of this lesson's quirks.

So how does the piece-list get built in the first place? Nobody hand-curates it. It is grown by an algorithm from 1994 called byte-pair encoding (BPE), and you are fully able to run it yourself. The whole idea in one sentence: start with everything in pieces, then repeatedly glue together whichever two neighbouring pieces appear together most often.

Do it by hand on a tiny "corpus" — four words: low low low lower. Start by splitting every word into individual letters (imagine each word ends with a marker, </w>, so "low" is really l o w</w> — the marker stops glues from running across words):

1. Count every neighbouring pair. The pair l-o appears once in each "low" (×3) and once in "lower" — 4 times total. The pair o-w appears 3 times. The pair e-r appears once. And so on.
2. Glue the winner. l-o is the most frequent pair, so it becomes a new single piece. The words are now [lo] w (×3) and [lo] w e r.
3. Recount and repeat. The pair [lo]-w appears 4 times — it wins, and "low" now exists as one piece. Next round [low]-e wins, and "lower" ends up as low + e + r after a couple more rounds.

That's all it is: count, glue the most common pair, repeat, until the list reaches its size budget (tens of thousands of entries). Run on real text, the same process is what turns frequent English sequences — ing, the, tion, def — into single pieces, while rare sequences stay in fragments. The final product is an ordered list of glue rules, and encoding any new text just means replaying those rules in the order they were learned.

Now run it for real in the trainer below — it is exactly the procedure you just did by hand. Watch which pairs win, and remember the last section's warning: the merges reflect the corpus you feed it, not "English in general". A tokenizer trained on English code splits Japanese badly, and that is a real, measured disadvantage for those languages: more pieces per sentence means more cost and less content fitting in the context window.

You now know how the piece-list was built. The other half of the story is how your text gets encoded against that list, and there is one detail there with real consequences. Two stages:

1. First, rough cuts. A fixed pattern chops the text into rough pieces — roughly word-sized, and it is this step that decides a space starts a new piece rather than ending the previous one. Glue rules never cross these cut lines.
2. Then, glues inside each piece, applied in the order they were learned (earliest first).

The detail with consequences: the space usually travels with the word that follows it. So " dog" (space glued on) and "dog" are two different pieces with different numbers, in GPT-style tokenizers and most others. Nothing is wrong with that — it is just how the list was built — but remember Lesson 1: the model can only predict what comes next, and it predicts from what the pieces actually say.

Here is the failure that follows, end to end. You write a raw completion prompt ending in a space: "The answer is ". The model's favourite continuation is " Paris" — but that piece carries its own leading space, and yours has already been spent. Emitting it would print two spaces ("is Paris"). Pieces for that exist, but text with double spaces almost never appeared during training, so the model never learned to expect anything there — the odds over the whole next piece go flat, and what little weight remains sits on oddities. You get a coin-flip instead of a confident answer, purely because your prompt ended half-way through a piece.

Two things limit the damage in practice. First, it mostly bites raw completion prompts, where your text is literally the beginning being continued; chat apps wrap your message in turn markers and generation starts after those, so the effect largely disappears. Second, it is not a GPT quirk — it follows from subword tokenization itself, so it holds for any model whose tokenizer glues the space to the following word (GPT, Llama, Mistral, Claude and the rest). The rule that survives everywhere: do not end a prompt half-way through what should be a single piece — a space at the end, or a cut in the middle of a word.

Try the live tokenizer below — it uses a cut-down copy of GPT-4's real piece-list (6,057 of the ~100,000 entries, but every id shown is genuine), with the real numbers underneath each piece. Start with the "leading space" preset: strawberry at the start of a line splits into three pieces (str + aw + berry), while ␣strawberry with its leading space is a single piece. Same letters, three pieces versus one: three times the passes through the model, three times the space it takes in the context window, and three different numbers reaching the next stage instead of one. Then try the other presets — numbers, code indentation, a Japanese sentence — and watch the "characters per token" count at the bottom change. That number is the closest thing to a "how good is this tokenizer for this language" score, and it is why the Japanese and Russian you paste in costs more than the English.

Time to collect the payoff. There is a large family of "this model is an idiot" moments that are really one fact in disguise: the model cannot see letters — only piece-numbers. The famous example: ask a chatbot how many r's are in "strawberry" and smart models confidently say 2. You can watch why in the tokenizer above: the model receives roughly str + aw + berry — three opaque numbers. The three r's are physically inside those chunks, but nothing in the input says "letters live here". The model has to know spellings indirectly, from text that talked about spellings, the way you know a word's spelling partly by feel rather than by looking.

The same single fact explains a whole cluster of failures and quirks:

• Reversing words, rhyming, letter games. All of it operates on letters the model never actually receives. It is good at them only to the extent it memorised the answers.
• Fragile arithmetic. Whether 1234 arrives as one piece, or 12+34, or digit-by-digit, changes how digits line up between the two numbers being multiplied — and alignment is what column arithmetic depends on. Some model families deliberately split every digit for exactly this reason (Exercise 5 quizzes you on the fix).
• Per-language cost. Billing is per piece. A language that averages 3 characters per piece costs roughly twice what English costs per sentence, and fills the model's view twice as fast. Not sentiment — arithmetic.
• Glitch tokens. Some pieces made it onto the list but almost never appeared in training text, so their embeddings stayed near-random. Feeding one in can derail a model completely. The famous one is SolidGoldMagikarp, which for a while made GPT-3 spit out garbled nonsense when it appeared in a prompt.