3. Embeddings: meaning as direction

Last lesson ended with your sentence chopped into pieces and each piece replaced by a number — an id, like 1842. But notice what that id actually is: a label, a shelf number. Nothing about the number 1842 resembles "dog", and nothing tells the machine what dog-ness is. If ids were assigned by lottery, dog and puppy might get numbers a million apart. A shelf number carries no meaning.

So the machine's next move is a lookup. Somewhere in memory there is an enormous table — one row per vocabulary entry — and row 1842 is a long list of numbers, say:

dog → [0.21, -0.87, 0.44, … ]  (thousands of numbers)

That list is the embedding of "dog". The table itself is the embedding matrix, and its size tells you everything: "100,000 by 4096" means 100,000 vocabulary entries, each stored as a list of 4096 numbers — about 410 million numbers in the table. Nobody chose them by hand. They started as random noise and were nudged, billions of times, by the training process from Lesson 12 until predictions got better. You can't read a single row and understand it; the meaning is in the pattern, which only becomes visible when you compare rows.

Why bother turning a label into a 4096-number list? Because of what lists of numbers can do that labels can't: be compared and combined. "Dog" and "puppy" get lists that are close together; "dog" and "because" get lists that point in different directions. The machine can then do arithmetic on closeness — which is the whole content of the next two sections. Before that, one thing to get straight, because everything below leans on it:

A list of numbers is an arrow. This sounds like a leap; it isn't. Take the list [2, 1]. Read it as instructions: "from the centre of a map, walk 2 steps east, then 1 step north." Draw the straight line from start to finish and you have an arrow. A 4096-number list is the same idea with 4096 directions instead of two — hard to picture, but the arithmetic doesn't care; it's identical. So when this lesson says "meaning as direction", it means: every piece of text becomes an arrow, and similar things point similar ways.

One warning about those 4096 numbers: no single one means anything on its own. There is no "animal" slot or "plural" slot — concepts are spread across many numbers at once, several concepts sharing the same numbers. (The technical name is superposition.) You only ever see meaning by comparing arrows — which is exactly what the next section is about.

You can already feel what "similar direction" means — two arrows pointing roughly the same way look similar. The machine can't see, so it needs arithmetic that answers the same question. Two operations do almost all the work, and you are going to do both by hand.

Operation 1: the dot product — a "do they agree?" score. Take two arrows, multiply their numbers position by position, and add the results up. That's the whole recipe:

a = [2, 1] and b = [1, 2]
dot = (2×1) + (1×2) = 4

Why would anyone call that "similarity"? Look at what the multiplication does to matching signs: two positives multiplied give a positive contribution; a positive times a negative gives a negative one. So when the two arrows "pull the same way" on a dimension, that dimension adds to the score; when they pull opposite ways, it subtracts. Big positive score = they agree on most dimensions. Near zero or negative = they don't. The Σ symbol you'll see in textbooks (a·b = Σ aᵢbᵢ) just means "add these up over every dimension" — two additions in this example, thousands in a real model, same recipe.

Operation 2: cosine — the same score, with length cancelled out. The dot product has a flaw: double the length of one arrow and the score doubles too, even though the arrow points exactly where it did. Long arrows would crowd short ones out of every comparison. The fix is to divide the dot product by both lengths:

cos = dot ÷ (length of a × length of b)
length of [2,1] = √(2² + 1²) = √5 ≈ 2.24  (Pythagoras — the straight-line distance from the centre to the arrow's tip)
cos = 4 ÷ (2.24 × 2.24) = 0.80

Because the lengths cancelled, cosine can only ever say one of three things, and it always lands between −1 and 1: 1 = exactly the same direction, 0 = at right angles, unrelated, −1 = opposite directions. In the interactive, the readout's "angle" is just this number translated into degrees — drag a tip and watch them move together.

Why bother cancelling length? Because in trained embeddings, length mostly tracks how frequent a word is, not what it means — and meaning is what we want to compare. This is why every embedding-search tool you'll meet (Lesson 16) measures the angle, not the raw distance.

Try both by hand before you drag anything. For a = [2, 1] and b = [1, 2]: the dot product is 4, the cosine is 0.80 — those are exactly the numbers the readout shows at the interactive's starting position, so you can check your arithmetic against the machine's. Then drag one tip so the arrows point the same way but one is clearly longer, and find the pair of readout numbers that proves length and direction are separate things.

Now the part that makes people do a double-take. You've seen arrows compared. It turns out you can also add and subtract them — pair up the numbers and do the arithmetic position by position — and on trained embeddings the results sometimes land on another word's arrow. Here is the famous demonstration, from word2vec back in 2013:

king − man + woman ≈ queen

Read it as walking directions. Start at the king arrow. Subtracting man means walking backwards along your own arrow — backing off toward the centre — and adding woman means then walking out along woman's direction. In the toy space below those two moves land you near queen's arrow, at cosine 0.99. Why would geometry do that? Nobody programmed the analogy: the space was trained to predict text, and English text repeats this pattern — wherever you see one member of a king/queen, man/woman, prince/princess pair, the others tend to appear in similar company, so prediction pushed the arrows into consistent clusters.

That's the honest version of "meaning as direction" — and here is the honest warning that goes with it: the analogy is a tendency, not a law. It works cleanly for a few showcase pairs and gets messier the further you push it; real embeddings are optimised for prediction, not for tidy geometry, and the toy space below is hand-designed to show the effect at its clearest. What you should take away is the reliable part: relationships between words show up as consistent directions in the space, and later parts of the model compute with those directions.

Play with the space below. Click words and read the ranked list of nearest neighbours — check that cat's top matches are dog and puppy (the animals cluster), and that run, running and ran sit nearly on top of each other (same meaning, different spellings — the space groups the meaning, not the letters). Then try the analogy tool: pick any three words, and it computes first minus second plus third, returning whichever word's arrow the result lands closest to.

One distinction finishes the lesson, and it's the one most explanations — and most people — get wrong, so be precise about it. It starts from an honest limitation: the embedding table is a lookup, and a lookup gives the same answer every time.

The word bank has one row in the table. Financial institution and river side get the identical starting arrow, because the table knows nothing about your sentence — you look a word up before reading it, the same way you look up a word in a paper dictionary before you've read the paragraph. (If you've ever seen a dictionary give you the financial meaning first for a sentence about fishing, you know this limitation isn't unique to machines.)

So how does the model eventually tell the two senses apart? The lookup stage is finished by then — it hands off, like a relay runner passing the baton, to the next station: the transformer blocks (station 3 of Lesson 1's assembly line). What they do is rewrite the arrows, and here is the one idea to hold onto: each word's arrow knows where it sits in the sentence — whether it's the first word, the third, the last. And each block lets every word look around at the other words and blend what it finds into its own arrow. In "river bank", the arrow at bank consults the arrow at river and drifts away from the money sense. In "bank account", it consults account and drifts the other way. Same starting arrow — different finished arrow, because of who the word was sitting next to.

The fixed thing is the starting arrow; the finished, context-shaped one is what the next stage actually predicts from. Building those context-shaped arrows is precisely the job of attention (Lesson 5), and it's the main thing that separates a modern LLM from the older word2vec systems, which had no way to do it.

Two facts you'll need later — skim them now, meet them properly in Lessons 8–9:

• The arrow at each position has a name: the residual stream. Picture it as a running notes page per word — every block reads the current page and adds its findings to it. ("Residual" just means "what's left over so far".) It is the working memory for that position.
• Many models reuse the same table on the way out. It was used at the start to look each word's arrow up; at the end it's turned around and used to score candidate answers — the model's final question is literally "which vocabulary word's arrow does my finished arrow point at most?" (This is weight tying; it saves hundreds of millions of parameters.) One subtlety for later: that scoring uses the raw dot product, not the cosine — so a long arrow can beat a short one pointing the same way.