5. Attention: queries, keys and values

Take the sentence: "The trophy did not fit in the suitcase because it was too big." What does it refer to?

The trophy. Change one word — "too small" — and it becomes the suitcase. Resolving this requires the representation at the position of it to pull in information from a specific earlier position, chosen based on content, not on fixed distance.

Older architectures handled this badly. Recurrent networks squeezed the whole past through one fixed-size hidden state, sequentially, so long-range links were both lossy and slow to train. Attention replaced that with direct, content-addressed connections: every position can read from every earlier position in one step, and which one it reads from is learned.

From the vector x at each position, three learned matrices produce three different vectors:

• Query q = W_Q x — "what am I looking for?"
• Key k = W_K x — "what do I offer, as an advertisement?"
• Value v = W_V x — "what do I actually hand over if you pick me?"

Those are per-position vectors. Stack the query for every position in the sequence as the rows of one matrix and you have Q, of shape [T, d_k]; the same stacking gives K and V. The capital letters in the equation below are always the whole sequence at once.

The database analogy is exact enough to be worth stating once: it is a soft lookup. A hard lookup matches one key and returns its value. Attention matches every key to a degree, and returns a blend of all values weighted by how well each matched.

Keys and values are separate on purpose. The thing that makes a position findable need not be the thing it contributes. A position holding the word Paris might advertise "I am a city name" via its key while contributing "capital, France, European" via its value.

Attention(Q, K, V) = softmax( QKᵀ / √d_k + M ) V

QKᵀ — every query dotted with every key, giving a [T, T] score matrix. Entry (i, j) is how relevant position j is to position i. This is the source of the quadratic cost: double the sequence, quadruple the matrix.

÷ √d_k — the scaling. Dot products of d_k-dimensional vectors grow in magnitude roughly like √d_k. Without scaling, with d_k = 128 the scores are large, the softmax saturates into a near one-hot spike, and its gradients vanish. Dividing by √d_k keeps scores at a scale where the softmax stays soft and trainable. It is a small term with a large effect.

+ M — the causal mask. Set every entry where j > i to −∞ so that after the softmax those weights are exactly zero. Position 5 cannot see position 6. Without this the model would trivially cheat during training by reading the answer it is being asked to predict, and would be useless at generation time when the future genuinely does not exist.

softmax — turn each row of scores into weights that are positive and sum to 1.

× V — the output at position i is the weighted average of all value vectors, using row i's weights. That result is added back into the residual stream.

Rows are queries, columns are keys. Row i sums to 1 and shows where position i looked. The upper-right triangle is always blank because of the causal mask.

Some patterns recur across essentially every trained model, and recognising them is genuinely useful when debugging:

• The diagonal — attending to yourself. Very common in early layers.
• Previous-token heads — a stripe one step off the diagonal, effectively reconstructing local order.
• Attention sinks — a strong column on the very first token. When a head has nothing useful to attend to it still must distribute weight summing to 1, so it dumps it on a neutral position. This is a real phenomenon with practical consequences: evicting the first token from a KV cache degrades quality badly.
• Induction heads — the pattern behind in-context learning. Having seen … A B … earlier, on encountering A again the head attends to the token that followed A last time and copies it. This is how a model picks up a format from your examples in the prompt, and we return to it in Lesson 15.