7. Position: how the model knows word order

Look again at the attention equation. Scores are dot products between queries and keys; the output is a weighted sum. Shuffle the input positions and every score simply moves with them — the computed values are identical. Self-attention treats its input as an unordered set.

That is a genuine problem, since word order carries a great deal of meaning. It is also a genuine feature: it is exactly what allows all positions to be processed in parallel, which is why transformers train so much faster than recurrent networks. The transformer's answer is not to restore sequential processing but to put position information into the vectors themselves.

The original 2017 transformer added a fixed sinusoidal vector to each token embedding. Dimension pairs oscillate at geometrically spaced frequencies:

PE(pos, 2i) = sin(pos / 10000^(2i/d))    PE(pos, 2i+1) = cos(pos / 10000^(2i/d))

Fast dimensions distinguish neighbouring positions; slow ones distinguish far-apart regions. Together they give every position a unique fingerprint, a bit like the digits of a binary counter running at different rates.

GPT-2 and BERT instead used learned absolute embeddings: one trainable vector per position. Simple and effective, with one fatal limitation — position 4000 has no vector at all if you only trained up to 2048. The model cannot extrapolate one step beyond its training length.

Rotary position embedding is what essentially every current model uses, and the idea is neat.

Take the query and key vectors, split them into 2-dimensional pairs, and rotate each pair by an angle proportional to the position. Different pairs rotate at different rates, again geometrically spaced — pair i turns by pos / 10000^(2i/d) radians, the same base as the sinusoidal formula above. The first pairs complete a full turn every few tokens; the last ones barely move across the whole context. That spread is what the interpolation methods below act on: stretching the slow pairs buys range, and leaving the fast ones alone keeps neighbouring positions distinguishable.

The payoff is in what happens to the dot product. Rotating two vectors by angles mθ and nθ makes their dot product depend only on the difference (m − n). So the attention score between two positions automatically encodes how far apart they are, not where they sit absolutely. The same phrase gets the same internal relationships whether it appears at position 10 or position 10,000.

Other properties that matter in practice:

• It is applied to Q and K inside every attention layer, not added once at the bottom.
• It touches no parameters — it is a fixed geometric transform.
• Attention naturally decays with distance, which is a sensible prior for language.
• Because the encoding is relative and continuous, you can stretch it after training to extend context.

A model trained on 8k tokens does not simply work at 128k. The rotation angles at position 100,000 are outside anything it saw, and quality collapses. The common techniques:

• Position interpolation — scale positions down so that 32k tokens occupy the angle range the model was trained on. Positions become fractional. Cheap, and works with a short fine-tune, but it compresses local resolution.
• NTK-aware scaling / YaRN — scale the frequency bands unevenly: leave high-frequency (local) dimensions alone and stretch the low-frequency (global) ones. Preserves local precision much better.
• Long-context fine-tuning — after any of the above, train on genuinely long documents so the model learns to use the range.