11. Context windows and the KV cache

Naive generation is absurdly wasteful. To produce token 101 you run the model over tokens 1–100. To produce token 102 you run it over tokens 1–101 — recomputing everything you just computed.

The causal mask makes the fix possible. Position 50 can only attend to positions ≤ 50, so adding token 101 cannot change anything about position 50. Its key and value vectors are final the moment they are computed.

So cache them. For every layer and every head, store the K and V vectors of every token processed so far. Generating a new token then requires computing Q, K, V for one position, attending against the whole cache, and appending the new K and V.

Note that only K and V are cached, never Q. A query is used once, at the step that created it, and then is of no further use.

This split is the single most useful thing to understand about LLM serving.

Prefill — processing your prompt. All prompt tokens go through the model together in one parallel pass, filling the cache. This is a big matrix–matrix multiplication: the GPU is doing heavy arithmetic and is compute-bound. Cost scales with prompt length (plus a quadratic term from attention). It determines your time-to-first-token.

Decode — generating the response, one token at a time. Each step is a matrix–vector multiplication: to produce a single token, the GPU must read every weight in the model from memory and use each one for very little arithmetic. The chip is starved for data, not for compute. Decode is memory-bandwidth-bound.

Consequences that surprise people:

• Decoding one token at a time uses a tiny fraction of a GPU's arithmetic capability. The hardware is mostly idle, waiting on memory.
• The weights are not the only thing read each step — the attention at every layer reads the whole KV cache too. That is why decoding slows down as a conversation grows: at 8k tokens the cache in the example below is 21.5 GB against 140 GB of weights, but at 128k it is 344 GB and has become the larger of the two reads.
• Batching is nearly free during decode. The weights are read once and shared across all sequences in the batch, so serving 32 users costs barely more time per step than serving one. This is why batch throughput is the metric providers optimise.
• Making weights smaller — quantization, Lesson 17 — speeds up decode almost linearly, because the bottleneck is bytes moved, not maths done.
• Output tokens are typically priced higher than input tokens, and this is why: input is processed in parallel, output is strictly sequential.

cache_bytes = 2 × n_layers × n_kv_heads × d_head × seq_len × bytes_per_value

The 2 is for K and V. Worked example — a 70B-class model: 80 layers, 64 heads with d_head 128, fp16 (2 bytes), 8k tokens:

2 × 80 × 64 × 128 × 8192 × 2 ≈ 21.5 GB

That is on top of 140 GB of weights, for a single sequence. Now serve 20 users concurrently and the cache alone is 430 GB. This is the reason grouped-query attention exists: with 8 KV groups instead of 64 heads, that 21.5 GB becomes 2.7 GB.

Other techniques in production use: paged attention (vLLM), which stores the cache in fixed-size blocks like operating-system virtual memory to eliminate fragmentation; prefix caching, which reuses the cache for a shared system prompt across requests; and cache quantization to 8 or 4 bits.

The context window is the maximum number of tokens in a single forward pass — prompt plus generated output together. Hitting it means one of three things happened:

• The position encoding runs past what the model was trained on (Lesson 7).
• The KV cache no longer fits in memory.
• The provider set a limit for cost reasons.

Practical strategies when you exceed it: sliding-window attention (each token only attends to the last N, keeping cost linear but genuinely losing distant information); summarising older conversation turns; retrieval instead of stuffing (Lesson 16); or the attention sink trick, keeping the first few tokens permanently in the cache alongside a sliding window, which preserves quality far better than a naive window.