17. Making models cheap: quantization, LoRA, MoE, speculative decoding

Weights are trained in 16-bit floats. They do not need to stay there. Quantization stores them in fewer bits, mapping the original range onto a coarse grid.

Why this helps so much: from Lesson 11, decode is memory-bandwidth-bound. Halving the bytes read per token roughly halves the time per token. You get both a memory saving and a speedup, which is unusual.

Two approaches. Post-training quantization (GPTQ, AWQ) converts an existing model using a small calibration set — minutes to hours, no retraining. Quantization-aware training simulates the rounding during training so the model adapts to it; better results, much more expensive.

The subtlety that makes 4-bit work at all is outliers. A small number of activation dimensions have magnitudes far larger than the rest, and naively quantizing them destroys quality. Modern methods handle those channels separately, or quantize in small blocks with per-block scales rather than one scale for a whole tensor.

Full fine-tuning of a 70B model means updating all 70B weights and storing optimizer state for each — hundreds of gigabytes of GPU memory. Low-Rank Adaptation avoids nearly all of it.

The observation: the change a fine-tune makes to a weight matrix is empirically low-rank. So freeze the original W and learn a product of two thin matrices:

W' = W + BA    where B is [d, r], A is [r, d], r ≪ d

For a 4096×4096 matrix, full fine-tuning trains 16.8M parameters. With rank 8, LoRA trains 2 × 4096 × 8 = 65,536 — about 0.4%. Memory drops accordingly, since optimizer state is only needed for the trainable parameters.

Practical consequences:

• Adapters are tiny (megabytes), so you can keep dozens per base model and swap them per request.
• BA can be merged into W after training, giving exactly zero added inference latency.
• QLoRA combines a 4-bit frozen base with LoRA adapters trained in higher precision, which is what makes fine-tuning a 70B model on a single GPU possible.

The limit: LoRA adapts behaviour and style well. It is a poor tool for injecting large amounts of new factual knowledge — retrieval is the right instrument for that.

In a dense model, every parameter participates in every token. MoE breaks that assumption.

Replace each feed-forward network with N parallel FFNs ("experts") plus a small router. For each token, the router picks the top k experts (commonly 2 of 8, or 8 of 256) and only those run. The result is weighted by the router's scores.

So a model can hold, say, 8× the parameters while doing roughly the same arithmetic per token. This is the total versus active parameters distinction: a model advertised as 47B total / 13B active has 47B parameters worth of capacity and 13B parameters worth of per-token compute.

The costs are real:

• All experts must be in memory even though only a few run, so VRAM tracks total parameters, not active ones.
• Routing must be load-balanced or a few experts get all the traffic; an auxiliary balancing loss is standard.
• Training is less stable, and distributed serving involves substantial cross-device communication.

MoE is why some very capable models are surprisingly fast: you are paying inference cost for a much smaller model than the parameter count suggests.

Decode is sequential and bandwidth-bound: one token per full read of the weights, with the GPU's arithmetic units mostly idle. Speculative decoding exploits that idle capacity.

1. A small, fast draft model generates the next k tokens cheaply — say 5.
2. The large model processes all 5 candidates in a single forward pass, in parallel, and computes what it would have predicted at each position.
3. Accept the longest prefix where the draft matches what the large model would have sampled; reject the rest and continue from there.

The remarkable property, given the right acceptance rule, is that the output distribution is identical to running the large model alone. It is not an approximation — it is pure latency optimisation, typically 2–3× on easy text where the draft agrees often.

It works because verifying 5 tokens costs nearly the same as generating 1: both require reading all the weights once, and the extra arithmetic is free capacity. Variants include Medusa (extra prediction heads instead of a separate draft model) and n-gram lookup drafting for repetitive text.