8. The transformer block

Modern (pre-norm) transformer block, exactly as implemented:

x = x + Attention(Norm(x))
x = x + FFN(Norm(x))

Two sub-layers. Each one normalises a copy of the input, computes something, and adds the result back to the untouched original. Note what is not happening: nothing is ever overwritten. Each sub-layer contributes an increment.

The division of labour is clean:

• Attention moves information between positions. It is the only part of the model that looks sideways.
• The FFN transforms each position independently. It never looks sideways at all.

Interleaving them is what lets a stack of blocks build progressively richer, context-aware representations: gather, process, gather, process.

The x + is easy to overlook and is doing two crucial jobs.

Gradient flow. In a plain deep stack, the gradient is multiplied by each layer's Jacobian on the way back; over 96 layers those factors compound and the signal explodes or vanishes. A residual connection gives the gradient an identity path — an additive route straight from the loss to the earliest layers. Without it, models this deep would not train at all.

The residual stream as a channel. The more modern reading: the vector flowing through the model is a shared communication bus. Every sub-layer reads from it (through its projection matrices) and writes to it (by adding). An early layer can deposit a piece of information that a layer 40 blocks later retrieves, without any intervening layer needing to preserve it deliberately.

This picture also explains a practical fact: because everything is added into one stream, features occupy directions in that space, and different circuits communicate by writing to and reading from different directions.

LayerNorm takes the vector at a position, subtracts its mean, divides by its standard deviation, then scales and shifts by learned parameters γ and β. It keeps activation magnitudes in a stable range so that training does not drift into overflow or collapse.

RMSNorm drops the mean subtraction and the shift, dividing only by the root-mean-square. It is cheaper, empirically just as good, and is what Llama and most recent models use.

The placement question turned out to matter more than the choice of norm:

• Post-norm (original 2017): x = Norm(x + Sublayer(x)). The residual path itself gets normalised, which weakens the clean identity route. Deep post-norm models need careful learning-rate warmup and are prone to divergence.
• Pre-norm (everything modern): x = x + Sublayer(Norm(x)). The residual path stays untouched all the way from input to output. Far more stable, trains at higher learning rates, scales to great depth.

One consequence of pre-norm: the residual stream itself is never normalised on its way through, so its magnitude grows steadily with depth — visible in the figure below. Every pre-norm model therefore ends with a final norm after the last block, before the output projection. It is not an optional extra; without it the unbounded stream hits the output projection directly.

For one block with d = d_model and a 4× FFN, ignoring biases and norm parameters:

So a 32-layer model with d_model = 4096 holds roughly 32 × 12 × 4096² ≈ 6.4 billion parameters in its blocks, plus the embedding table. The useful rule of thumb: parameters ≈ 12 × n_layers × d_model², with about two thirds of it in the feed-forward networks.