4. Neurons, matrices and nonlinearity

Last lesson ended with the embedding table "handing off" to the transformer blocks. But notice what that handoff actually requires: something that can change the arrows. A lookup table can't — it just reads out a fixed row. The whole model, from the second station onward, is chains of things-that-change-numbers. This lesson builds the smallest one, from scratch.

Here is the whole design of a "neuron" (in quotes because it has nothing to do with biology beyond being small and numerous). It does exactly four arithmetic steps:

1. Multiply each input by a private number of its own. Those numbers are called weights — one per input, learned during training. A weight is a volume knob: 2.0 means "amplify what this input says", −1.0 means "treat it as evidence against me", 0 means "ignore it".
2. Add the products up. One number so far: a single verdict compiled from all the inputs.
3. Add one more learned number, the bias. Think of it as a threshold of doubt: it shifts the verdict up or down before anyone looks at it, deciding how easily this neuron gets excited at all.
4. Squash the result through a small fixed function — the activation. The standard one, ReLU, just does max(0, result): pass positive numbers through unchanged, clamp negatives to zero. "Silent unless you have something positive to say."

Do it by hand, with the exact numbers the widget below starts on. Inputs [2, 1, 0.5], weights [0.5, −1.0, 2.0], bias 1.0, ReLU:

step 1+2: (0.5×2) + (−1×1) + (2×0.5) = 1 − 1 + 1 = 1
step 3:      1 + bias 1 = 2
step 4:      ReLU(2) = 2  (positive, passes unchanged)
output: 2

That's a neuron. Written as one formula — which is how you'll see it everywhere — the same thing is y = f(w₁x₁ + w₂x₂ + … + b), and now you can read it: products, then sum, then bias, then squash.

And here is the connection to last lesson, which makes this more than arithmetic trivia. Step 1+2 — multiply matching pairs and add — is the dot product you just learned. Remember what a dot product measures: how much two arrows point the same way. So a neuron is really asking: "how much does this input point in my direction?" — where "my direction" is its weight vector, one of thousands of little questions the model learned to ask. The squash (step 4) then decides how loudly to answer: silence, or a number proportional to the match.

In the widget below, drag the input sliders and watch the four steps recompute line by line — then switch the activation to "none" and drag an input until the sum goes negative. That difference is the entire subject of section 3.

One neuron asks one question. A useful layer asks thousands — side by side, in one move. Put, say, 3 neurons next to each other, each with its own weight vector, and stack those vectors as rows of a rectangle of numbers. A rectangle of numbers is a matrix; feeding one vector into that row of neurons is the same arithmetic as multiplying the matrix by the vector. This is why you'll hear "a layer is a matrix multiplication" — it's not a metaphor, it's bookkeeping for "lots of dot products at once".

Do one by hand — three numbers in, two questions asked. The input vector [1, 2, 0] meets two neurons with weights [2, 0, 1] and [1, 3, 0]:

question 1: (1×2) + (2×0) + (0×1) = 2
question 2: (1×1) + (2×3) + (0×0) = 7
answer: [2, 7]

Three numbers in, two numbers out — and every answer is one dot product: first number × first number, plus second × second, plus third × third, read straight across. That's the whole mystery of "shapes", which is the one thing worth practicing here. A shape like [2, 3] just means "2 rows of 3 numbers". The rule for what's legal: the inner numbers must match and cancel — [2, 3] × [3, 2] → [2, 2]. The widget below is exactly this computation — the same input row [1, 2, 0] and the same two neuron rows — with the answer grid already filled in; hover any output cell and it shows you which row and column produced it.

Two facts about why this matters beyond arithmetic practice:

It's all one operation. The model never processes "one neuron" or even "one layer" as a separate event — a whole block of thousands of dot products is fired as a single matrix multiply, and a whole sentence's worth of tokens is batched into the same multiply (a [T, d] slab of numbers, one row per token). That batching is exactly why graphics cards — built for giant matrix multiplies — are what LLMs run on. Essentially all of an LLM's compute is this one operation, done in different shapes.

One warning for when you read code. You'll see the same layer written two ways: Wx with the weight matrix shaped [out, in], or xW with it shaped [in, out]. Nothing changes but the order of writing — but if you mix the two conventions, every shape you check will look wrong. Before trusting a shape in a paper or a codebase, check which convention it writes in.

You caught a preview of this in the neuron widget: with the activation switched to ReLU, a negative sum becomes zero; with "none", the negative number passes straight through. Small detail, huge consequence. This section is why.

First, a name. A function is linear when scaling the input scales the output in exact proportion — double what goes in, double what comes out — and when you can chain two of them into one. Multiplying by 2 is linear. max(0, x) is not: double −4 and you get 0 either way, not double zero. The activation functions are all deliberately not linear — that's the whole point, and here's why:

Suppose you skipped the squashing and just stacked plain matrix multiplies. Two layers in a row: the output is W₂(W₁x). But doing "multiply by W₁, then multiply by W₂" is the same as multiplying once by one combined matrix — you can check this on tiny examples, and it holds for any size. So a hundred stacked linear layers are, mathematically, one layer wearing a costume. Depth would buy nothing. The squashing between layers is what stops each new layer from collapsing into the previous one.

Why does depth matter at all? Because the squashing lets layers compose in ways a single multiply never can. A linear map can only stretch, shrink and rotate the whole space uniformly — like dragging a sheet of graph paper: straight lines stay straight, parallel lines stay parallel. A bent function can fold the space. And some questions need folding: imagine the words of a sentence as dots on a plane where "animal" dots and "verb" dots each form a little cloud, but the clouds are intermingled. No single straight cut separates them — but bend the plane once and they come apart. Each squashed layer adds a little more bending, and the model's thousands of tiny decisions are built from those bends.

The specific squashes you'll meet, all plotted in the figure below (they differ only in how they bend, not whether):

• ReLU — max(0, x). The workhorse: cheap, and it made deep networks trainable. Its flaw: a neuron stuck in the negative region outputs zero no matter what arrives, and learns nothing more — "dies". The widget lets you drive a neuron into that state.
• GELU — ReLU with a soft knee: small negatives slip through faintly instead of being clamped hard. The default in GPT-family models.
• SwiGLU — used in Llama and most current open models. Not one function but a wiring pattern: it splits the layer's numbers into two halves and lets one half act as a volume knob on the other ("gate" it) — because whether a fact matters often depends on a second fact, and a multiplication between the halves expresses that. It needs three weight matrices instead of two, so implementations shrink the hidden width (to about ⅔ × 4 × d_model) to keep the parameter count level — and still come out better on quality. Its raw ingredient is SiLU, x·sigmoid(x), the soft-knee function plotted below.
• tanh — the pre-2012 default, shown for contrast. It squashes everything between −1 and +1, which sounds tidy, but "push any input, however extreme, and the output barely moves" is exactly the flatness that starves learning. Escaping it is what ReLU was invented for.

Everything built so far — neurons, their matrix form, the squashing between — assembles into the part of a transformer block that does the private thinking (the other part, attention, which is about words consulting each other, is next lesson). It's called the feed-forward network, and it is exactly: widen, squash, narrow:

FFN(x) = W_down · activation(W_up · x)

Read it right to left, the way you read a pipeline: the arrow arrives as 4096 numbers; W_up expands it to 16384 (four times wider); the activation squashes each of those 16384 numbers; W_down contracts the result back to 4096, and that's what continues down the line.

Why expand at all, and why squeeze back? Think of the wide middle as a row of inspectors. 4096 numbers arrive; 16384 inspectors each ask one small question of them ("is this heading toward a verb?", "does this look like the second half of a phone number?" — the questions are learned, not designed, and they don't correspond neatly to English). Each inspector who finds something adds a note. The squeeze at the end is the editor: it decides which notes actually matter for what happens next. Without the wide middle there aren't enough inspectors; without the squeeze, every note would get shouted downstream.

The sizes are not decorative: at this default width the two matrices hold roughly 134 million numbers per layer, and the feed-forward parts of a model hold roughly two thirds of all its parameters — attention takes most of the rest. If you ever wondered where "70 billion parameters" are hiding, this is where most of them are. (SwiGLU, the gated pattern from the last section, is the same expand-squash-contract with three narrower matrices instead of two; GPT-family models use the two-matrix layout above.)

One more reason to care where the parameters are: interpretability research suggests this expand/contract structure is where much of the model's factual knowledge lives — the up-projection acting like a set of keys, the down-projection like the values they retrieve. Lesson 1 said there is no database; this is the machinery that plays that role instead.