Core techniques · 4

Layer normalization

Layer normalization — Ba, Kiros & Hinton, 2016 — normalizes each sample across its own features, not across the batch the way batch norm does. That makes it independent of batch size and identical in train and eval — which is why transformers reach for it instead. Everything below runs in this page, in both languages.

It normalizes each row across its features

Give it rows with wildly different means and spreads. LayerNorm([4]) centres and scales each row on its own, over its four features — so every row of the output comes out with mean about zero.

Batch-independent, and the same in train and eval

Because each row is normalised on its own, a row's answer does not depend on the batch around it — and there are no running statistics, so train() and eval() do the same thing. This is exactly the property batch norm lacks.

Your turn: centre the rows

The block reports the largest row mean of its output. Raw rows carry big means, so it is far from zero — pass x through the LayerNorm so each row is centred and the largest row mean drops to zero.

Rows centred, drawn

A batch with uneven row means goes in (top) and comes out with every row centred (bottom). Where batch norm evened the columns, layer norm evens the rows — each sample on its own, which is why it needs no batch.

What to remember

Layer norm normalises each sample across its own features (its row), not across the batch.
So it is batch-independent and identical in train and eval — there are no running statistics.
That is why transformers use it: attention has no fixed batch structure to normalise over.
In borch: nn.LayerNorm([features]).

If a block above errors, that is worth knowing. Every example on this page runs against the same library the tests run against — nothing here is a screenshot. Press Reset to get the original code back.