Core techniques · 3

Batch normalization

Batch normalization — Ioffe & Szegedy, 2015 — normalizes each feature across the batch to zero mean and unit variance, which lets deeper networks train at higher learning rates. Like dropout it behaves differently in train and eval mode: at eval it uses fixed running statistics, not the batch in front of it — and mixing that up is a classic bug. Everything below runs in this page, in both languages.

It normalizes a batch to mean 0, variance 1

Give it a batch whose features have a large mean and spread. In training mode it subtracts each feature's batch mean and divides by its batch standard deviation, so every column of the output comes out centred and scaled.

Train uses the batch; eval uses running statistics

In training mode a row's output depends on the other rows in its batch — the same query normalised against a different batch gives a different answer. In eval mode the layer uses the running statistics it accumulated (here still the defaults, mean 0 and variance 1), so the batch it sits in no longer matters.

Your turn: a stable answer needs eval()

The block runs the same query row through the layer in two different batches and reports how far its answer moves. For inference that should be zero — the batch a sample happens to sit in must not change its prediction. Switch the layer to eval().

Before and after, drawn

A batch with large, uneven column means goes in (top) and comes out centred and scaled (bottom). Batch norm evens the columns so the next layer sees a well-behaved distribution.

What to remember

• Batch norm normalises each feature across the batch to mean 0, variance 1.
• It lets deeper networks train at higher learning rates.
• train uses the batch's own statistics; eval uses fixed running statistics — a sample's answer must not depend on its batch at inference.
• In borch: nn.BatchNorm1d / 2d, toggled by train() / eval().