Core techniques · 2

Dropout: training with noise

Dropout — Srivastava et al., 2014 — zeros activations at random while training, so the network cannot lean on any single unit; it is cheap, strong regularisation. At evaluation it switches off and becomes the identity — and forgetting that switch is the single most common beginner bug. Everything below runs in this page, in both languages.

It zeros activations at random

A Dropout(0.5) layer, while training, keeps each value with probability 1 - p and zeros the rest — then scales the survivors by 1 / (1 - p) so the average magnitude is unchanged (inverted dropout).

Different every run — until eval()

Because the mask is random, two forward passes in training mode disagree. Call eval() and the layer becomes the identity: the same input gives the same output, every time. train() switches it back on.

Your turn: measure in eval mode

The block measures the same input twice and reports the gap between the two runs. It should be zero — a measurement must be repeatable — but the layer is still dropping. Switch it to eval() so the two runs agree.

Dropout, drawn

A grid of ones through a dropout layer in training mode: about half the cells go dark (zeroed) and the rest brighten (scaled up). Run it again for a different mask — that randomness is the regularisation.

What to remember

• Dropout zeros each activation with probability p while training, and scales the survivors by 1/(1-p).
• It stops the network leaning on any single unit — cheap, strong regularisation.
• At eval() it turns off and becomes the identity; forgetting the switch is the classic bug.
• In borch: nn.Dropout(p), toggled by train() / eval().