Core techniques · 1
Adam: an adaptive optimizer
Plain SGD takes the same step size for every parameter. Adam — Kingma & Ba, 2014 — keeps a running average of each parameter's gradient and of its square, and steps by their ratio, so a weight with small but steady gradients still moves. On an ill-conditioned problem it converges where SGD crawls. Everything below runs in this page — edit any block and press Run.
One loop, two optimizers
Here one feature is thirty times larger than the others. SGD has to keep its step small enough for that big feature, which leaves the rest barely moving; Adam gives each parameter its own effective rate. Same model, same data, same steps — only the optimiser differs.
A rate per parameter
Adam keeps two running averages for every weight: the gradient m and its
square v. The update is m / (sqrt(v) + eps), so a parameter
is scaled by how large its own gradients have been. betas are the two
averages' decay rates — the defaults [0.9, 0.999] are what the paper
recommends and rarely need changing.
Your turn: give Adam a learning rate
The model below should learn the line y = 3x + 1, but Adam cannot move
while the step size is zero. Set a real rate — 0.2 lands it well inside
the step budget here — and the page says when it has it.
Watch Adam converge
The comparison above gave two numbers; here is the curve. Adam drives the loss down on the same ill-conditioned problem where plain SGD only crawled — its per-parameter rate is what makes the difference visible.
The rate itself is fragile
Adam gives each parameter its own effective rate; with plain SGD the one rate is yours to choose, and the choice is unforgiving. The same small regression is trained three times below, changing only the learning rate — each row of the heatmap is a run, left to right the steps, brightness the loss. One row diverges, one converges, one barely moves.
What to remember
- Adam keeps two running averages per parameter — the gradient and its square — and steps by their ratio.
- That gives each parameter its own effective rate, so it converges where SGD crawls on ill-conditioned problems.
betas([0.9, 0.999]) are the averages' decay; the defaults rarely need changing.- Adam normalises its own steps, so it tolerates a wider learning rate than SGD.