Learn · 9

Residual networks

Lesson 5 stacked convolutions. Stack enough of them and training stops working — not because the model is too small, but because almost nothing reaches the first layer. A residual network adds one line to each block, + x, and that line is the whole idea. Here you can measure what it carries.

The block

A BasicBlock is two convolutions and a way back. When the block changes the shape — more channels, or a stride that halves the image — the way back cannot be the input itself, so a 1×1 convolution reshapes it. When the shape is unchanged there is no shortcut layer at all: the tensor is added as it arrived.

What the shortcut carries

The argument for + x is usually drawn rather than measured. It does not have to be. Below, the same eight convolutions are stacked twice — once plainly, once with each layer's output added back to its input — and the gradient that reaches the first layer's weight is read out of both.

The two runs share a seed, so the layers start identical. Only the wiring differs.

The ratio is not a constant. It depends on the seed, the depth and the activation, and this page prints whatever this run produced rather than a number typed in once and left there. What survives the change of seed is the direction: the plain stack arrives small, the residual stack does not.

A small ResNet, trained

A stem, two blocks — one keeping the shape, one halving it — then pool, flatten and a linear head. That is the arrangement of every ResNet; a real one differs by repeating the middle, not by structure.

AdaptiveAvgPool2d is here now. torchvision's ResNet ends with that name, and for a year a line copied from there stopped on it: the 1-D and 3-D layers existed, the 2-D one did not, and all three are the same one-line call to adaptivePool("avg", size). This lesson taught a workaround — a GlobalAvgPool module of its own — and the ledger carried the name as a hole, until a review of what was missing found this at the top of the list and wrote it. The workaround is gone from the block above; the layer takes the output size, [1, 1] here, and given a [2, 16, 5, 7] input returns [2, 16, 1, 1], where a fixed AvgPool2d(8) would return [2, 16, 0, 0] and raise nothing.

A ready-made ResNet18Cifar lives in the model catalogue, built from the same Conv2d and batch-norm layers as the block above — nothing in it is a special case the library knows about. The net above takes a [4, 1, 23, 17] input and still answers [4, 3] — that is what the adaptive pool buys, and what a fixed one cannot do.

Serving it — the same forward, replayed

Trained, the network answers questions, and a question is a forward with no tape. Run eagerly, every call encodes its forty-odd dispatches again from JavaScript and the GPU idles between them. borch.compiled(model) puts the model in eval mode, folds each batch norm into the convolution before it, records the forward once for an input shape and replays it after that — the same answer, most of the JavaScript gone. The first call is the slow one: the kernels compile, once per shape on this machine, and the library says so on the console if it takes long. Measure eager first — compiled folds the network in place.

The numbers are this machine's. On an M4 Max the eager forward of the book's ResNet-18 is 1.8 ms and the replay 1.4; on an RTX 5080 the same replay is 0.6 ms while eager varies run to run with the card's clocks. What does not vary: the first call is tens to hundreds of milliseconds (the browser keeps the compiled kernels for the next visit), and every call after it is the replay. A page that must answer at once calls the step once while it loads.

Why depth needs a skip

A stack of layers that each shrink their input a little starves the gradient: run it deeper and deeper and plot how much reaches the input — it falls toward zero. A skip connection is the fix.

What to remember

A skip connection adds the input back — y = f(x) + x — so gradients reach early layers directly.
Without it, gradients shrink with depth (the vanishing gradient drawn above).
A BasicBlock is conv → BN → relu → conv → BN, plus the skip, then relu.
The shortcut gets a 1×1 conv only when the shape changes.