Learn · 9

Residual networks

Lesson 5 stacked convolutions. Stack enough of them and training stops working — not because the model is too small, but because almost nothing reaches the first layer. A residual network adds one line to each block, + x, and that line is the whole idea. Here you can measure what it carries.

The block

A BasicBlock is two convolutions and a way back. When the block changes the shape — more channels, or a stride that halves the image — the way back cannot be the input itself, so a 1×1 convolution reshapes it. When the shape is unchanged there is no shortcut layer at all: the tensor is added as it arrived.

What the shortcut carries

The argument for + x is usually drawn rather than measured. It does not have to be. Below, the same eight convolutions are stacked twice — once plainly, once with each layer's output added back to its input — and the gradient that reaches the first layer's weight is read out of both.

The two runs share a seed, so the layers start identical. Only the wiring differs.

The ratio is not a constant. It depends on the seed, the depth and the activation, and this page prints whatever this run produced rather than a number typed in once and left there. What survives the change of seed is the direction: the plain stack arrives small, the residual stack does not.

A small ResNet, trained

A stem, two blocks — one keeping the shape, one halving it — then pool, flatten and a linear head. That is the arrangement of every ResNet; a real one differs by repeating the middle, not by structure.

AdaptiveAvgPool2d is here now. torchvision's ResNet ends with that name, and for a year a line copied from there stopped on it: the 1-D and 3-D layers existed, the 2-D one did not, and all three are the same one-line call to adaptivePool("avg", size). This lesson taught a workaround — a GlobalAvgPool module of its own — and the ledger carried the name as a hole, until a review of what was missing found this at the top of the list and wrote it. The workaround is gone from the block above; the layer takes the output size, [1, 1] here, and given a [2, 16, 5, 7] input returns [2, 16, 1, 1], where a fixed AvgPool2d(8) would return [2, 16, 0, 0] and raise nothing.

A ready-made ResNet18Cifar lives in the model catalogue, built from the same Conv2d and batch-norm layers as the block above — nothing in it is a special case the library knows about. The net above takes a [4, 1, 23, 17] input and still answers [4, 3] — that is what the adaptive pool buys, and what a fixed one cannot do.