Tutorial · 9

Autoencoder

No labels here. The network is asked to reproduce its own input through a gap two numbers wide, which it can only manage by finding what the pictures actually vary by. The pictures are made with exactly two degrees of freedom, so afterwards you can check whether it found them — most of this tutorial is that check.

1 · Two hundred and fifty-six pictures with two facts in them

Each picture is a soft dot on a 16×16 field, and the only thing that changes between them is where the dot is. Every image is 256 pixels; every image is also two numbers. Nothing in the training tells the network that.

2 · The gap

Two convolutions down to a code of 2, then nn.ConvTranspose2d — a convolution run backwards, which is how you get bigger rather than smaller — back up to 16×16. The loss compares the output with the input, so no labels exist and none are needed.

3 · What the two numbers turned out to be

Nobody told it about coordinates. Encode the test pictures and compare each code against the (cx, cy) they were painted from — not by eye but with a correlation, and both codes against both coordinates, because the network is free to swap, flip or rotate the pair and would still be right.

A high correlation in a row that is not the diagonal is not a failure — it means the network numbered the world in its own order. What would be a failure is a table of small numbers everywhere, and that is worth knowing how to read before you trust any latent space.

4 · Codes that came from no picture

The decoder takes two numbers and returns an image, and nothing stops those numbers from being ones no picture ever produced. Walking from one real code to another gives the frames in between; walking well outside the range they occupy gives what the model has never had a reason to get right.

The line between two codes is a fair question and the answer is a dot sliding across the field. The far ones are not — an autoencoder is a compressor, not a generator, and it makes no promise whatsoever about codes it never saw. Models that do make that promise (a VAE, a diffusion model) buy it by training for it, and the price is in the loss function rather than in the architecture.