Capstones · 2

Say what you measured on

A model reports an accuracy of 0.92 and it feels like a fact. It is not one yet. A number is a measurement only when it carries two things it almost never does: what it was measured on — the images the model learned, or images it never saw — and how wide the measurement is, because a number read off forty examples can be ten points from the truth. torch.workbench trains a model in this tab and keeps both. Here you make it lie, then make it tell the truth.

Every accuracy has a width

Before any model, the arithmetic. An accuracy measured on n held-out examples carries a standard error; at ninety-five per cent confidence that is a band around the number, and telling two models apart needs a gap wider still — two estimates, each with its own band. This runs anywhere, no GPU required; it is exactly what workbench.interval and workbench.resolution compute for a real run.

This lesson's real run is Python, in a tab with WebGPU. torch.workbench is a Python-surface tool — it trains a model, so it lives where the autograd is, not in borch.ts. On a machine with no adapter the blocks below say so in a sentence rather than a traceback; the arithmetic above runs everywhere.

Make it lie, then make it tell the truth

One dataset, one model, trained twice — the only change is val, the share of images held back for scoring. With val=0 the accuracy is read off the very images the model trained on; with val=0.3 it is read off images it never saw. The set here is pure noise with labels that mean nothing, so there is nothing to learn — which is what makes the gap between the two numbers the whole lesson.

The first run scores high on the images it memorised. The second, on images it never saw, falls to roughly one-in-three — chance, for three classes, which is the honest report: there was no signal, and the held-out split is what says so. A number without measured_on beside it hides exactly this difference.

And how wide is the honest number?

A held-out accuracy is itself measured on a sample, so it too has a width. A small split gives a wide one — the run below reports how far apart two models would have to score before the difference is more than the luck of which rows landed in the split.

What to remember

An accuracy without measured_on is not a measurement — held-out and the training images are different claims, and only one generalises.
Never score on the images the model trained on; hold a share back with val and score there.
Every held-out number has a width set by how many rows it was measured on; a small split cannot separate close models.
torch.workbench trains in the tab and keeps both facts — measured_on and resolution — so the number reports itself honestly.

If a block above errors, that is worth knowing. Every example on this page runs against the same library the tests run against — nothing here is a screenshot. Press Reset to get the original code back.