Tutorial · 3

Curve fitting

A straight line cannot bend. This tutorial shows what a hidden layer and a non-linearity actually add, by fitting a curve that a linear model provably cannot reach — and by looking at the fit rather than the loss number.

1 · The target, and what a line can do

The target is sin(3x) · x on [−2, 2]. First fit it with a single linear layer, then look at the result. The loss will stop going down and the picture explains why.

2 · Add one hidden layer

Two linear layers with a Tanh between them. Without the non-linearity this would still be a line — two matrix multiplies collapse into one. The activation is the entire difference, and you can test that claim by deleting it.

3 · Look at it

Numbers in a column are hard to judge. Draw both curves into an image instead — the truth in one channel, the fit in another, so where they agree the line turns pale and where they disagree it splits into colour.

4 · Where it fails

Ask it about x it has never seen. Neural networks interpolate; they do not extrapolate, and a fit that looks perfect inside its range can be nonsense one step outside it. This is worth seeing once, early.