Tutorial · 10

Least squares

Not everything is trained. When a model is linear in its parameters the best answer can be computed outright — no learning rate, no steps, no seed — and linalg is the part of borch that does it. This tutorial solves what can be solved, then spends the rest of its time on the more useful skill: knowing when an exact answer is worth nothing.

1 · The line, twice

Forty noisy points off a straight line. Fit it with gradient descent the way the earlier tutorials do, then fit it again with linalg.lstsq, which is one call and no loop. They land on the same place because they are answering the same question.

lstsq is not an approximation of the trained answer — it is the exact minimum, and the trained one approaches it. When your model is linear in its parameters, reaching for an optimizer is a choice to arrive later, and the reason to make that choice anyway is size: the closed form wants the whole matrix at once.

2 · Exactly the wrong question

Twelve noisy points from a smooth curve, fitted by polynomials of degree 1, 3, 5 and 11. Degree 11 has as many unknowns as there are points, so it passes through every one of them and its training error goes to zero. Then each is asked about the two hundred points it never saw.

The last row is the useful one. linalg.cond says how much an answer can move when the input moves a little, and it is computable before you look at any result — a large condition number is a warning that arrives in time to be acted on, unlike a test error, which arrives after you have already believed the fit.

3 · PCA is a singular value decomposition

A cloud of two-dimensional points that mostly lie along one direction. Centre it, take linalg.svd, and the rows of vt come out as the directions of the cloud, longest first, with s saying how long. Nothing is fitted; this is a property of the matrix.

4 · How many dimensions are actually in there

Eight columns built out of three hidden factors, plus noise. The singular values say how much each direction carries, and the answer is supposed to be three. Counting them means choosing where "small" begins, so this counts twice — once at a threshold written down here, and once by linalg.matrixRank, which chooses its own.

While the noise stays well under the signal there is a cliff after the third value and a person counts three; by the last row there is no cliff and the count is whatever threshold was picked. linalg.matrixRank answers eight throughout, and it is not wrong — its threshold is the numerical one, near float32's precision, and a direction a thousandth the size of the largest is still not an absent one. It answers "is anything there at all", where the question was "is anything there worth keeping", and only one of those two is about your data.