Foundations · 6

Eigenvalues & the SVD

Last chapter you solved with a matrix; this one takes it apart. Almost everything a matrix does to space is captured by a handful of numbers — and finding them is how compression, PCA, recommender systems and low-rank fine-tuning (LoRA) all work. Two decompositions do the taking-apart: the eigendecomposition, for the directions a matrix only stretches, and the SVD, which generalises it to any matrix. Everything below runs in this page, in both languages.

Eigenvectors: the directions a matrix only stretches

Multiply most vectors by a matrix and they turn. A special few come out pointing the same way, only longer or shorter — those are its eigenvectors, and the factor each is scaled by is its eigenvalue: A v = λ v. They are the axes along which the matrix acts simply.

For a symmetric matrix the eigenvalues are real and the eigenvectors are orthogonal, so eigh is the tool — it returns the eigenvalues in values and the eigenvectors as the columns of vectors. Take the first pair and confirm that A·v really is just λ·v.

The SVD: any matrix, three simple pieces

Eigenvectors need a square matrix; most matrices in machine learning are not square. The singular value decomposition has no such limit. It writes any matrix as A = U diag(S) Vᵀ — geometrically, rotate (Vᵀ), stretch along the axes by the singular values S, then rotate again (U). The singular values are non-negative and sorted largest first: they rank how much each direction matters.

Why a few numbers are enough

Here is the payoff, and it is easier to see than to say. The matrix below is secretly built from only three independent directions, with a little noise on top. Run it and look at the plotted spectrum: the first three singular values tower over the rest. That cliff is why images, embeddings and weight matrices compress — keep the few large singular values, drop the long tail, and you have thrown away almost nothing. It is also exactly what LoRA exploits when it fine-tunes with a low-rank update.

Your turn: rebuild A from its SVD

Putting the pieces back is the check that you have them right. The block reconstructs A from U, S and V but leaves off the transpose on V. Add it so U diag(S) Vᵀ returns A and the gap falls to zero.

What to remember

• Eigenvector: a direction a matrix only scales — A v = λ v. Use eigh for symmetric matrices (real eigenvalues, orthogonal eigenvectors).
• SVD: any matrix is U diag(S) Vᵀ — rotate, scale by the singular values, rotate. svd returns U, S, V.
• Singular values are sorted and usually decay fast. Keeping the top few is low-rank approximation — the idea under compression, PCA, and LoRA.
• In borch: A.eigh(), A.svd(); both read back with await.