The SVD: every matrix is a rotate, a stretch, a rotate

Eigenvectors are wonderful, but they only exist cleanly for square matrices — and even then only when there are enough of them. Real data matrices are rectangular, and image, embedding and vocabulary matrices are wildly so. The singular value decomposition rescues the entire idea: every matrix, of any shape, is a rotation followed by a non-uniform stretch followed by another rotation.

The matrix below is the sum of three rank-1 pieces with weights 4, 2 and 1. Slide the rank and watch how much of the original survives.

The decomposition #

For any $A \in \mathbb{R}^{m\times n}$ there exist orthogonal matrices $U \in \mathbb{R}^{m\times m}$ and $V \in \mathbb{R}^{n\times n}$, and a diagonal matrix $\Sigma \in \mathbb{R}^{m\times n}$ with non-negative entries, such that

$A = U \Sigma V^\top.$

Written term by term, this is a sum of rank-1 layers:

$A = \sum_{i=1}^{r} \sigma_i\, u_i v_i^\top, \qquad \sigma_1 \ge \sigma_2 \ge \cdots \ge \sigma_r > 0,$

where $r$ is the rank of $A$. The $u_i$ are orthonormal left singular vectors (directions in the output space) and the $v_i$ are orthonormal right singular vectors (directions in the input space). Unlike eigenvectors, this sum always exists and the terms are always mutually orthogonal.

Low-rank approximation #

The reason the SVD is everywhere is a theorem (Eckart–Young): truncating the sum at $k$ terms gives the best possible rank-$k$ approximation in the Frobenius norm,

$A_k = \sum_{i=1}^{k} \sigma_i\, u_i v_i^\top.$

The squared error of the truncation is exactly the sum of the discarded squared singular values:

$\lVert A - A_k \rVert_F^2 = \sum_{i=k+1}^{r} \sigma_i^2.$

So a quickly decaying spectrum means the matrix is compressible. If the first few singular values dominate, you can store $k(m + n)$ numbers instead of $mn$ and lose almost nothing. That is image compression, and it is also why LoRA can fine-tune a large model by adding a tiny low-rank update rather than changing every weight.

Relationships worth knowing #

• With the eigendecomposition. The right singular vectors are eigenvectors of $A^\top A$ and the left are eigenvectors of $AA^\top$; the singular values are the square roots of the (non-negative) eigenvalues of either. When $A$ is symmetric and positive semidefinite, SVD and eigendecomposition coincide.
• With PCA. Center the data matrix, take its SVD, and the right singular vectors are the principal directions; the squared singular values divided by the sample size are the variances along them.
• Condition number. $\sigma_1 / \sigma_r$ measures how unequally the map stretches directions. A large value means small input changes can become large output changes — the source of numerical instability in solving systems.

Check yourself

Eduspheria wiki · Mathematics for AI, Linear algebra

0 / 5 answered

1. 1The SVD factors a matrix into U, Σ and V. What are the diagonal entries of Σ called?

   Check Show answerShort answer
2. 2A matrix has singular values 5, 3 and 1. What is the squared Frobenius error of its best rank-1 approximation?

   Check Hint Show answerNumeric answer
3. 3Which property do singular values always have?

   A.They are always integersB.They sum to the rankC.They are non-negative and realD.They equal the eigenvalues

   Check Show answerMultiple choice
4. 4A 100×50 matrix is approximated by rank 3. How many numbers per term does one rank-1 layer store using its two vectors?

   Check Hint Show answerNumeric answer
5. 5Truncated SVD is optimal among all rank-k approximations in the Frobenius norm.

   TrueFalse

   Check Show answerTrue / false

Where next: probability — from transformations of space to the mathematics of uncertainty.