Singular Values & the SVD

Study Sheet

Singular Values & the SVD

Every matrix is rank-one layers, sorted by importance

The Master Decomposition

Tip
Singular values: stretch factors for ANY matrix

Eigenvalues need square matrices and can be complex; SINGULAR VALUES exist for every matrix and are always real and nonnegative: σi=λi(ATA)\sigma_i = \sqrt{\lambda_i(A^{\mathsf T}A)} (ATAA^{\mathsf T}A is symmetric positive semidefinite, so this is legal).

Meaning: σ1\sigma_1 is the largest stretch AA applies to any unit vector; the number of nonzero σ\sigma's is the RANK; and ∑σi2\sum\sigma_i^2 equals the sum of squares of all entries. The unit circle maps to an ellipse with semi-axes σ1,σ2\sigma_1, \sigma_2 — every matrix, however ugly, is geometrically just rotate–stretch–rotate.

Tip
The SVD and best low-rank approximation

The singular value decomposition writes A=∑iσi uiviTA = \sum_i \sigma_i\,\mathbf u_i\mathbf v_i^{\mathsf T}: a sum of RANK-ONE layers, ordered by importance σ1≥σ2≥⋯\sigma_1 \ge \sigma_2 \ge \cdots. The Eckart–Young theorem: keeping just the first kk layers gives the BEST possible rank-kk approximation, with error exactly σk+1\sigma_{k+1}.

This is data compression as a theorem: an image is a matrix, most of its singular values are tiny, and storing a few layers reproduces it almost perfectly. Principal component analysis is the same truncation applied to centered data — "find the directions that matter, drop the rest."

Side note
Eigen vs. singular: when they agree

For symmetric positive-definite matrices, singular values ARE the eigenvalues; for the diagonal matrix diag(3,−4)\mathrm{diag}(3, -4) they are the absolute values 3,43, 4 — stretch has no sign. For a genuinely non-symmetric matrix the two sets differ, and the singular values are the honest geometry: a matrix with all eigenvalues zero (nilpotent) can still stretch vectors enormously, and its σ1\sigma_1 says by how much.

Try it
Try it: a rank-one matrix bare-handed

Find the singular values of A=(2412)A = \begin{pmatrix}2&4\\1&2\end{pmatrix}. Work: rows are proportional — rank 11, so σ2=0\sigma_2 = 0 and σ12=tr(ATA)=4+16+1+4=25\sigma_1^2 = \mathrm{tr}(A^{\mathsf T}A) = 4 + 16 + 1 + 4 = 25: σ1=5\sigma_1 = 5.

Structure check: A=(21)(1;;2)A = \binom21(1;;2), and ∣(21)∣⋅∣(1,2)∣=5⋅5=5\left|\binom21\right|\cdot|(1,2)| = \sqrt5\cdot\sqrt5 = 5 ✓ — a rank-one matrix's only singular value is the product of the lengths of its two factors.

Proofs & Why It Matters

Tip
Proof: σ₁ is the maximum stretch

Maximize ∣Ax∣2=xT(ATA)x|A\mathbf x|^2 = \mathbf x^{\mathsf T}(A^{\mathsf T}A)\mathbf x over unit vectors x\mathbf x. That is a quadratic form in the symmetric matrix ATAA^{\mathsf T}A, and the previous topic proved its maximum on the unit sphere is the largest eigenvalue λ1(ATA)\lambda_1(A^{\mathsf T}A).

Take square roots: max⁡∣Ax∣=λ1=σ1\max|A\mathbf x| = \sqrt{\lambda_1} = \sigma_1. ■\blacksquare SIGNIFICANCE: σ1\sigma_1 is the operator norm — it bounds how much a matrix can amplify errors, which is why σ1/σn\sigma_1/\sigma_n (the condition number) decides whether a linear system is numerically trustworthy. Every numerical analyst's first question about a matrix is about its singular values.

Side note
Where you have already met the SVD

Recommendation engines factor the user-movie ratings matrix into a few rank-one "taste" layers; latent semantic analysis does it to word-document counts; noise reduction truncates small singular values because noise spreads evenly across all of them while signal concentrates in the top few. The SVD is the closest thing linear algebra has to a universal tool.

Going Deeper: Worked Problems

Example
Worked: singular values of a non-square matrix

Find the singular values of A=(100111)A = \begin{pmatrix}1&0\\0&1\\1&1\end{pmatrix} (3×23\times2).

Step 1 — ATA=(2112)A^{\mathsf T}A = \begin{pmatrix}2&1\\1&2\end{pmatrix} (a 2×22\times2, so exactly two singular values).

Step 2 — its eigenvalues: trace 44, det 33: λ=3,1\lambda = 3, 1.

Step 3 — singular values σ1=3\sigma_1 = \sqrt3, σ2=1\sigma_2 = 1.

Step 4 — interpret: AA maps the plane into 3-space, stretching by 3\sqrt3 along (1,1)/2(1,1)/\sqrt2 and by 11 along (1,−1)/2(1,-1)/\sqrt2; the check σ12+σ22=4=\sigma_1^2 + \sigma_2^2 = 4 = sum of squared entries (1+0+0+1+1+11+0+0+1+1+1) ✓ confirms the arithmetic.

Example
Worked: how much a rank-one approximation loses

For A=(3045)A = \begin{pmatrix}3&0\\4&5\end{pmatrix}, find σ1,σ2\sigma_1, \sigma_2 and the error of the best rank-one approximation.

Step 1 — ATA=(25202025)A^{\mathsf T}A = \begin{pmatrix}25&20\\20&25\end{pmatrix}.

Step 2 — eigenvalues: trace 5050, det 225225: λ2−50λ+225=(λ−45)(λ−5)\lambda^2 - 50\lambda + 225 = (\lambda-45)(\lambda-5).

Step 3 — σ1=45=35≈6.71\sigma_1 = \sqrt{45} = 3\sqrt5 \approx 6.71, σ2=5≈2.24\sigma_2 = \sqrt5 \approx 2.24.

Step 4 — by Eckart–Young the best rank-one approximation errs by exactly σ2=5\sigma_2 = \sqrt5 in the spectral norm; the fraction of "energy" captured is σ12σ12+σ22=4550=90%\dfrac{\sigma_1^2}{\sigma_1^2 + \sigma_2^2} = \dfrac{45}{50} = 90\%. Sanity: σ1σ2=15=∣det⁡A∣\sigma_1\sigma_2 = 15 = |\det A| ✓ — the product of singular values is always the absolute determinant.