Singular Values & the SVD

Meaning: $\sigma_1$ is the largest stretch $A$ applies to any unit vector; the number of nonzero $\sigma$'s is the rank; and $\sum\sigma_i^2$ equals the sum of squares of all entries. The unit circle maps to an ellipse with semi-axes $\sigma_1, \sigma_2$ — every matrix, however ugly, is geometrically just rotate–stretch–rotate. Example: $A = \begin{pmatrix}2&2\\-1&1\end{pmatrix}$ has $A^{\mathsf T}A = \begin{pmatrix}5&3\\3&5\end{pmatrix}$ with eigenvalues $8$ and $2$, so $\sigma_1 = 2\sqrt2$ and $\sigma_2 = \sqrt2$. The unit vector $\mathbf v_1 = \tfrac{1}{\sqrt2}(1, 1)$ goes to $(2\sqrt2, 0)$ and $\mathbf v_2 = \tfrac{1}{\sqrt2}(1, -1)$ goes to $(0, -\sqrt2)$: perpendicular in, perpendicular out, stretched by $\sigma_1$ and $\sigma_2$.

This is data compression as a theorem: an image is a matrix, most of its singular values are tiny, and storing a few layers reproduces it almost perfectly. Principal component analysis is the same truncation applied to centered data — "find the directions that matter, drop the rest."

The key fact is that the images $A\mathbf v_i$ are automatically perpendicular: $(A\mathbf v_i)\cdot(A\mathbf v_j) = \mathbf v_i^{\mathsf T}(A^{\mathsf T}A\mathbf v_j) = \lambda_j\,\mathbf v_i\cdot\mathbf v_j$, which is $0$ for $i \ne j$ and $\lambda_i = \sigma_i^2$ for $i = j$. So $A$ takes one perpendicular set of directions to another, stretching by $\sigma_i$; $AV = U\Sigma$ says exactly that, and multiplying by $V^{\mathsf T}$ (which is $V^{-1}$) gives $A = U\Sigma V^{\mathsf T}$.

Output directions: $A\mathbf v_1 = \tfrac{1}{\sqrt2}(4, 0)$, so $\mathbf u_1 = \tfrac{A\mathbf v_1}{2\sqrt2} = (1, 0)$; $A\mathbf v_2 = \tfrac{1}{\sqrt2}(0, -2)$, so $\mathbf u_2 = \tfrac{A\mathbf v_2}{\sqrt2} = (0, -1)$. Assemble:

Check: $\Sigma V^{\mathsf T} = \begin{pmatrix}2&2\\1&-1\end{pmatrix}$, and $U$ negates the second row, giving $\begin{pmatrix}2&2\\-1&1\end{pmatrix}$ ✓. Also $\sigma_1\sigma_2 = 4 = |\det A|$ ✓.

A rank-one product $\mathbf a\mathbf b^{\mathsf T}$ has the single singular value $|\mathbf a|\,|\mathbf b|$, here $3\cdot\sqrt2 = 3\sqrt2$, with $\mathbf u_1 = \mathbf a/|\mathbf a| = \tfrac13(1, 2, 2)$ and $\mathbf v_1 = \mathbf b/|\mathbf b| = \tfrac{1}{\sqrt2}(1, 1)$:

Check against $A^{\mathsf T}A = \begin{pmatrix}9&9\\9&9\end{pmatrix}$, whose eigenvalues are $18$ and $0$: $\sigma_1 = \sqrt{18} = 3\sqrt2$ ✓ and $\sigma_2 = 0$. The sum of squared entries is $1 + 1 + 4 + 4 + 4 + 4 = 18 = \sigma_1^2$ ✓.

Notice $3 \pm 2\sqrt2 = (\sqrt2 \pm 1)^2$, so $\sigma_1 = \sqrt2 + 1 \approx 2.41$ and $\sigma_2 = \sqrt2 - 1 \approx 0.41$. Their product is $(\sqrt2)^2 - 1 = 1 = |\det A|$ ✓: the shear preserves area but squeezes the unit circle into a long thin ellipse. The eigenvalues $1, 1$ only report that no direction is stretched without also being turned; the singular values report how much stretching actually happens.

The layers are orthogonal to each other (their $\mathbf u$'s and $\mathbf v$'s are), so squared sizes add like Pythagoras: the squared entries of $A$ total $\sum\sigma_i^2$, and each discarded layer removes exactly its $\sigma_i^2$. The largest stretch of $A - A_k$ is the largest remaining singular value, $\sigma_{k+1}$. Each layer is one column $\mathbf u_i$ ($m$ numbers), one row $\mathbf v_i$ ($n$ numbers) and one weight — hence the storage count.

So $A_1 = \sigma_1\mathbf u_1\mathbf v_1^{\mathsf T} = 3\sqrt5\cdot\tfrac{1}{\sqrt{10}}\cdot\tfrac{1}{\sqrt2}\begin{pmatrix}1\\3\end{pmatrix}\begin{pmatrix}1&1\end{pmatrix} = \tfrac32\begin{pmatrix}1&1\\3&3\end{pmatrix}$. The error is $A - A_1 = \begin{pmatrix}\tfrac32&-\tfrac32\\-\tfrac12&\tfrac12\end{pmatrix} = \begin{pmatrix}\tfrac32\\-\tfrac12\end{pmatrix}\begin{pmatrix}1&-1\end{pmatrix}$, itself rank one, with singular value $\sqrt{\tfrac94 + \tfrac14}\cdot\sqrt2 = \sqrt5 = \sigma_2$ ✓ — Eckart–Young, visible in the numbers. The energies: $A_1$ has squared entries $\tfrac94(1 + 1 + 9 + 9) = 45$ out of $A$'s $9 + 16 + 25 = 50$.

That is $10(100 + 100 + 1) = 2010$ numbers instead of $100\cdot100 = 10{,}000$ — about $20\%$ of the storage, for an image that keeps $99\%$ of its energy (the relative Frobenius error is $\sqrt{0.01} = 10\%$). The saving grows with size: for a $1000\times1000$ image at rank $50$ the count is $50\cdot2001 = 100{,}050$ numbers, about $10\%$ of a million.

Take square roots: $\max|A\mathbf x| = \sqrt{\lambda_1} = \sigma_1$. $\blacksquare$ Significance: $\sigma_1$ is the operator norm — it bounds how much a matrix can amplify errors, which is why $\sigma_1/\sigma_n$ (the condition number) decides whether a linear system is numerically trustworthy. Every numerical analyst's first question about a matrix is about its singular values.

Step 1 — $A^{\mathsf T}A = \begin{pmatrix}2&1\\1&2\end{pmatrix}$ (a $2\times2$, so exactly two singular values).

Step 2 — its eigenvalues: trace $4$, det $3$: $\lambda = 3, 1$.

Step 3 — singular values $\sigma_1 = \sqrt3$, $\sigma_2 = 1$.

Step 4 — interpret: $A$ maps the plane into 3-space, stretching by $\sqrt3$ along $(1,1)/\sqrt2$ and by $1$ along $(1,-1)/\sqrt2$; the check $\sigma_1^2 + \sigma_2^2 = 4 =$ sum of squared entries ($1+0+0+1+1+1$) ✓ confirms the arithmetic.

Step 1 — $A^{\mathsf T}A = \begin{pmatrix}25&20\\20&25\end{pmatrix}$.

Step 2 — eigenvalues: trace $50$, det $225$: $\lambda^2 - 50\lambda + 225 = (\lambda-45)(\lambda-5)$.

Step 3 — $\sigma_1 = \sqrt{45} = 3\sqrt5 \approx 6.71$, $\sigma_2 = \sqrt5 \approx 2.24$.

Step 4 — by Eckart–Young the best rank-one approximation errs by exactly $\sigma_2 = \sqrt5$ in the spectral norm; the fraction of "energy" captured is $\dfrac{\sigma_1^2}{\sigma_1^2 + \sigma_2^2} = \dfrac{45}{50} = 90\%$. Sanity: $\sigma_1\sigma_2 = 15 = |\det A|$ ✓ — the product of singular values is always the absolute determinant.