\documentclass{article}
\input{../macros.tex}

\begin{document}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section*{Lecture 29:  Singular value decomposition}

The {\em singular value decomposition} of a matrix is usually referred
to as the {\em SVD}.  This is the final and best factorization of a
matrix:
$$\mb A = \mb U \mb \Sigma \mb V^T$$ where $\mb U$ is orthogonal, $\mb
\Sigma$ is diagonal, and $\mb V$ is orthogonal.

In the decomoposition
$\mb A = \mb U \mb \Sigma \mb V^T$, $\mb A$ can be {\em any} matrix.
We know that if $\mb A$ is symmetric positive definite its
eigenvectors are orthogonal and we can write $\mb A = \mb Q \Lambda
\mb Q^T$.  This is a special case of a SVD, with $\mb U = \mb V = \mb
Q$.  For more general $\mb A$, the SVD requires two different matrices
$\mb U$ and $\mb V$.

We've also learned how to write $\mb A = \mb S \mb \Lambda \mb S^{-1}$, where
$\mb S$ is the matrix of $n$ distinct eigenvectors of $\mb A$.  However, $\mb S$
may not be orthogonal; the matrices $\mb U$ and $\mb V$ in the SVD
will be.

\subsection*{How it works}

We can think of $\mb A$ as a linear transformation taking a vector
$\vb v_1$ in its row space to a vector $\vb u_1 = \mb A \vb v_1$ in
its column space.  The SVD arises from finding an orthogonal basis for
the row space that gets transformed into an orthogonal basis for the
column space:  $\mb A \vb v_i = \sigma_i \vb u_i$.

It's not hard to find an orthogonal basis for the row space -- the
Gram-Schmidt process gives us one right away.  But in general, there's
no reason to expect $\mb A$ to transform that basis to another
orthogonal basis.

\smallskip

You may be wondering about the vectors in the nullspaces of $\mb A$
and $\mb A^T$.  These are no problem -- zeros on the diagonal of $\mb
\Sigma$ will take care of them.

\subsection*{Matrix language}

The heart of the problem is to find an orthonormal basis $\vb v_1, \vb
v_2, ... \vb v_r$ for the row space of $\mb A$ for which
\begin{align*}
\mb A \onebyfour {\vb v_1} {\vb v_2} \cdots {\vb v_r} 
&= \onebyfour {\sigma_1 \vb u_1} {\sigma_2 \vb u_2} \cdots {\sigma_r \vb u_r}\\
&= \onebyfour {\vb u_1} {\vb u_2} \cdots {\vb u_r}
\left[ \begin{array}{cccc}
\sigma_1 &&&\\
&\sigma_2&&\\
&&\ddots&\\
&&&\sigma_r
\end{array}\right],
\end{align*} with $\vb u_1, \vb u_2, ... \vb u_r$ an orthonormal basis for the
column space of $\mb A$.  Once we add in the nullspaces, this equation
will become $\mb A \mb V = \mb U \mb \Sigma$.  (We can complete the
orthonormal bases $\vb v_1, ... \vb v_r$ and $\vb u_1, ... \vb u_r$ to
orthonormal bases for the entire space any way we want.  Since
$\vb v_{r+1}, ... \vb v_{n}$ will be in the nullspace of $\mb A$, the diagonal
entries $\sigma_{r+1}, ... \sigma_n$ will be $0$.)

\smallskip

The columns of $\mb U$ and $\mb V$ are bases for the row and
column spaces, respectively.  Usually $\mb U \neq \mb V$, but if $\mb A$ is positive
definite we can use the {\em same} basis for its row and column space!

\subsection*{Calculation}

Suppose $\mb A$ is the invertible matrix $\small \twobytwo 44{-3}3$.  We
want to find vectors $\vb v_1$ and $\vb v_2$ in the row space $\R^2$,
$\vb u_1$ and $\vb u_2$ in the column space $\R^2$, and positive
numbers $\sigma_1$ and $\sigma_2$ so that the $\vb v_i$ are
orthonormal, the $\vb u_i$ are orthonormal, and the $\sigma_i$ are the
scaling factors for which $\mb A \vb v_i = \sigma_i u_i$.

This is a big step toward finding orthonormal matrices $\mb V$ and $\mb U$ and a diagonal matrix $\mb \Sigma$ for which:
$$\mb A \mb V = \mb U \mb \Sigma.$$ Since $\mb V$ is orthogonal, we
can multiply both sides by $\mb V^{-1} = \mb V^T$ to get:
$$\mb A = \mb U \mb \Sigma \mb V^T.$$ Rather than solving for $\mb U$,
$\mb V$ and $\mb \Sigma$ simultaneously, we multiply both sides by
$\mb A^T = \mb V \mb \Sigma^T \mb U^T$ to get:
\begin{align*}
\mb A^T \mb A &= \mb V \mb \Sigma \mb U^{-1} \mb U \mb \Sigma \mb V^T\\
&= \mb V \mb \Sigma^2 \mb V^T\\
&= \mb V \left[ \begin{array}{cccc}
\sigma_1^2 &&&\\
& \sigma_2^2 &&\\
&& \ddots &\\
&&& \sigma_n^2
\end{array}\right] \mb V^T.
\end{align*}
This is in the form $\mb Q \mb \Lambda \mb Q^T$; we can now find $\mb
V$ by diagonalizing the symmetric positive definite (or semidefinite) matrix $\mb A^T
\mb A$.  The columns of $\mb V$ are eigenvectors of $\mb A^T \mb A$
and the eigenvalues of $\mb A^T\mb A$ are the values $\mb \sigma_i^2$.
(We choose $\mb \sigma_i$ to be the positive square root of $\mb
\lambda_i$.)

To find $\mb U$, we do the same thing with $\mb A \mb A^T$.

\subsection*{SVD example}

We return to our matrix $\mb A = \small \twobytwo 44{-3}3$.  We start
by computing 
\begin{align*}
\mb A^T \mb A &= \twobytwo 4{-3}4{3} \twobytwo 44{-3}3\\
&= \twobytwo {25}{7}{7}{25}.
\end{align*}
The eigenvectors of this matrix will give us the vectors $\vb v_i$, and the
eigenvalues will gives us the numbers $\sigma_i$.

Two orthogonal eigenvectors of $\mb A^T\mb A$ are $\small \twobyone
11$ and $\small \twobyone 1{-1}$.  To get an orthonormal basis, let
$\vb v_1 = \small \twobyone {1/\sqrt{2}} {1/\sqrt{2}}$ and $\vb v_2 =
\twobyone {1/\sqrt{2}} {-1/\sqrt{2}}$.  These have eigenvalues $\sigma_1^2 = 32$
and $\sigma_2^2 = 18$.  We now have:
$$
\begin{array}{cccccc}
\mb A && \mb U & \mb \Sigma & \mb V^T\\
\twobytwo 44{-3}3 &=& \twobytwo{}{}{}{} & \twobytwo {4\sqrt{2}} 0 0 {3 \sqrt{2}} &
\twobytwo {1/\sqrt{2}} {1/\sqrt{2}} {1/\sqrt{2}} {-1/\sqrt{2}}.
\end{array}
$$

We could solve this for $\mb U$, but for practice we'll find $\mb U$
by finding orthonormal eigenvectors $\vb u_1$ and $\vb u_2$ for $\mb A
\mb A^T = \mb U \mb \Sigma^2 \mb U^T$.
\begin{align*}
\mb A \mb A^T &= \twobytwo 44{-3}3 \twobytwo 4{-3}43\\
&= \twobytwo {32}00{18}.
\end{align*}
Luckily, $\mb A \mb A^T$ happens to be diagonal.  It's tempting to let $\vb u_1 = \small \twobyone 10$ and $\vb u_2 = \small
\twobyone 01$, as Professor Strang did in the lecture, but because $\mb A \vb v_2 = \small \twobyone 0{-3\sqrt 2}$ we instead have $\vb u_2 = \small \twobyone 0{-1}$ and $\mb U = \small \twobytwo 100{-1}$.  Note that this also gives us a chance to double check our calculation of $\sigma_1$ and $\sigma_2$.

Thus, the SVD of $\mb A$ is:
$$
\begin{array}{cccccc}
\mb A && \mb U & \mb \Sigma & \mb V^T\\
\twobytwo 44{-3}3 &=& \twobytwo 100{-1} & \twobytwo {4\sqrt{2}} 0 0 {3 \sqrt{2}} &
\twobytwo {1/\sqrt{2}} {1/\sqrt{2}} {1/\sqrt{2}} {-1/\sqrt{2}}.
\end{array}
$$

\subsection*{Example with a nullspace}

Now let $\mb A = \small \twobytwo 4386$.  This has a one dimensional
nullspace and one dimensional row and column spaces.

The row space of $\mb A$ consists of the multiples of $\vb \small \twobyone
43$.  The column space of $\mb A$ is made up of multiples of $\vb \small
\twobyone 48$.  The nullspace and left nullspace are perpendicular to
the row and column spaces, respectively.

Unit basis vectors of the row and column spaces are $\vb v_1 = \small
\twobyone {.8}{.6}$ and $\vb u_1 = \small \twobyone {1/\sqrt{5}}
          {2/\sqrt{5}}$.  
To compute $\sigma_1$ we find the nonzero eigenvalue of $\mb A^T\mb A$.
\begin{align*}
\mb A^T\mb A &= \twobytwo 4836 \twobytwo 4386\\
&= \twobytwo {80}{60}{60}{45}.
\end{align*}
Because this is a rank $1$ matrix, one eigenvalue must be $0$.  The
other must equal the trace, so $\sigma_1^2 = 125$.  After finding unit
vectors perpendicular to $\vb u_1$ and $\vb v_1$ (basis vectors for
the left nullspace and nullspace, respectively) we see that the SVD of
$\mb A$ is:
$$
\begin{array}{ccccc}
\twobytwo 4386 &=& \frac{1}{\sqrt{5}} \twobytwo 122{-1} &
\twobytwo {\sqrt{125}}000 & \twobytwo {.8}{.6}{.6}{-.8}.\\
\mb A && \mb U & \mb \Sigma & \mb V^T
\end{array}
$$

\bigskip

The singular value decomposition combines topics in linear algebra ranging from positive definite matrices to the four fundamental subspaces.
\smallskip\\
\begin{tabular}{ll}
$\vb v_1, \vb v_2, ... \vb v_r$ & is an orthonormal basis for the row space.\\
$\vb u_1, \vb u_2, ... \vb u_r$ & is an orthonormal basis for the column space.\\
$\vb v_{r+1}, ... \vb v_n$ & is an orthonormal basis for the nullspace.\\
$\vb u_{r+1}, ... \vb u_m$ & is an orthonormal basis for the left nullspace.
\end{tabular}\\
\smallskip\\
These are the ``right'' bases to use, because $\mb A \vb v_i =
\sigma_i \vb u_i$.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\end{document}