\documentclass{article}
\input{../macros.tex}

\begin{document}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section*{Lecture 15: Projections onto subspaces}

\subsection*{Projections}

If we have a vector $\vb b$ and a line determined by a vector $\vb a$, how do we find the point on the line that is closest  to $\vb b$?

\begin{figure}[h]
\centering
  \mypic{lec15fig1}\\
  \caption{The point closest to $\vb b$ on the line determined by $\vb a$.}\label{fig:l15_g1}
\end{figure}

We can see from Figure~\ref{fig:l15_g1} that this closest point $\vb
p$ is at the intersection formed by a line through $\vb b$ that
is orthogonal to $\vb a$.  If we think of $\vb p$ as an approximation
of $\vb b$, then the length of $\vb e = \vb b - \vb p$ is the error in
that approximation.

We could try to find $\vb p$ using trigonometry or calculus, but it's easier to
use linear algebra.  Since $\vb p$ lies on the line through $\vb a$,
we know $\vb p = x \vb a$ for some number $x$.  We also know that $\vb a$ is perpendicular to $\vb e = \vb b - \vb x \vb a$:
\begin{eqnarray*}
\vb a^T (\vb b - x \vb a) &=& 0\\
x \vb a^T \vb a &=& \vb a^T \vb b\\
x &=& \frac{\vb a^T \vb b}{\vb a^T\vb a},
\end{eqnarray*}
and $\displaystyle{\vb p = \vb a x = \vb a \frac{\vb a^T \vb b}{\vb a^T \vb a}}$.  Doubling $\vb b$ doubles $\vb p$.  Doubling $\vb a$ does not affect $\vb p$.

\subsection*{Projection matrix}

We'd like to write this projection in terms of a {\em projection
  matrix} $\mb P$: $\vb p = \mb P \vb b$.  $$\vb p = \vb x \vb a = \frac{\vb a \vb a^T \vb a}{\vb a^T \vb a},$$ 
so the matrix is: $$P = \frac{\vb a \vb a^T}{\vb a^T \vb a}.$$  Note that
$\vb a \vb a^T$ is a three by three matrix, not a number; matrix
multiplication is not commutative.

The column space of $\mb P$ is spanned by $\vb a$ because for any $\vb
b$, $\mb P \vb b$ lies on the line determined by $\vb a$.  The rank of $\mb P$ is $1$.
$\mb P$ is symmetric.  $\mb P^2 \vb b = \mb P \vb b$ because the
projection of a vector already on the line through $\vb a$ is just that
vector.  In general, projection matrices have the properties:
$$\mb P^T = \mb P \qquad \mbox{and} \qquad \mb P^2 = \mb P.$$

\subsection*{Why project?}

As we know, the equation $\mb A \vb x = \vb b$ may have no solution.
The vector $\mb A \vb x$ is always in the column space of $\mb A$, and
$\vb b$ is unlikely to be in the column space.  So, we project $\vb b$
onto a vector $\vb p$ in the column space of $\mb A$ and solve $\mb A
\vh x = \vb p$.

\subsection*{Projection in higher dimensions}

In $\R^3$, how do we project a vector $\vb b$ onto the closest point
$\vb p$ in a plane?

If $\vb a_1$ and $\vb a_2$ form a basis for the plane, then that plane
is the column space of the matrix $\mb A = \onebytwo {\vb a_1}{\vb
  a_2}$.


We know that $\vb p = \hat x_1 \vb a_1 + \hat x_2 \vb a_2 = \mb A \vh x$.  We
want to find $\vh x$.  There are many ways to show that $\vb e = \vb b
- \vb p = \vb b - \mb A \vh x$ is orthogonal to the plane we're
projecting onto, after which we can use the fact that $\vb e$ is perpendicular to $\vb a_1$
and $\vb a_2$:
$$\vb a_1^T (\vb b - \mb A \vh x) = 0 \quad \mbox{and} \quad \vb a_2^T (\vb b - \mb A \vh x) = 0.$$
In matrix form, $\mb A^T (\vb b - \mb A \vh x) = \vb 0$.
When we were projecting onto a line, $\mb A$ only had one column and so
this equation looked like: $\mb a^T (\vb b - x \vb a ) = \vb 0$.

Note that $\vb e = \vb b - \mb A \vh x$ is in the nullspace of $\mb
A^T$ and so is in the left nullspace of $\mb A$.  We know that
everything in the left nullspace of $\mb A$ is perpendicular to the
column space of $\mb A$, so this is another confirmation that our
calculations are correct.

We can rewrite the equation $\mb A^T (\vb b - \mb A \vh x) = \vb 0$ as:
$$\mb A^T \mb A \vh x = \mb A^T \vb b.$$ When projecting onto a line,
$\mb A^T \mb A$ was just a number; now it is a square matrix.  So instead of dividing by $\vb a^T \vb a$ we now have to multiply by $(\mb A^T \mb A)^{-1}$

In $n$ dimensions,
\begin{eqnarray*}
\vh x &=& (\mb A^T \mb A)^{-1} \mb A^T \vb b\\
\vb p = \mb A \vh x &=& \mb A (\mb A^T \mb A)^{-1} \mb A^T \vb b\\
\mb P &=& \mb A (\mb A^T \mb A)^{-1} \mb A^T.
\end{eqnarray*}

It's tempting to try to simplify these expressions, but if $\mb A$ isn't a square matrix we can't say that
$(\mb A^T \mb A)^{-1} = \mb A^{-1} (\mb A^T)^{-1}$.  If $\mb A$ does
happen to be a square, invertible matrix then its column space is the
whole space and contains $\vb b$.  In this case $\mb P$ is
the identity, as we find when we simplify.
It is still true that:
$$\mb P^T = \mb P \qquad \mbox{and} \qquad \mb P^2 = \mb P.$$

\subsection*{Least Squares}

\begin{figure}[h]
\centering
  \mypic{lec16fig1}\\
  \caption{Three points and a line close to them.}\label{fig:l15_g2}
\end{figure}

Suppose we're given a collection of data points $(t, b)$:
$$\{(1,1), (2,2), (3,2)\}$$ and we want to find the closest line $b = C +
Dt$ to that collection.
If the line went through all three points, we'd have:
\begin{eqnarray*}
C + D &=& 1\\
C + 2D &=& 2\\
C + 3D &=& 2,
\end{eqnarray*}
which is equivalent to:
$$
\begin{array}{cccc}
\left[ \begin{array}{rr} 1&1\\1&2\\1&3\end{array}\right] &
\twobyone CD &=& \threebyone 122\\
\mb A & \vb x && \vb b
\end{array}.
$$

In our example the line does not go through all three points, so this
equation is not solvable.  Instead we'll solve:
$$\mb A^T \mb A \vh x = \mb A^T \vb b.$$

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\end{document}