\documentclass{article}
\input{../macros.tex}

\begin{document}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section*{Lecture 16: Projection matrices and least squares}

\subsection*{Projections}

Last lecture, we learned that $\mb P = \mb A (\mb A^T
\mb A)^{-1} \mb A^T$ is the matrix that projects a vector $\vb b$ onto
the space spanned by the columns of $\mb A$.  If $\vb b$ is
perpendicular to the column space, then it's in the left nullspace $N(\mb A^T)$ of
$\mb A$ and $\mb P \vb b = \vb 0$. If $\vb b$ is in the
column space then $\vb b = \mb A \vb x$ for some $\vb x$, and $\mb P
\vb b = \vb b$.

A typical vector will have a component $\vb p$ in the column space and
a component $\vb e$ perpendicular to the column space (in the left
nullspace); its projection is just the component in the column space.

The matrix projecting $\vb b$ onto $N(\mb A^T)$ is $I-P$:
\begin{eqnarray*}
\vb e &=& \vb b - \vb p\\
\vb e &=& (\mb I - \mb P)\vb b.
\end{eqnarray*}
Naturally, $\mb I - \mb P$ has all the properties of a projection matrix.

\subsection*{Least squares}

\begin{figure}[h]
\centering
  \mypic{lec16fig1}\\
  \caption{Three points and a line close to them.}\label{fig:l16_g1}
\end{figure}

We want to find the closest line $b = C + Dt$ to the points $(1,1)$, $(2,2)$, and $(3,2)$.  The process we're going to use is
called {\em linear regression}; this technique is most useful if none
of the data points are {\em outliers}.

By ``closest'' line we mean one that minimizes the error represented by
the distance from the points to the line.  We measure that error by
adding up the squares of these distances.  In other words, we want to
minimize $|| \mb A \vb x - \vb b ||^2 = ||\vb e||^2$.  

If the line went through all three points, we'd have:
\begin{eqnarray*}
C + D &=& 1\\
C + 2D &=& 2\\
C + 3D &=& 2,
\end{eqnarray*}
but this system is unsolvable.  It's equivalent to $\mb A \vb x = \vb
b$, where:
$$
\mb A = \left[ \begin{array}{rr} 1&1\\1&2\\1&3\end{array}\right], \vb x =
\twobyone CD \mbox{ and } \vb b= \threebyone 122.
$$

There are two ways of viewing this.  In the space of the line we're
trying to find, $e_1$, $e_2$ and $e_3$ are the vertical distances from
the data points to the line.  The components $p_1$, $p_2$ and $p_3$ are the values of $C + Dt$ near each data point;
$\vb p \approx \vb b$.

In the other view we have a vector $\vb b$ in $\R^3$, its projection
$\vb p$ onto the column space of $\mb A$, and its projection $\vb e$
onto $N(\mb A^T)$.

\smallskip

We will now find $\vh x = \twobyone {\hat C} {\hat D}$ and $\vb p$.
We know:
\begin{eqnarray*}
\mb A^T \mb A \vh x &=& \mb A^T \vb b\\
\twobytwo 3 6 6 {14} \twobyone {\hat C} {\hat D} &=& \twobyone 5 {11}.
\end{eqnarray*}
From this we get the {\em normal equations}:
\begin{eqnarray*}
3 \hat C + 6 \hat D &=& 5\\
6 \hat C + 14 \hat D &=& 11.
\end{eqnarray*}
We solve these to find $\hat D = 1/2$ and $\hat C = 2/3$.

We could also have used calculus to find the minimum of the following function of two variables:
$$e_1^2 +e_2^2 + e_3^2 = (C+D-1)^2 + (C+2D-2)^2 + (C+3D-2)^2.$$

Either way, we end up solving a system of linear equations to find
that the closest line to our points is $b = \frac{2}{3} + \frac{1}{2}
t$.

This gives us:
\begin{center}
\begin{tabular}{|r|r|r|}
\hline
$i$ & $p_i$ & $e_i$\\
\hline
$1$ & $7/6$ & $-1/6$\\
$2$ & $5/3$ & $1/3$\\
$3$ & $13/6$ & $-1/6$\\
\hline
\end{tabular}
\end{center}
or $\vb p = \small \threebyone {7/6}{5/3}{13/6}$ and $\vb e = \small
\threebyone{-1/6}{2/6}{-1/6}$.  Note that $\vb p$ and $\vb e$ are
orthogonal, and also that $\vb e$ is perpendicular to the columns of $\mb
A$.

\subsection*{The matrix $\mb A^T \mb A$}

We've been assuming that the matrix $\mb A^T\mb A$ is invertible.  Is
this justified?  

\smallskip

If $\mb A$ has independent columns, then $\mb A^T \mb A$ is
invertible.

\smallskip

To prove this we assume that $\mb A^T\mb A \vb x = \vb 0$, then show that
it must be true that $\vb x = \vb 0$:
\begin{eqnarray*}
\mb A^T \mb A \vb x &=& \vb 0\\
\vb x^T \mb A^T \mb A \vb x &=& \vb x^T \vb 0\\
(\mb A \vb x)^T(\mb A \vb x) &=& \vb 0\\
\mb A \vb x &=& \vb 0.
\end{eqnarray*}
Since $\mb A$ has independent columns, $\mb A \vb x = \vb 0$ only when
$\vb x = \vb 0$.

\smallskip

As long as the columns of $\mb A$ are independent, we can use linear
regression to find approximate solutions to unsolvable systems of
linear equations.  The columns of $\mb A$ are guaranteed to be
independent if they are {\em orthonormal}, i.e. if they are
perpendicular unit vectors like $\threebyone 100$, $\threebyone 010$
and $\threebyone 001$, or like $\twobyone {\cos \theta}{\sin \theta}$
and $\twobyone {-\sin \theta}{\cos \theta}$.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\end{document}