\documentclass{article}
\input{../macros.tex}

\begin{document}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section*{Lecture 14: Orthogonal vectors and subspaces}

In this lecture we learn what it means for vectors, bases and
subspaces to be {\em orthogonal}.  The symbol for this is $\perp$.

The ``big picture'' of this course is that the row space of a matrix' is
orthogonal to its nullspace, and its column space is orthogonal to
its left nullspace.

\begin{center}
\begin{tabular}{ccc}
row space & \qquad \qquad \qquad &column space\\
dimension $r$ & &dimension $r$ \\
$\perp$ && $\perp$\\
nullspace & &left nullspace $N(\mb A^T)$\\
dimension $n-r$ && dimension $m-r$
\end{tabular}
\end{center}

\subsection*{Orthogonal vectors}

{\em Orthogonal} is just another word for {\em perpendicular}.  Two
vectors are {\em orthogonal} if the angle between them is $90$
degrees.  If two vectors are orthogonal, they form a right triangle
whose hypotenuse is the sum of the vectors.  Thus, we can use the
Pythagorean theorem to prove that {\em the dot product} $\vb x^T \vb y = \vb
y^T \vb x$ {\em is zero} exactly when $\vb x$ and $\vb y$ are orthogonal.
(The length squared $||\vb x||^2$ equals $\vb x^T \vb x$.)

Note that all vectors are orthogonal to the zero vector.

\subsection*{Orthogonal subspaces}

Subspace $S$ is {\em orthogonal} to subspace $T$ means: every vector
in $S$ is orthogonal to every vector in $T$.  The blackboard is not
orthogonal to the floor;  two vectors in the line where
the blackboard meets the floor aren't orthogonal to each other.

In the plane, the space containing only the zero vector and any line
through the origin are orthogonal subspaces.  A line through the
origin and the whole plane are never orthogonal subspaces.  Two lines
through the origin are orthogonal subspaces if they meet at right
angles.

\subsection*{Nullspace is perpendicular to row space}

The row space of a matrix is orthogonal to the nullspace, because $\mb
A \vb x = \vb 0$ means the dot product of $\vb x$ with each row of
$\mb A$ is $0$.  But then the product of $\vb x$ with any combination
of rows of $\mb A$ must be $0$.

The column space is orthogonal to the left nullspace of $\mb A$
because the row space of $\mb A^T$ is perpendicular to the nullspace
of $\mb A^T$.

In some sense, the row space and the nullspace of a matrix subdivide
$\R^n$ into two perpendicular subspaces.  For  $\mb A =
\left[ \small \begin{array}{rrr} 1 & 2 & 5 \\ 2 & 4 & {10}\end{array} \right]$, the row space has dimension $1$ and
basis $\small \threebyone 1 2 5$ and the nullspace has dimension $2$
and is the plane through the origin perpendicular to the vector $\small \threebyone 1 2 5$.

Not only is the nullspace orthogonal to the row space, their
dimensions add up to the dimension of the whole space.  We say that
the nullspace and the row space are {\em orthogonal complements} in
$\R^n$.  The nullspace contains all the vectors that are perpendicular
to the row space, and vice versa.

\smallskip

We could say that this is part two of the fundamental theorem of
linear algebra.  Part one gives the dimensions of the four subspaces,
part two says those subspaces come in orthogonal pairs, and part three
will be about orthogonal bases for these subspaces.

\subsection*{$N(\mb A^T \mb A) = N(\mb A)$}

Due to measurement error, $\mb A \vb x = \vb b$ is often unsolvable if $m > n$.  Our next challenge is to find the best
possible solution in this case.  The matrix $\mb A^T \mb A$ plays a
key role in this effort: the central equation is $\mb A^T \mb A \vh x
= \mb A^T \vb b$.

We know that $\mb A^T \mb A$ is square ($n \times n$) and symmetric.
When is it invertible?

Suppose $ \mb A = \left[ \small \begin{array}{rr} 1 & 1 \\ 1 & 2 \\ 1 & 5 \end{array}\right]$.  Then: 
$$\mb A^T \mb A = \left[ \begin{array}{rrr} 1 & 1 & 1 \\ 1 & 2 &
    5 \end{array}\right] \left[ \begin{array}{rr} 1 & 1 \\ 1 & 2 \\ 1
    & 5 \end{array}\right] = \twobytwo 3 8 8 {30}$$ is
invertible. $\mb A^T \mb A$ is not always invertible.  In fact:
\begin{eqnarray*}
N(\mb A^T \mb A) &=& N(\mb A)\\
\mbox{rank of } \mb A^T \mb A &=& \mbox{rank of } \mb A.
\end{eqnarray*}
We conclude that $\mb A^T \mb A$ is invertible exactly when $\mb A$ has
independent columns.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\end{document}