\documentclass{article}
\input{../macros.tex}

\begin{document}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section*{Lecture 17: Orthogonal matrices and Gram-Schmidt}

In this lecture we finish introducing orthogonality.  Using an
orthonormal basis or a matrix with orthonormal columns makes
calculations much easier. The Gram-Schmidt process starts with any basis and produces an orthonormal basis that spans the same
space as the original basis.

\subsection*{Orthonormal vectors}

The vectors $\vb q_1, \vb q_2, ... \vb q_n$ are {\em orthonormal} if:
$$\vb q_i^T \vb q_j = \left\{ \begin{array}{rl} 0 & \mbox{if } i \neq
  j\\ 1 & \mbox{if } i = j.\end{array}\right.$$ In other words, they
all have (normal) length $1$ and are perpendicular (ortho) to each
other.  Orthonormal vectors are always independent.

\subsection*{Orthonormal matrix}

If the columns of $\mb Q = \onebythree {\vb q_1} {...} {\vb q_n}$ are
  orthonormal, then $\mb Q^T \mb Q = \mb I$ is the identity.

Matrices with orthonormal columns are a new class of important
matrices to add to those on our list: triangular, diagonal,
permutation, symmetric, reduced row echelon, and projection matrices.
We'll call them ``orthonormal matrices''.

A square orthonormal matrix $\mb Q$ is called an {\em orthogonal
  matrix}.  If $\mb Q$ is square, then $\mb Q^T \mb Q = \mb I$ tells
us that $\mb Q^T = \mb Q^{-1}$.

\smallskip

For example, if $\mb Q = \small \threebythree 001100010$ then $\mb Q^T
= \small \threebythree 010001100$.  Both $\mb Q$ and $\mb Q^T$ are
orthogonal matrices, and their product is the identity.

\smallskip

The matrix $\mb Q = \small \twobytwo {\cos \theta} {-\sin\theta} {\sin
  \theta} {\cos \theta}$ is orthogonal.  The matrix $\small \twobytwo
1 1 1 {-1}$ is not, but we can adjust that matrix to get the
orthogonal matrix $\mb Q = \frac{1}{\sqrt 2} \small \twobytwo 1 1 1
{-1}$.  We can use the same tactic to find some larger orthogonal
matrices called {\em Hadamard matrices}:
$$\mb Q = \frac{1}{2}\left[ \begin{array}{rrrr} 1 & 1 & 1 & 1\\
1 & -1 & 1 & -1\\
1 & 1 & -1 & -1\\
1 & -1 & -1 & 1
\end{array}\right].$$

\smallskip

An example of a rectangular matrix with orthonormal columns is:
$$\mb Q = \frac{1}{3} \left[\begin{array}{rr}
1 & -2\\ 2 & -1 \\ 2 & 2\end{array}\right].$$
We can extend this to a (square) orthogonal matrix:
$$\frac{1}{3} \left[\begin{array}{rrr}
1 & -2 & 2 \\ 2 & -1 & -2 \\ 2 & 2 & 1\end{array}\right].$$

These examples are particularly nice because they don't include complicated
square roots.

\subsection*{Orthonormal columns are good}

Suppose $\mb Q$ has orthonormal columns.  The matrix that projects
onto the column space of $\mb Q$ is:
$$\mb P = \mb Q^T (\mb Q^T \mb Q)^{-1} \mb Q^T.$$ If the columns of
$\mb Q$ are orthonormal, then $\mb Q^T \mb Q = \mb I$ and $\mb P = \mb
Q \mb Q^T$.  If $\mb Q$ is square, then $\mb P = \mb I$ because the
columns of $\mb Q$ span the entire space.

Many equations become trivial when using a matrix with orthonormal
columns.  If our basis is orthonormal, the projection component $\hat x_i$ is just $\vb q_i^T \vb b$
because $\mb A^T \mb A \vh x = \mb A^T \vb b$ becomes $\vh x = \mb Q^T
\vb b$.

\subsection*{Gram-Schmidt}

With elimination, our goal was ``make the matrix triangular''.  Now
our goal is ``make the matrix orthonormal''.

We start with two independent vectors $\vb a$ and $\vb b$ and want to find
 orthonormal vectors $\vb q_1$ and $\vb q_2$ that span the same plane.
We start by finding orthogonal vectors $\vb A$ and $\vb B$
that span the same space as $\vb a$ and $\vb b$.  Then the unit
vectors $\vb q_1 = \frac{\vb A}{||\vb A||}$ and $\vb q_2 = \frac {\vb
  B}{||\vb B||}$ form the desired orthonormal basis.

\smallskip

Let $\vb A = \vb a$.  We get a vector orthogonal to $\vb A$ in the
space spanned by $\vb a$ and $\vb b$ by projecting $\vb b$ onto $\vb a$
and letting $\vb B = \vb b - \vb p$.  ($\vb B$ is what we previously
called $\vb e$.)
$$\vb B = \vb b - \frac{\vb A^T\vb b}{\vb A^T \vb A}\vb A.$$ If we
multiply both sides of this equation by $\vb A^T$, we see that $\vb A^T
\vb B = 0$.

What if we had started with three independent vectors, $\vb a$, $\vb b$
and $\vb c$?  Then we'd find a vector $\vb C$ orthogonal to both $\vb A$
and $\vb B$ by subtracting from $\vb c$ its components in the $\vb A$
and $\vb B$ directions:
$$\vb C = \vb c - \frac{\vb A^T\vb c}{\vb A^T \vb A}\vb A - \frac{\vb B^T\vb c}{\vb B^T \vb B}\vb B.$$

\smallskip

For example, suppose $\vb a = \small \threebyone 111$ and $\vb b = \small \threebyone 102$.  Then $\vb A = \vb a$ and: 
\begin{eqnarray*} \vb B 
&=& \threebyone 102 - \frac{\vb A^T \vb b}{\vb A^T \vb A} \threebyone 111\\
&=& \threebyone 102 - \frac{3}{3} \threebyone 111\\
&=& \threebyone 0{-1}1.
\end{eqnarray*}
Normalizing, we get: $$\mb Q = \onebytwo {\vb q_1} {\vb q_2} = \left[ \small 
\begin{array}{rr}
1/\sqrt{3} & 0\\
1/\sqrt{3} & -1/\sqrt{2}\\
1/\sqrt{3} & 1/\sqrt{2}
\end{array}\right].$$
The column space of $\mb Q$ is the plane spanned by $\vb
a$ and $\vb b$.

\bigskip

When we studied elimination, we wrote the process in terms of matrices and found
$\mb A = \mb L \mb U$.  A similar equation $\mb A = \mb
Q \mb R$ relates our starting matrix $\mb A$ to the result $\mb Q$ of
the Gram-Schmidt process.  Where $\mb L$ was lower triangular, $\mb R$ is upper triangular.

Suppose $\mb A = \onebytwo {\vb a_1} {\vb a_2}$.  Then:
$$\begin{array}{cccc} \mb A && \mb Q & \mb R\\ \onebytwo {\vb a_1}
  {\vb a_2} &=& \onebytwo {\vb q_1} {\vb q_2} & \twobytwo {\vb a_1^T
    \vb q_1} {\vb a_2^T \vb q_1} {\vb a_1^T \vb q_2} {\vb a_2^T
    \vb q_2}\end{array}.$$ If $\mb R$ is upper triangular, then it
should be true that $\vb a_1^T \vb q_2 = 0$. This must be true
because we chose $\vb q_1$ to be a unit vector in the direction of
$\vb a_1$.  All the later $\vb q_i$ were chosen to be
perpendicular to the earlier ones.  

Notice that $\mb R = \mb Q^T \mb A$.  This makes sense; $\mb Q^T \mb Q = \mb I$.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\end{document}