\documentclass{article}
\input{../macros.tex}

\begin{document}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section*{Lecture 28: Similar matrices and Jordan form}

We've nearly covered the entire heart of linear algebra -- once we've
finished singular value decompositions we'll have seen all the
most central topics.

\subsection*{$\mb A^T\mb A$ is positive definite}

A matrix is {\em positive definite} if $\vb x^T \mb A \vb x > 0$ for all $\vb x \neq \vb 0$.  This is a very important class of matrices; positive definite matrices appear in the form of $\mb A^T\mb A$ when computing least squares
solutions.  In many situations, a rectangular matrix is
multiplied by its transpose to get a square matrix.

\smallskip

Given a symmetric positive definite matrix $\mb A$, is its inverse also
symmetric and positive definite?  Yes, because if the (positive) eigenvalues of $\mb A$ are $\lambda_1, \lambda_2, \cdots \lambda_d$ then the
eigenvalues $1/\lambda_1, 1/\lambda_2, \cdots 1/\lambda_d$ of $\mb A^{-1}$ are also positive.

\smallskip

If $\mb A$ and $\mb B$ are positive definite, is $\mb A + \mb B$
positive definite?  We don't know much about the eigenvalues of $\mb A
+ \mb B$, but we can use the property $\vb x^T \mb A \vb x > 0$ and $\vb x^T \mb B \vb x > 0$ to
show that  $\vb x^T
(\mb A + \mb B) \vb x > 0$ for $\vb x \neq 0$ and so $\mb A + \mb B$
is also positive definite.

\smallskip

Now suppose $\mb A$ is a rectangular ($m$ by $n$) matrix. $\mb A$ is
almost certainly not symmetric, but $\mb A^T \mb A$ is square and
symmetric.  Is $\mb A^T \mb A$ positive definite?  We'd rather not try
to find the eigenvalues or the pivots of this matrix, so we ask when
$\vb x^T \mb A^T \mb A \vb x$ is positive.

Simplifying $\vb x^T \mb A^T \mb A \vb x$ is just a matter of moving
parentheses:
$$\vb x^T (\mb A^T \mb A) \vb x = (\mb A \vb x)^T (\mb A \vb x) = |\mb
A \vb x|^2 \geq 0.$$ The only remaining question is whether $\mb A \vb x = \vb 0$.  If $\mb A$ has rank $n$ (independent
columns), then $\vb x^T (\mb A^T \mb A) \vb x = \vb 0$ only if $\vb x
= \vb 0$ and $\mb A$ is positive definite.

\smallskip

Another nice feature of positive definite matrices is that you never
have to do row exchanges when row reducing -- there are never $0$'s or
unsuitably small numbers in their pivot positions.

\subsection*{Similar matrices $\mb A$ and $\mb B = \mb M^{-1}\mb A \mb M$}

Two square matrices $\mb A$ and $\mb B$ are {\em similar} if $\mb B = \mb M^{-1} \mb A \mb M$ for some
matrix $\mb M$.  This allows us to
put matrices into families in which all the matrices in a family are
similar to each other.  Then each family can be represented by a
diagonal (or nearly diagonal) matrix.

\subsubsection*{Distinct eigenvalues}

If $\mb A$ has a full set of eigenvectors we can create its
eigenvector matrix $\mb S$ and write $\mb S^{-1} \mb A \mb S = \mb
\Lambda$.  So $\mb A$ is similar to $\mb \Lambda$ (choosing $\mb M$ to be this $\mb S$).

If $\mb A = \small \twobytwo 2112$ then $\mb \Lambda = \small
\twobytwo 3001$ and so $\mb A$ is similar to $\small \twobytwo 3001$.  But $\mb A$ is also similar to:
$$\begin{array}{ccccc}
\mb M^{-1} & \mb A &\mb M && \mb B\\
\twobytwo 1{-4}01 & \twobytwo 3001 & \twobytwo {1}{4}{0}{1} 
&=& \twobytwo 1{-4}{0}{1} \twobytwo 2916\\
&&&=& \twobytwo {-2}{-15}16.
\end{array}$$
In addition, $\mb B$ is similar to $\mb \Lambda$.  All these similar
matrices have the same eigenvalues, $3$ and $1$; we can check this
by computing the trace and determinant of $\mb A$ and $\mb B$.

\smallskip

Similar matrices have the same eigenvalues!

\smallskip

In fact, the matrices similar to $\mb A$ are all
the $2$ by $2$ matrices with eigenvalues $3$ and $1$.  Some other members of this family are $\small \twobytwo 3701$ and $\small \twobytwo 1703$.
To prove that similar matrices have the same eigenvalues, suppose $\mb A \vb x = \lambda \vb x$.
We modify this equation to include $\mb B = \mb M^{-1}\mb A \mb M$:
\begin{eqnarray*}
\mb A \mb M \mb M^{-1}\vb x &=& \lambda  \vb x\\
\mb M^{-1}\mb A \mb M \mb M^{-1}\vb x &=& \lambda \mb M^{-1} \vb x\\
\mb B \mb M^{-1}\vb x &=& \lambda \mb M^{-1} \vb x.
\end{eqnarray*}
The matrix $\mb B$ has the same $\lambda$ as an eigenvalue. $\mb M^{-1} \vb
x$ is the eigenvector. 

If two matrices are similar, they have the same eigenvalues and the
same number of independent eigenvectors (but probably not the same eigenvectors).

\smallskip

When we diagonalize $\mb A$, we're finding a diagonal matrix $\mb \Lambda$ that is
similar to $\mb A$.  If two matrices have the same $n$ distinct eigenvalues, they'll be similar to the same diagonal matrix.

\subsubsection*{Repeated eigenvalues}

If two eigenvalues of $\mb A$ are the same, it may not
be possible to diagonalize $\mb A$.  Suppose $\mb \lambda_1 =
\lambda_2 = 4$.  One family of matrices with eigenvalues $4$ and $4$
contains only the matrix $\small \twobytwo 4004$.  The matrix $\small
\twobytwo 4104$ is not in this family.

There are two families of similar matrices with eigenvalues $4$ and
$4$.  The larger family includes $\small \twobytwo 4104$.  Each of the
members of this family has only one eigenvector.

The matrix $\small \twobytwo 4004$ is the only member of the other
family, because:
$$\mb M^{-1} \twobytwo 4004 \mb M = 4 \mb M^{-1}M = \twobytwo 4004$$
for any invertible matrix $\mb M$.


\subsection*{Jordan form}

Camille Jordan found a way to choose a ``most diagonal''
representative from each family of similar matrices; this
representative is said to be in {\em Jordan normal form}.  For
example, both $\small \twobytwo 4104$ and $\small \twobytwo 4004$ are
in Jordan form.  This form used to be the climax of linear algebra, but 
not any more.  Numerical applications rarely need it.

We can find more members of the family represented by $\small
\twobytwo 4104$ by choosing diagonal entries to get a trace of $4$,
then choosing off-diagonal entries to get a determinant of $16$:
$$\twobytwo 4104, \twobytwo 51{-1}3, \twobytwo 40{17}4, \twobytwoc a b
{(8a-a^2-16)/b} {8-a}.$$ (None of these are diagonalizable, because if
they were they would be similar to $\small \twobytwo 4004$. That matrix is only similar to itself.)
What about this one?
$$\mb A = 
\left[ \begin{array}{cccc}
0 & 1 & 0 & 0\\
0 & 0 & 1 & 0\\
0 & 0 & 0 & 0\\
0 & 0 & 0 & 0
\end{array}\right]$$
Its eigenvalues are four zeros.  Its rank is $2$ so the dimension of its nullspace is $4-2 = 2$.  It will have two independent eigenvectors and two ``missing'' eigenvectors.  When we look instead at
$$\left[ \begin{array}{cccc}
0 & 1 & 7 & 0\\
0 & 0 & 1 & 0\\
0 & 0 & 0 & 0\\
0 & 0 & 0 & 0
\end{array}\right],$$
its rank and the dimension of its nullspace are still $2$, but it's
not as nice as $\mb A$.  $\mb B$ {\em is} similar to $\mb A$, which is the Jordan normal form representative of
this family.  $\mb A$ has a $1$ above the diagonal for every missing
eigenvector and the rest of its entries are $0$.

Now consider:
$$\mb C = \left[ \begin{array}{cccc}
0 & 1 & 0 & 0\\
0 & 0 & 0 & 0\\
0 & 0 & 0 & 1\\
0 & 0 & 0 & 0
\end{array}\right].$$
Again it has rank $2$ and its nullspace has dimension $2$.  Its four
eigenvalues are $0$.  Surprisingly, it is not similar to $\mb A$.  We can see this by breaking the matrices into their {\em Jordan blocks}:
$$\mb A = 
\left[ \begin{array}{ccc|c}
0 & 1 & 0 & 0\\
0 & 0 & 1 & 0\\
0 & 0 & 0 & 0\\
\hline
0 & 0 & 0 & 0
\end{array}\right], \qquad \mb C = \left[ \begin{array}{cc|cc}
0 & 1 & 0 & 0\\
0 & 0 & 0 & 0\\
\hline
0 & 0 & 0 & 1\\
0 & 0 & 0 & 0
\end{array}\right].$$
A Jordan block $\mb J_i$ has a repeated eigenvalue $\lambda_i$ on the
diagonal, zeros below the diagonal and in the upper right hand corner,
and ones above the diagonal:
$$\mb J_i = \left[\begin{array}{ccccc}
\lambda_i & 1 & 0 & \cdots & 0\\
0 & \lambda_i & 1 & & 0\\
\vdots & & \ddots && \vdots\\
0 & 0 &   & \lambda_i & 1\\
0 & 0 & \cdots & 0 & \lambda_i
\end{array}\right].$$
Two matrices may have the same eigenvalues and the same number of eigenvectors, but if their Jordan blocks
are different sizes those matrices can not be similar.

\smallskip

Jordan's theorem says that every square matrix $\mb A$ is similar to a Jordan matrix $\mb J$, with Jordan blocks on the diagonal:
$$\mb J = \left[\begin{array}{cccc}
J_1 & 0 & \cdots & 0\\
0 & J_2 & \cdots & 0\\
\vdots & &\ddots & \vdots\\
0 & 0 & \cdots & J_d
\end{array}\right].$$
In a Jordan matrix, the eigenvalues are on the diagonal and there may be ones above
the diagonal; the rest of the entries are zero.  The number of
blocks is the number of eigenvectors -- there is one eigenvector per
block.

\bigskip

To summarize:
\begin{itemize}
\item If $\mb A$ has $n$ distinct eigenvalues, it is diagonalizable and its Jordan matrix is the diagonal matrix $\mb J = \mb \Lambda$.
\item If $\mb A$ has repeated eigenvalues and ``missing'' eigenvectors, then its Jordan matrix
  will have $n-d$ ones above the diagonal.
\end{itemize}
We have not learned to compute the Jordan matrix of a matrix which is
missing eigenvectors, but we do know how to diagonalize a matrix which
has $n$ distinct eigenvalues.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\end{document}