\documentclass{article}
\input{../macros.tex}

\begin{document}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section*{Lecture 22: Diagonalization and powers of $\mb A$}

We know how to find eigenvalues and eigenvectors.  In this lecture we
learn to {\em diagonalize} any matrix that has $n$ independent
eigenvectors and see how diagonalization simplifies calculations.  The lecture concludes by using eigenvalues and eigenvectors to solve {\em difference equations}.

\subsection*{Diagonalizing a matrix $\mb S^{-1}\mb A\mb S = \mb \Lambda$}

If $\mb A$ has $n$ linearly independent eigenvectors, we can put
those vectors in the columns of a (square, invertible) matrix $\mb S$.
Then 
\begin{eqnarray*}
\mb A \mb S &=& \mb A \onebyfour {\vb x_1}{\vb x_2}{\cdots}{\vb x_n}\\ 
&=&\onebyfour {\lambda_1 \vb x_1}{\lambda_2 \vb x_2}{\cdots}{\lambda_n
  \vb x_n}\\
&=& \mb S \left[ \begin{array}{rrrr} 
\lambda_1 & 0 & \cdots & 0\\
0 & \lambda_2 &  & 0\\
\vdots &  & \ddots & \vdots\\
0 & \cdots & 0 & \lambda_n\end{array}\right]
= \mb S \mb \Lambda.
\end{eqnarray*}
Note that $\mb \Lambda$ is a
diagonal matrix whose non-zero entries are the eigenvalues of $\mb A$.
Because the columns of $\mb S$ are independent, $\mb S^{-1}$ exists and we can multiply both sides of $\mb A \mb S = \mb S \mb \Lambda$ by $\mb S^{-1}$:
$$\mb S^{-1} \mb A \mb S = \mb \Lambda.$$ Equivalently, $\mb A =
\mb S \mb \Lambda \mb S^{-1}$.

\subsection*{Powers of $\mb A$}

What are the eigenvalues and eigenvectors of $\mb A^2$?
\begin{eqnarray*}
\text{If } \mb A \vb x &=& \lambda \vb x,\\
\text{then } \mb A^2 \vb x &=& \lambda \mb A\vb x = \lambda^2 \vb x.
\end{eqnarray*}
The eigenvalues of $\mb A^2$ are the squares of the eigenvalues of
$\mb A$.  The eigenvectors of $\mb A^2$ are the same as the eigenvectors
of $\mb A$.  If we write $\mb A = \mb S \mb \Lambda \mb S^{-1}$
then: $$\mb A^2 = \mb S \mb \Lambda \mb S^{-1} \mb S \mb \Lambda \mb
S^{-1} = \mb S \mb \Lambda^2 \mb S^{-1}.$$ Similarly, $\mb A^k = \mb S
\mb \Lambda^k \mb S^{-1}$ tells us that raising the eigenvalues of
$\mb A$ to the $k$th power gives us the eigenvalues of $\mb A^k$, and
that the eigenvectors of $\mb A^k$ are the same as those of $\mb A$.

\bigskip

Theorem: If $\mb A$ has $n$ independent eigenvectors with eigenvalues
$\lambda_i$, then $\mb A^k \rightarrow \mb 0$ as $k \rightarrow
\infty$ if and only if all $|\lambda_i| < 1$.

\bigskip

$\mb A$ is guaranteed to have $n$ independent eigenvectors (and be
{\em diagonalizable}) if all its eigenvalues are different.  Most
matrices do have distinct eigenvalues.

\subsection*{Repeated eigenvalues}

If $\mb A$ has repeated eigenvalues, it may or may not have $n$
independent eigenvectors.  For example, the eigenvalues of the
identity matrix are all $1$, but that matrix still has $n$ independent
eigenvectors.

If $\mb A$ is the triangular matrix $\small \twobytwo 2102$ its
eigenvalues are $2$ and $2$.  Its eigenvectors are in the nullspace of
$\mb A-\lambda \mb I = \small \twobytwo 0100$ which is spanned by $\vb
x = \small \twobyone 10$.  This particular $\mb A$ does not have two independent
eigenvectors.

\subsection*{Difference equations $\vb u_{k+1} = \mb A \vb u_k$}

Start with a given vector $\vb u_0$.  We can create a sequence of vectors in which each new vector is $\mb A$ times the previous vector: $\vb u_{k+1} = \mb A \vb u_k$.  $\vb u_{k+1} = \mb A \vb u_k$ is a {\em first order difference equation}, and $\vb u_k = \mb A^k
\vb u_{0}$ is a solution to this system.

We get a more satisfying solution if we write $\vb u_0$ as a combination of eigenvectors of $\mb A$:
$$\vb u_0 = c_1 \vb x_1 + c_2 \vb x_2 + \cdots + c_n \vb x_n  = \mb S \vb c.$$  Then: 
$$\mb A \vb u_0 =  c_1 \lambda_1 \vb x_1 +  c_2 \lambda_2 \vb x_2 + \cdots +
 c_n \lambda_n\vb x_n$$
and: 
$$\vb u_k = \mb A^k \vb u_0 =  c_1 \lambda_1^k \vb x_1 +  c_2 \lambda_2^k \vb x_2 + \cdots +
 c_n \lambda_n^k\vb x_n = \mb \Lambda^k \mb S \vb c.$$

\subsection*{Fibonacci sequence}

The Fibonacci sequence is $0, 1, 1, 2, 3, 5, 8, 13, ...$.  In general,
$F_{k+2} = F_{k+1} + F_{k}$.  If we could understand this in terms of
matrices, the eigenvalues of the matrices would tell us how fast the
numbers in the sequence are increasing.

$\vb u_{k+1} = \mb A \vb u_k$ was a first order system.  $F_{k+2} =
F_{k+1} + F_{k}$ is a second order scalar equation, but we can convert
it to first order linear system by using a clever trick.  If $\vb u_k
= \small \twobyone {F_{k+1}}{F_k}$, then:
\begin{eqnarray}
F_{k+2} &=& F_{k+1} + F_{k}\\
F_{k+1} &=& F_{k+1}.
\end{eqnarray}
is equivalent to the first order system $\vb u_{k+1} = \small
\twobytwo 1110 \vb u_k$.

What are the eigenvalues and eigenvectors of $\mb A = \small \twobytwo
1110$?  Because $\mb A$ is symmetric, its eigenvalues will be real and its
eigenvectors will be orthogonal.  Because $\mb A$ is a two by two
matrix we know its eigenvalues sum to $1$ (the trace) and their
product is $-1$ (the determinant).
$$|\mb A - \lambda \mb I| = \twobytwodetc {1-\lambda}11{-\lambda} =
\lambda^2 - \lambda - 1$$ Setting this to zero we find $\lambda =
\frac{1 \pm \sqrt{1+4}}{2}$; i.e. $\lambda_1 = \frac{1}{2}(1+\sqrt 5)
  \approx 1.618$ and $\lambda_2 = \frac{1}{2}(1-\sqrt 5) \approx
    -.618$.  The growth rate of the $F_k$ is controlled by $\lambda_1$, the
    only eigenvalue with absolute value greater than $1$.  This tells
    us that for large $k$, $F_k \approx c_1\left(\frac{1+\sqrt
      5}{2}\right)^{k}$ for some constant $c_1$.  (Remember
    $\vb u_k = \mb A^{k} \vb u_0 = c_1 \lambda_1^{k} x_1 + c_2
    \lambda_2^{k} x_2$, and here $\lambda_2^k$ goes to zero since $|\lambda_2| < 1$.)

To find the eigenvectors of $\mb A$ note that:
$$(\mb A - \lambda \mb I)\vb x = \twobytwoc {1-\lambda}11{-\lambda}
\vb x$$ equals $\vb 0$ when $\vb x = \small \twobyone \lambda 1$, so $\vb x_1 =
\small \twobyone {\lambda_1} 1$ and $\vb x_2 = \small \twobyone
       {\lambda_2} 1$.

Finally, $\vb u_0 = \small \twobyone {F_1}{F_0} = \small \twobyone 10 = c_1 \vb x_1 + c_2 \vb x_2$ tells us that $c_1 = -c_2 = \frac{1}{\sqrt 5}$.  Because $\twobyone {F_{k+1}}{F_k} = \vb u_k = c_1 \lambda_1^{k} x_1 + c_2
    \lambda_2^{k} x_2$, we get: $$F_k = \frac{1}{\sqrt{5}}\left(\frac{1+\sqrt
      5}{2}\right)^{k} - \frac{1}{\sqrt 5} \left(\frac{1-\sqrt
      5}{2}\right)^{k}.$$
Using eigenvalues and eigenvectors, we have found a {\em closed form expression} for the Fibonacci numbers.

\bigskip

Summary:  When a sequence evolves over time according to the rules of a first order
system, the eigenvalues of the matrix of that system determine the
long term behavior of the series.  To get an exact formula for the series we find the eigenvectors of the matrix and then solve
for the coefficients $c_1, c_2,...$

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\end{document}