\documentclass{article}
\input{../macros.tex}

\begin{document}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section*{Lecture 24b: Exam 2 Review}

\subsection*{Material covered by the exam}

\begin{itemize}
\item Orthogonal matrices $\mb Q = \onebythree {\vb q_1} {...} {\vb q_n}$.  $\mb Q^T \mb Q = I$.

Projections -- Least Squares ``best fit'' solution to $\mb A \vb x = \vb b$.

Gram-Schmidt process for getting an orthonormal basis from any basis.

\item $\det \mb A$

Properties 1-3 that define the determinant.

Big formula for the determinant with $n!$ terms, each with $+$ or $-$.

Cofactors formula, leading to a formula for $\mb A^{-1}$.

\item Eigenvalues $\mb A \vb x = \lambda \vb x$.

$\det(\mb A - \lambda \mb I) = 0$.

Diagonalization:  If $\mb A$ has $n$ independent eigenvectors, then $\mb S^{-1} \mb A \mb S = \Lambda$ (this is $\mb A \vb x = \lambda \vb x$ for all $n$ eigenvectors at once).

Powers of $\mb A$:  $\mb A^k = (\mb S \mb \Lambda \mb S^{-1})^k = \mb S \mb \Lambda^k \mb S^{-1}$.
\end{itemize}

\subsection*{Sample questions}

\begin{enumerate}

\item Let $\vb a = \small \threebyone 212$.  

\begin{enumerate}[a)]

\item Find the projection matrix $\mb P$ that projects onto $\vb a$.

To answer this, we just use the formula for $\mb P$.  Ordinarily $P = \mb A (\mb A^T \mb A)^{-1}\mb A^T$, but here $\mb A$ is a column vector so:
$$\mb P = \frac {\vb a\, \vb a^T}{\vb a^T\vb a} = \frac{1}{9}\threebythree 424212424.$$

\item What is the rank of $\mb P$?

$\mb P$ has rank $1$ because each of its columns is some multiple of its second column, or because it projects onto a one dimensional subspace.

\item What is the column space of $\mb P$?

The line determined by $\vb a$.

\item What are the eigenvalues of $\mb P$?

Since $\mb P$ has rank $1$ we know it has a repeated eigenvalue of $0$.  We can use its trace or the fact that it's a projection matrix to find that $\mb P$ has the eigenvalue $1$.

The eigenvalues of $\mb P$ are 0, 0 and 1.

\item Find an eigenvector of $\mb P$ that has eigenvalue $1$.

Eigenvector $\vb a = \small \threebyone 212$ has eigenvalue one.  Because $\mb P$ is a projection matrix, any vector in the space it's projecting onto will be an eigenvector with eigenvalue $1$.

\item Suppose $\vb u_{k+1} = \mb P \vb u_k$ with initial condition $\vb u_0 = \small \threebyone 990$.  Find $\vb u_k$.

We're repeatedly projecting a vector onto a line:
$$\vb u_1 = \mb P \vb u_0 = \vb a \frac {\vb a^T \vb u_0}{\vb a^T\vb a} = \vb a \frac{27}{9} = 3\vb a = \small \threebyone 636.$$

$\vb u_2$ is the projection of $\vb u_1$ onto the line determined by $\vb a$.  But $\vb u_1$ already lies on the line through $\vb a$.  In fact, $\vb u_k = \mb P^k \vb u_0 = \mb P \vb u_0 = \small \threebyone 636$.

\item The exam might have a difference equation $\vb u_{k+1} = \mb A \vb u_k$ in which $\mb A$
is not a projection matrix with $\mb P^k = \mb P$.  In that case
we would find its eigenvalues and eigenvectors to calculate $\vb u_0 = c_1
\vb x_1 + c_2 \vb x_2 + c_3 \vb x_3$.  Then $\vb u_k =
c_1\lambda_1^k \vb x_1 + c_2 \lambda_2^k \vb x_2 + c_3 \lambda_3^k \vb x_3$.
(For the projection matrix $\mb P$ above, two eigenvalues are $0$ and the third is $1$, so two terms vanish and for the third $\lambda^k = 1$.  Then $\vb u_1 = \vb u_2 = \vb u_3 = \cdots$.)

\end{enumerate}

\item We're given the following data points:
\begin{center}
\begin{tabular}{|r|r|}
\hline
$t$ & $y$\\
\hline
1 & 4\\
2 & 5\\
3 & 8\\
\hline
\end{tabular}.
\end{center}

\begin{enumerate}[a)]

\item Find the straight line through the origin that best fits these points.

The equation of this line will be $y = Dt$.  There's only one unknown, $D$.
We would like a solution to the 3 equations:
\begin{eqnarray*}
1\cdot D &=& 4\\
2\cdot D &=& 5\\
3\cdot D &=& 8.
\end{eqnarray*}
or $\small \threebyone 123 D = \small \threebyone 458$, if we put this in the form $\mb A \vb x = \vb b$.  To find the best value for $D$ we solve the equation:
\begin{eqnarray*} 
\mb A^T\mb A \hat D &=& \mb A^T \vb b\\
14 \hat D = 38\\
\hat D = \frac{38}{14} = \frac{19}{7}.
\end{eqnarray*}

We conclude that the best fit line through the origin is $y =
\frac{19}{7}t$.  We can roughly check our answer by noting that the
line $y = 3t$ runs fairly close to the data points.

\item What vector did we just project onto what line?

There are two ways to think about least squares problems.  The first
is to think about the best fit line in the $ty$-plane.  The other way
is to think in terms of projections -- we're projecting $\vb b =
\small \threebyone 458$ onto the column space of $\mb A$ (the line
through $(1,2,3)$) to get as close as possible to a solution to $\mb A \vb x = \vb b$.

\end{enumerate}

\item The vectors $\vb a_1 = \threebyone 123$ and $\vb a_2 = \threebyone 111$ determine a plane.  Find two orthogonal vectors in the plane.

To answer this question we use the Gram-Shmidt process.  We start
with $\vb a_1$ and find a second vector $\vb B$ perpendicular
to $\vb a_1$ by subtracting the component of $\vb a_2$ that lies in
the $\vb a_1$ direction.
\begin{eqnarray*}
\vb B &=& \vb a_2 - \frac{\vb a_1^T \vb a_2}{\vb a_1^T \vb a_1} \vb a_1\\
&=& \threebyone 111 - \frac{6}{14} \threebyone 123\\
&=& \threebyone 111 - \threebyone {3/7}{6/7}{9/7}\\
&=& \threebyone {4/7}{1/7}{-2/7}.
\end{eqnarray*}

Because the dot product of $\vb a_1$ and $\vb B$ is zero, we know this
answer is correct.  These are our orthogonal vectors.

\item We're given a 4 by 4 matrix $\mb A$ with eigenvalues $\lambda_1$, $\lambda_2$, $\lambda_3$, $\lambda_4$.

\begin{enumerate}[a)]
\item What conditions must the $\lambda_i$ satisfy for the matrix to be invertible?

$\mb A$ is invertible if and only if none of the $\lambda_i$ are $0$.

If one of the $\lambda_i$ is zero, then there is a non-zero vector in
the nullspace of $\mb A$ and $\mb A$ is not invertible.

\item What is $\det \mb A^{-1}$?

The eigenvalues of $\mb A^{-1}$ are the inverses of the eigenvalues of
$\mb A$, so $\displaystyle{\det \mb A^{-1} = \left(\frac{1}{\lambda_1}\right)
\left( \frac{1}{\lambda_2}\right) \left(\frac{1}{\lambda_3}\right)
\left( \frac{1}{\lambda_4}\right)}$.

\item What is $\text{trace}(\mb A + \mb I)$?

We know the trace of $\mb A$ is the sum of the eigenvalues of $\mb
A$, so $\text{trace}(\mb A + \mb I) = (\lambda_1 + 1) + (\lambda_2 +
1) + (\lambda_3 + 1) + (\lambda_4 + 1) = \lambda_1 + \lambda_2 +
\lambda_3 + \lambda_4 + 4$.

\end{enumerate}

\item Remember the family of {\em tridiagonal matrices}; for example:
$$\mb A_4 = \left[ \begin{array}{cccc}
1 & 1 & 0 & 0\\
1 & 1 & 1 & 0\\
0 & 1 & 1 & 1\\
0 & 0 & 1 & 1
\end{array}\right].$$
Let $D_n = \det \mb A_n$.

\begin{enumerate}[a)]
\item Use cofactors to show that $D_n = a D_{n-1} + b D_{n-2}$ and
  find values for $a$ and $b$.

Using the cofactor formula we find that the determinant of $\mb A_4$ is:
\begin{eqnarray*}
D_4 &=& 1 \threebythreedet 110111011 - 1 \threebythreedet 110011011 + 0 - 0\\
&=& 1 D_3 - 1 \cdot 1 D_2\\
&=& D_3 - D_2.
\end{eqnarray*}
In general, $D_n = D_{n-1} - D_{n-2}$.  The answer is $a =1$ and $b=-1$.

\item In part (a) you found a recurrence relation $D_n = a D_{n-1} + b
  D_{n-2}$.  Find a way to predict the value of $D_n$ for any $n$.
  Note: if your computations are all correct it should be true that
  $\lambda_1^6 = \lambda_2^6 = 1$.

We can quickly compute $D_1 = 1$ and $D_2 = \small \twobytwodet 1111 = 0$.

We set up the system for $D_n = D_{n-1}-D_{n-2}$:
$$\twobyonec {D_n}{D_{n-1}} = \twobytwo 1{-1}10 \twobyone
{D_{n-1}}{D_{n-2}}$$ to get an equation of the form $\vb u_k = \mb A
\vb u_{k-1}$.

To find the eigenvalues $\lambda_i$, solve $\det(A - \lambda I) = 0$:
$$\twobytwodetc {1-\lambda} {-1}1 {-\lambda} = \lambda^2 - \lambda + 1
= 0.$$ The quadratic formula gives us $\displaystyle{\lambda = \frac{1
    \pm \sqrt{1-4}}{2}}$, so: $$\lambda_1 = \frac{1+\sqrt 3 i}{2} = e^{i
  \pi/3} \text{ and } \lambda_2 = \frac{1-\sqrt 3 i}{2} = e^{-i \pi/3}.$$  The
magnitude of these complex numbers is $1$.  This tells us that the
system is stable.  The fact that $\lambda_1^6 = \lambda_2^6$ tells
us that $\mb A^6 = \mb I$ and so the sequence of vectors $\vb u_k =
A^k \vb u_{k-1}$ will repeat every $6$ steps.

To finish answering the problem, we can use the recurrence relation
$D_n = D_{n-1} - D_{n-2}$ starting from $D_1 = 1$ and $D_2 = 0$ to find $D_3 = -1$, $D_4 = -1$, $D_5 = 0$ and
$D_6 = 1$.  The sequence will then repeat, with $D_7 = D_1 = 1$, $D_8
= D_2 = 0$ and so on.  If $n = 6j + k$ for positive integers $j$ and
$k$, then $D_n = D_k$.

\end{enumerate}

\item Consider the following family of symmetric matrices:
$$\mb A_2 = \twobytwo 0110, \quad \mb A_3 = \threebythree 010102020, \quad
\mb A_4 = \left[ \begin{array}{cccc}
0 & 1 & 0 & 0\\
1 & 0 & 2 & 0\\
0 & 2 & 0 & 3\\
0 & 0 & 3 & 0
\end{array}\right], ...$$

\begin{enumerate}[a)]

\item Find the projection matrix $\mb P$ onto the column space of $\mb A_3$.

We know that $\mb A_3$ is singular because column 3 is a multiple of
column 1, so $\mb P$ is a projection matrix onto a plane.  Columns $1$ and $2$ form
a basis for the column space of $\mb A_3$, so we could use the formula $\mb P = \mb A(\mb A^T\mb
A)^{-1}\mb A^T$ with $\mb A = \small \threebytwo 011002$ to find:
$$\mb P = \threebythreec{1/5}{0}{2/5}010{2/5}0{4/5}.$$
However, there may be a quicker way to solve this problem.

To check our work we multiply $\mb P$ by the column vectors of $\mb A$
to see that $\mb P \mb A = \mb A$.

\item What are the eigenvalues and eigenvectors of $\mb A_3$?

$$|\mb A_3 - \lambda \mb I| = \threebythreedet {-\lambda} 1 0 1
  {-\lambda} 2 0 2 {-\lambda} = -\lambda^3 + 5 \lambda.$$ Setting
  $|\mb A_3 - \lambda \mb I| = 0$ gives us $\lambda (-\lambda^2 + 5) =
  0$, so $\lambda_1 =0$, $\lambda_2 = \sqrt 5$, $\lambda_3 =
  -\sqrt{5}$.  

We check that the trace of $\mb A_3$ equals the sum of its
eigenvalues.

Next we solve $(\mb A_3 - \lambda I) \vb x = \vb 0$ to find our
eigenvectors.  A good strategy for doing this is to choose one
component of $\vb x$ to set equal to $1$, then determine what the
other components of $\vb x$ must be for the product to equal the zero
vector.

$(\mb A_3 - 0 \mb I)\vb x = \vb 0$ has the solution $\vb
x_1 = \threebyone {-2}01$.

$(\mb A_3 - \sqrt 5 \mb I)\vb x = \vb 0$ has the solution $\vb
x_2 = \threebyone 1{\sqrt 5}2$.

$(\mb A_3 + \sqrt 5 \mb I)\vb x = \vb 0$ has the solution $\vb
x_2 = \threebyone 1{-\sqrt 5}2$.

If time permits, we can check this answer by multiplying each
eigenvector by $\mb A_3$.

\item (This is not difficult.)  What is the projection matrix onto the
  column space of $\mb A_4$?

How could this not be difficult?  If $\mb A_4$ is invertible, then its
column space is $\R^4$ and the answer is $\mb P = \mb I$.

To confirm that $\mb A_4$ is invertible, we can check that its determinant is non-zero.  This is not difficult if we use cofactors:
$$\det \mb A_4 = (-1)\left( 1 \cdot \twobytwodet 0330 + 2\cdot \twobytwodet 0300 \right) = 9.$$

Because $\mb A_4$ is invertible, the projection matrix onto its column
space is $\mb I$.

\item Bonus question:  Prove or disprove that $A_n$ is singular if $n$ is odd and invertible if $n$ is even.
\end{enumerate}

\end{enumerate}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\end{document}
