\documentclass{article}
\input{../macros.tex}

\begin{document}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section*{Lecture 34:  Final course review}

Once more, we review questions from a previous exam to prepare ourselves for an upcoming exam.

\begin{enumerate}

\item Suppose we know that $\mb A$ is an $m$ by $n$ matrix of rank $r$, $\mb A \vb x = \small \threebyone 100$ has no solution, and $\mb A \vb x = \small \threebyone 010$ has exactly one solution.

\begin{enumerate}[a)]

\item What can we say about $m$, $n$ and $r$?

The product $\mb A \vb x$ is a vector in three dimensions, so $m = 3$.

The fact that $\mb A \vb x = \small \threebyone 100$ has no solution
tells us that the column space is not all of $\R^3$.  In
addition, we know that the column space contains $\small \threebyone
010$, so $r$ is not zero:  $1 \leq r < 3$.

The fact that $\mb A \vb x = \small \threebyone 010$ has exactly one
solution tells us that the nullspace of $\mb A$ contains only the
zero vector and so $n=r$.  Hence $1 \leq n < 3$.

\item Write down an example of a matrix $\mb A$ that fits this description.

The vector $\threebyone 010$ must be in the column space, so we'll
make it a column of $\mb A$.  The simplest way to answer this question
is to stop here.
$$\mb A = \threebyone 010.$$
In this solution, $n=r=1$ and $m=3$.

\bigskip

To find a solution in which $n=r=2$, add a second column.   Make sure
that $\small \threebyone 100$ is not in the column space:
$$\mb A = \threebytwo 001001.$$

\smallskip

There are many other correct answers to this question.

\item Cross out all statements that are false about any matrix with
  the given properties (which are $1 \leq r = n$, $m=3$).
\begin{enumerate}
\item \label{a} $\det \mb A^T \mb A = \det \mb A \mb A^T$
\item \label{b} $\mb A^T \mb A$ is invertible
\item \label{c} $\mb A \mb A^T$ is positive definite
\end{enumerate}

One good approach to this problem is to use our sample matrix to test each statement.

\begin{enumerate}

\item If we leave this part to last, we can quickly answer it ({\bf
    false}) using what we learn while answering the following two
  parts.

\item The matrix $\mb A^T \mb A$ is invertible if $r=n$; i.e. if the columns of $\mb A$ are independent.

The nullspace of our $\mb A$ contains only the zero vector,
so this statement is {\bf true}.

For each of our sample matrices, $\mb A^T\mb A$ equals the identity
and so is invertible.

Note that this means $\det \mb A^T \mb A \neq 0$.

\item We know that $m = 3$ and $r < 3$, so $\mb A \mb A^T$ will be a
  $3$ by $3$ matrix with rank less than $3$; it {\bf can't be positive
  definite}.  (It is true that for any matrix $\mb A$ with real valued entries, $\mb
A \mb A^T$ is positive semidefinte.)

For our test matrices, $\mb A \mb A^T$ has at least one row that's all
zeros, so $0$ is an eigenvalue (and is not positive).
\smallskip

Note also that $\det \mb A \mb A^T = 0$ and so statement (i) must be
false.  (However, if $\mb A$ and $\mb B$ are square matrices then
$\det \mb B \mb A = \det \mb A \mb B = \det \mb A \det \mb B$.)
\end{enumerate}

\item Prove that $\mb A^T \vb y = \vb c$ has at least one solution for
  every right hand side $\vb c$, and in fact has infinitely many
  solutions for every $\vb c$.

We know $\mb A^T$ is an $n$ by $m$ matrix with $m=3$ and rank $r=n<m$.
If $\mb A^T$ has full row rank, the equation $\mb A^T \vb y = \vb c$
is always solvable.  We have $n$ rows and rank $r=n$, so $\mb
A^T$ has full row rank.  Therefore $\mb A^T \vb y = \vb c$ has a
solution for every vector $\vb c$.

The solvable system $\mb A^T \vb y = \vb c$ will have infinitely many
solutions if the nullspace of $\mb A^T$ has positive dimension.  We
know $\dim(N(A^T)) = m-r > 0$, so $\mb A^T \vb y = \vb c$ has
infinitely many solutions for every $\vb c$.

\end{enumerate}

\item Suppose the columns of $\mb A$ are $\vb v_1$, $\vb v_2$ and $\vb v_3$.

\begin{enumerate}[a)]

\item \label{aa} Solve $\mb A \vb x = \vb v_1 - \vb v_2 + \vb v_3$.

This is just the ``column method'' of multiplying matrices from the
first lecture.  Choose  $\vb x = \small \threebyone 1 {-1} 1$.

\item \label{bb} True or false:  if $\vb v_1 - \vb v_2 + \vb v_3 = \vb 0$, then the solution to (\ref{aa}) is not unique.  Explain your answer.

{\bf True}.  Any scalar multiple of $\vb x = \threebyone 1 {-1} 1$ will be a
solution.  

Another way of answering this is to note that $\mb A^T$ has a
nontrivial null\-space, and we can always add any vector in the
nullspace to a solution $\vb x$ to get a different solution.

\item Suppose $\vb v_1$, $\vb v_2$ and $\vb v_3$ are orthonormal (forget about (\ref{bb})).  What combination of $\vb v_1$ and $\vb v_2$ is closest to $\vb v_3$?

  If we imagine the right triangle out from the origin formed by $a
  \vb v_1 + b \vb v_2$ and $\vb v_3$, the Pythagorean theorem tells us
  that $0 \vb v_1 + 0\vb v_2 = \vb 0$ is the closest point to $\vb
  v_3$ in the plane spanned by $\vb v_1$ and $\vb v_2$.

\end{enumerate}

\item Suppose we have the Markov matrix
$$\mb A = \threebythree {.2}{.4}{.3}{.4}{.2}{.3}{.4}{.4}{.4}.$$
Note that the sum of the first two columns of $\mb A$ equals twice the third column of $\mb A$.

\begin{enumerate}[a)]
\item \label{aaa} What are the eigenvalues of $\mb A$?

Zero is an eigenvalue because the columns of $\mb A$ are dependent.  ($\mb A$ is singular.)

One is an eigenvalue because $\mb A$ is a Markov matrix.

The third eigenvalue is $-.2$ because the trace of $\mb A$ is $.8$.
So $\lambda = 0, 1, -.2$.

\item Let $\vb u_k = \mb A^k \vb u(0)$.  If $\vb u(0) =
  \small \threebyone 0{10}0$, what is $\lim_{k \to \infty} \vb u_k$?

We'll start by computing $\vb u_k$ and then find the steady state.
This means finding a general expression of the form:
$$\vb u_k = c_1 \lambda_1^k \vb x_1 + c_2 \lambda_2^k \vb x_2 + c_3 \lambda_3^k \vb x_3.$$
When we plug in the eigenvalues we found in part (\ref{aaa}), this becomes
$$\vb u_k = \vb 0 + c_2 \vb x_2 + c_3 (-.2)^k \vb x_3.$$  We see that as $\vb k$ approaches infinity, $c_2 \vb x_2$ is the only term that does not go to zero.

The key eigenvector in any Markov process is the one with eigenvalue one.

To find $\vb x_2$, solve $(\mb A - 1I)\vb x_2 = \vb 0$:
$$\threebythree {-.8}{.4}{.3}{.4}{-.8}{.3}{.4}{.4}{-.6} \vb x_2 = \vb
0.$$ The best way to solve this might be by elimination.  However,
because the first two columns look like multiples of $4$ and the third
column looks like a multiple of $3$, we might get lucky and guess
$\vb x_2 = \small \threebyone 334$.

This gives us $\vb u_{\infty} = c_2 \small \threebyone 334$. We know
that in a Markov process, the sum of the entries of $\vb u_k$ is the same for all $k$.  The sum of the entries of $\vb u(0)$ is $10$, so $c_2 =1$ and  $\vb u_\infty = \small \threebyone 334$.

\end{enumerate}

\item Find a two by two matrix that:

\begin{enumerate}[a)]

\item projects onto the line spanned by $\vb a = \twobyone 4{-3}$.

The formula for this matrix is $\mb P = \displaystyle{\frac{\vb a \vb a^T}{\vb a^T \vb a}}$.  This gives us $$\mb P = \twobytwo {{16}/{25}}{-{12}/{25}}{-{12}/{25}}{{9}/{25}}.$$  (To test this answer, we can quickly check that $\det P = 0$.)

\item has eigenvalues $\lambda_1 = 0$ and $\lambda_2 = 3$ and
  eigenvectors $\vb x_1 = \twobyone 12$ and $\vb x_2 = \twobyone 21$.

Here the formula we need is $\mb A = \mb S \mb \Lambda \mb S^{-1}$.
\begin{eqnarray*}
\mb A &=& \twobytwo 1221 \twobytwo 0003 \twobytwo 1221^{-1}\\
&=& \twobytwo 1221 \twobytwo 0003 \twobytwo {-1/3}{2/3}{2/3}{-1/3}\\
\mb A &=& \twobytwo 4{-2}2{-1}.
\end{eqnarray*}

If time permits, we can check this by computing the products $\mb A \vb x_i$.

\item has real entries and cannot be factored as $\mb B^T \mb B$ for
  any $\mb B$.

We know that $\mb B^T \mb B$ will always be symmetric, so any
asymmetric matrix has this property.  For example, we could choose $\small
\twobytwo 0010$.

\item is not symmetric, but has orthogonal eigenvectors.

We know that symmetric matrices have orthogonal eigenvectors, but so
do  other types of matrices (e.g. skew symmetric and orthogonal) when
we allow complex eignevectors.

Two possible answers are:
$$\twobytwo 01{-1}0 \qquad \text{(skew symmetric)}$$
$$\twobytwo {\cos \theta }{-\sin \theta}{\sin \theta }{\cos \theta}\qquad \text{(orthogonal)}.$$

\end{enumerate}

\item Applying the least squares method to the system
$$\threebytwo 101112 \twobyone {c}{d} = \threebyone 341 = \vb b$$
  gives the best fit vector $\twobyone {\hat c}{\hat d} = \twobyone
  {11/3}{-1}$.

\begin{enumerate}[a)]

\item What is the projection $\vb p$ of $\vb b = \small \threebyone 341$ onto the
column space of $$\mb A = \small \threebytwo 101112?$$

We know that $11/3$ times the first column minus $1$ times the
second column is the closest point $P$ in the column space to $\small
\threebyone 341$, so the answer is
$$\mb A \twobyone {\hat c}{\hat d} = \frac{11}{3} \threebyone 111 -
\threebyone 012 = \threebyone {11/3}{8/3}{5/3}.$$

\item Draw the straight line problem that corresponds to this system.

Plotting the entries of the second column of $\mb A$ against the
entries of $\vb b$ we get the three points shown in
Figure~\ref{fig:l34_g1}.  The best fit line is $\hat c + \hat d t$.

\begin{figure}[h]
\centering
  \includegraphics{../Images/lec34fig1.eps}\\
  \caption{Three data points and their ``best fit'' line $\frac{11}{3}
    - t$.}\label{fig:l34_g1}
\end{figure}

\item Find a different vector $\vb b \neq \vb 0 \in \R^3$ so that the least squares solution is $\twobyone {\hat c}{\hat d} = \twobyone 00$.

We know that $\twobyone {\hat c}{\hat d}$ is the projection of $\vb b$
onto the column space, so to get a zero projection we need to find a
vector {\em orthogonal to the
columns}.  We could get the answer $\vb b = \small \threebyone 1{-2}1$ by inspection,
or we could use the cross product of the columns to find a value for $\vb b$.

\end{enumerate}

\end{enumerate}

Thank you for taking this course!

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\end{document}