Lecture 34: Final course review

Once more, we review questions from a previous exam to prepare ourselves for an upcoming exam.

1. Suppose we know that A is an m by n matrix of rank r, Ax = [ \begin{matrix}1 \\ 0 \\ 0 \end{matrix} ] has

no solution, and Ax = [ \begin{matrix}0 \\ 1 \\ 0 \end{matrix} ] has exactly one solution.

a) What can we say about m, n and r? The product Ax is a vector in three dimensions, so m = 3.

The fact that Ax = [ \begin{matrix}1 \\ 0 \\ 0 \end{matrix} ] has no solution tells us that the column

space is not all of ^{3}. In addition, we know that the column space

contains [ \begin{matrix}0 \\ 1 \\ 0 \end{matrix} ], so r is not zero: 1 ≤ r < 3.

The fact that Ax = [ \begin{matrix}0 \\ 1 \\ 0 \end{matrix} ] has exactly one solution tells us that the

nullspace of A contains only the zero vector and so n = r. Hence 1 ≤ n < 3.

b) Write down an example of a matrix A that fits this description.

The vector [ \begin{matrix}0 \\ 1 \\ 0 \end{matrix} ] must be in the column space, so we’ll make it a

column of A. The simplest way to answer this question is to stop here.

A = [ \begin{matrix}0 \\ 1 \\ 0 \end{matrix} ] .

In this solution, n = r = 1 and m = 3.

To find a solution in which n = r = 2, add a second column. Make

sure that [ \begin{matrix}1 \\ 0 \\ 0 \end{matrix} ] is not in the column space:

A = [ \begin{matrix}0 & 0 \\ 1 & 0 \\ 0 & 1 \end{matrix} ] .

There are many other correct answers to this question. c) Cross out all statements that are false about any matrix with the given

properties (which are 1 ≤ r = n, m = 3).

i. detA^{T}A = detAA^{T} ii. A^{T}A is invertible iii. AA^{T} is positive definite

One good approach to this problem is to use our sample matrix to test each statement.

i. If we leave this part to last, we can quickly answer it (false) using

what we learn while answering the following two parts. ii. The matrix A^{T}A is invertible if r = n; i.e. if the columns of A are

independent. The nullspace of our A contains only the zero vector, so this statement is true. For each of our sample matrices, A^{T}A equals the identity and so is invertible. Note that this means detA^{T}A ≠ 0.

iii. We know that m = 3 and r < 3, so AA^{T} will be a 3 by 3 matrix with rank less than 3; it can’t be positive definite. (It is true that for any matrix A with real valued entries, AA^{T} is positive semidefinte.) For our test matrices, AA^{T} has at least one row that’s all zeros, so 0 is an eigenvalue (and is not positive). Note also that detAA^{T} = 0 and so statement (i) must be false. (However, if A and B are square matrices then detBA = detAB = detA detB.)

d) Prove that A^{T}y = c has at least one solution for every right hand side c, and in fact has infinitely many solutions for every c. We know A^{T} is an n by m matrix with m = 3 and rank r = n < m. If A^{T} has full row rank, the equation A^{T}y = c is always solvable. We have n rows and rank r = n, so A^{T} has full row rank. Therefore A^{T}y = c has a solution for every vector c. The solvable system A^{T}y = c will have infinitely many solutions if the nullspace of A^{T} has positive dimension. We know dim(N(A^{T})) = m − r > 0, so A^{T}y = c has infinitely many solutions for every c.

2. Suppose the columns of A are v_{1}, v_{2} and v_{3}.

a) Solve Ax = v_{1} − v_{2} + v_{3}. This is just the “column method” of multiplying matrices from the

first lecture. Choose x = [ \begin{matrix}1 \\ −1 \\ 1 \end{matrix} ]. b) True or false: if v_{1} − v_{2} + v_{3} = 0, then the solution to (2a) is not

unique. Explain your answer.

True. Any scalar multiple of x = [ \begin{matrix}1 \\ −1 \\ 1 \end{matrix} ] will be a solution.

Another way of answering this is to note that A^{T} has a nontrivial nullspace, and we can always add any vector in the nullspace to a solution x to get a different solution.

c) Suppose v_{1}, v_{2} and v_{3} are orthonormal (forget about (2b)). What combination of v_{1} and v_{2} is closest to v_{3}? If we imagine the right triangle out from the origin formed by av_{1} + bv_{2} and v_{3}, the Pythagorean theorem tells us that 0v_{1} + 0v_{2} = 0 is the closest point to v_{3} in the plane spanned by v_{1} and v_{2}.

3. Suppose we have the Markov matrix

A = [ \begin{matrix}.2 & .4 & .3 \\ .4 & .2 & .3 \\ .4 & .4 & .4 \end{matrix} ] .

Note that the sum of the first two columns of A equals twice the third column of A.

a) What are the eigenvalues of A? Zero is an eigenvalue because the columns of A are dependent. (A is singular.) One is an eigenvalue because A is a Markov matrix. The third eigenvalue is −.2 because the trace of A is .8. So λ = 0, 1, −.2.

b) Let u_{k} = A^{k}u(0). If u(0) = [ \begin{matrix}0 \\ 10 \\ 0 \end{matrix} ], what is lim_{k→∞} u_{k}?

We’ll start by computing u_{k} and then find the steady state. This means finding a general expression of the form:

u_{k} = c_{1}λ_{1}^{k}x_{1} + c_{2}λ_{2}^{k}x_{2} + c_{3}λ_{3}^{k}x_{3}.

When we plug in the eigenvalues we found in part (3a), this becomes

u_{k} = 0 + c_{2}x_{2} + c_{3}(−.2)^{k}x_{3}.

We see that as k approaches infinity, c_{2}x_{2} is the only term that does not go to zero. The key eigenvector in any Markov process is the one with eigenvalue one.

To find x_{2}, solve (A − 1I)x_{2} = 0:

[ \begin{matrix}−.8 & .4 & .3 \\ .4 & −.8 & .3 \\ .4 & .4 & −.6 \end{matrix} ] x_{2} = 0.

The best way to solve this might be by elimination. However, because the first two columns look like multiples of 4 and the third column looks like a multiple of 3, we might get lucky and guess x_{2} = [ \begin{matrix}3 \\ 3 \\ 4 \end{matrix} ].

This gives us u_{∞} = c_{2} [ \begin{matrix}3 \\ 3 \\ 4 \end{matrix} ]. We know that in a Markov process, the sum of the entries of u_{k} is the same for all k. The sum of the entries of u(0) is 10, so c_{2} = 1 and u_{∞} = [ \begin{matrix}3 \\ 3 \\ 4 \end{matrix} ].

4. Find a two by two matrix that:

a) projects onto the line spanned by a = [ \begin{matrix}4 \\ −3 \end{matrix} ]. The formula for this matrix is P = \frac{aa^{T}}{a^{T}a}. This gives us

P = [ \begin{matrix}16/25 & −12/25 \\ −12/25 & 9/25 \end{matrix} ] .

(To test this answer, we can quickly check that detP = 0.)

b) has eigenvalues λ_{1} = 0 and λ_{2} = 3 and eigenvectors x_{1} = [ \begin{matrix}1 \\ 2 \end{matrix} ] and

x_{2} = [ \begin{matrix}2 \\ 1 \end{matrix} ].

Here the formula we need is A = SΛS^{−1} .

A = [ \begin{matrix}1 & 2 \\ 2 & 1 \end{matrix} ] [ \begin{matrix}0 & 0 \\ 0 & 3 \end{matrix} ] [ \begin{matrix}1 & 2 \\ 2 & 1 \end{matrix} ]^{−1}
= [ \begin{matrix}1 & 2 \\ 2 & 1 \end{matrix} ] [ \begin{matrix}0 & 0 \\ 0 & 3 \end{matrix} ] [ \begin{matrix}−1/3 & 2/3 \\ 2/3 & −1/3 \end{matrix} ]
A = [ \begin{matrix}4 & −2 \\ 2 & −1 \end{matrix} ] .

If time permits, we can check this by computing the products Ax_{i}.

c) has real entries and cannot be factored as B^{T}B for any B. We know that B^{T}B will always be symmetric, so any asymmetric matrix has this property. For example, we could choose [ \begin{matrix}0 & 0 \\ 1 & 0 \end{matrix} ]. d) is not symmetric, but has orthogonal eigenvectors.

We know that symmetric matrices have orthogonal eigenvectors, but so do other types of matrices (e.g. skew symmetric and orthogonal) when we allow complex eignevectors. Two possible answers are:

[ \begin{matrix}0 & 1 \\ −1 & 0 \end{matrix} ]  (skew symmetric)
[ \begin{matrix}cos θ & − sin θ \\ sin θ & cos θ \end{matrix} ] (orthogonal).

5. Applying the least squares method to the system

[ \begin{matrix}1 & 0 \\ 1 & 1 \\ 1 & 2 \end{matrix} ] [ \begin{matrix}c \\ d \end{matrix} ] = [ \begin{matrix}3 \\ 4 \\ 1 \end{matrix} ] = b

gives the best fit vector [ ĉ ; d̂ ] = [ \begin{matrix}11/3 \\ −1 \end{matrix} ].

a) What is the projection p of b = [ \begin{matrix}3 \\ 4 \\ 1 \end{matrix} ] onto the column space of

A = [ \begin{matrix}1 & 0 \\ 1 & 1 \\ 1 & 2 \end{matrix} ]?

We know that 11/3 times the first column minus 1 times the second

column is the closest point P in the column space to [ \begin{matrix}3 \\ 4 \\ 1 \end{matrix} ], so the

answer is

A [ ĉ ; d̂ ] = \frac{11}{3} [ \begin{matrix}1 \\ 1 \\ 1 \end{matrix} ] − [ \begin{matrix}0 \\ 1 \\ 2 \end{matrix} ] = [ \begin{matrix}11/3 \\ 8/3 \\ 5/3 \end{matrix} ] .

b) Draw the straight line problem that corresponds to this system. Plotting the entries of the second column of A against the entries of b we get the three points shown in Figure 1. The best fit line is ĉ+ d̂t.

c) Find a different vector b ≠ 0 ∈ ^{3} so that the least squares solution is

[ ĉ ; d̂ ] = [ \begin{matrix}0 \\ 0 \end{matrix} ].

We know that [ ĉ ; d̂ ] is the projection of b onto the column space, so to

get a zero projection we need to find a vector orthogonal to the columns.

Figure 1: Three data points and their “best fit” line \frac{11}{3} − t. We could get the answer b = [ \begin{matrix}1 \\ −2 \\ 1 \end{matrix} ] by inspection, or we could use the cross product of the columns to find a value for b.

Thank you for taking this course!