\documentclass{article}
\input{../macros.tex}

\begin{document}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section*{Lecture 25: Symmetric matrices and positive definiteness}

Symmetric matrices are good -- their eigenvalues are real and each has
a complete set of orthonormal eigenvectors.  Positive definite
matrices are even better.

\subsection*{Symmetric matrices}

A {\em symmetric matrix} is one for which $\mb A = \mb A^T$.  If a
matrix has some special property (e.g. it's a Markov matrix), its
eigenvalues and eigenvectors are likely to have special properties as
well.  For a symmetric matrix with real number entries, the
eigenvalues are real numbers and it's possible to choose a complete
set of eigenvectors that are perpendicular (or even orthonormal).

If $\mb A$ has $n$ independent eigenvectors we can write $\mb A = \mb S \mb \Lambda \mb S^{-1}$.  If $\mb A$ is
symmetric we can write $\mb A = \mb Q \mb \Lambda \mb Q^{-1} = \mb Q
\mb \Lambda \mb Q^T$, where $\mb Q$ is an orthogonal matrix.
Mathematicians call this the {\em spectral theorem} and think of the
eigenvalues as the ``spectrum'' of the matrix.  In mechanics it's called
the {\em principal axis theorem}.

In addition, any matrix of the form $\mb Q \mb \Lambda \mb Q^T$ will
be symmetric.

\subsubsection*{Real eigenvalues}

Why are the eigenvalues of a symmetric matrix real?  Suppose $\mb A$
is symmetric and $\mb A \vb x = \lambda \vb x$.  Then we can conjugate
to get $\overline{\mb A} \overline{\vb x} = \overline \lambda \overline{\vb x}$.  If the
entries of $\mb A$ are real, this becomes $\mb A \overline{\vb x} = \overline \lambda
\overline{\vb x}$.  (This proves that complex eigenvalues of real valued matrices come in conjugate pairs.)

Now transpose to get $\overline{\vb x}^T \mb A^T = \overline{\vb x}^T \overline \lambda$.
Because $\mb A$ is symmetric we now have $\overline{\vb x}^T \mb A = \overline
{\vb x}^T \overline \lambda$.  Multiplying both sides of this equation on
the right by $\vb x$ gives:
$$\overline {\vb x}^T \mb A \vb x= \overline {\vb x}^T \overline \lambda \vb x.$$ On the
other hand, we can multiply $\mb A \vb x = \lambda \vb x$ on the left
by $\overline {\vb x}^T$ to get:
$$\overline {\vb x}^T \mb A \vb x = \overline {\vb x}^T \lambda \vb x.$$ Comparing
the two equations we see that $\overline {\vb x}^T \overline \lambda \vb x = \overline {\vb
x}^T \lambda \vb x$ and, unless $\overline {\vb x}^T \vb x$ is zero, we can
conclude $\lambda = \overline \lambda$ is real.

How do we know $\overline {\vb x}^T \vb x \neq 0$?
$$\overline {\vb x}^T \vb x = \onebyfour {\overline {x}_1} {\overline {x}_2} \cdots {\overline
  {x}_n} \fourbyone {x_1} {x_2} \vdots {x_n} = |x_1|^2 + |x_2|^2 +
\cdots + |x_n|^2.$$ If $\vb x \neq \vb 0$ then $\overline {\vb x}^T \vb x
\neq 0$.  

With complex vectors, as with complex numbers, multiplying by the
conjugate is often helpful.

\bigskip

Symmetric matrices with real entries have $\mb A = \mb A^T$, real
eigenvalues, and perpendicular eigenvectors.  If $\mb A$ has complex
entries, then it will have real eigenvalues and perpendicular
eigenvectors if and only if $\mb A = \overline {\mb A}^T$.  (The proof of this follows the same pattern.)

\subsubsection*{Projection onto eigenvectors}

If $\mb A = \mb A^T$, we can write:
\begin{eqnarray*}
\mb A &=& \mb Q \Lambda \mb Q^T\\
&=& \onebyfour {\vb q_1}{\vb q_2}\cdots{\vb q_n}
\left[\begin{array}{cccc} \lambda_1 &&&\\&\lambda_2&&\\&&\ddots&\\&&&\lambda_n
\end{array}\right] \fourbyone {\vb q_1^T}{\vb q_2^T}\vdots{\vb q_n^T}\\
&=& \lambda_1 \vb q_1 \vb q_1^T + \lambda_2 \vb q_2 \vb q_2^T + \cdots + \lambda_n \vb q_n \vb q_n^T\\
\end{eqnarray*}
The matrix $\vb q_k \vb q_k^T$ is the projection matrix onto $\vb
q_k$, so every symmetric matrix is a combination of perpendicular
projection matrices.

\subsubsection*{Information about eigenvalues}

If we know that eigenvalues are real, we can ask whether they are
positive or negative.  (Remember that the signs of the eigenvalues are
important in solving systems of differential equations.)

For very large matrices $\mb A$, it's impractical to compute
eigenvalues by solving $|\mb A - \lambda \mb I| = 0$.  However, it's
not hard to compute the pivots, and the signs of the pivots of a
symmetric matrix are the same as the signs of the eigenvalues:
$$\text{number of positive pivots} = \text{number of positive eigenvalues}.$$

Because the eigenvalues of $\mb A + b \mb I$ are just $b$ more than
the eigenvalues of $\mb A$, we can use this fact to find which
eigenvalues of a symmetric matrix are greater or less than any real
number $b$.  This tells us a lot about the eigenvalues of $\mb A$
even if we can't compute them directly.

\subsection*{Positive definite matrices}

A {\em positive definite matrix} is a symmetric matrix $\mb A$ for
which all eigenvalues are positive.  A good way to tell if a matrix is
positive definite is to check that all its pivots are positive.

Let $\mb A = \small \twobytwo 5223$.  The pivots of this matrix are
$5$ and $(\det \mb A) /5 = 11/5$.  The matrix is symmetric and its
pivots (and therefore eigenvalues) are positive, so $\mb A$ is a
positive definite matrix.  Its eigenvalues are the solutions to:
$$|\mb A - \lambda \mb I| =  \lambda^2 - 8\lambda + 11= 0,$$
i.e. $4 \pm \sqrt{5}$.

The determinant of a positive definite matrix is always positive but
the determinant of $\small \twobytwo {-1}00{-3}$ is also positive, and
that matrix isn't positive definite.  If all of the subdeterminants of $\mb A$ are positive (determinants of the $k$ by $k$ matrices in the upper left corner of $\mb A$, where $1 \leq k \leq n$), then $\mb A$ is
positive definite.

\smallskip

The subject of positive definite matrices brings together what we've
learned about pivots, determinants and eigenvalues of square matrices.
Soon we'll have a chance to bring together what we've learned in
this course and apply it to non-square matrices.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\end{document}