\documentclass{article}
\input{../macros.tex}

\begin{document}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section*{Lecture 2: Elimination with matrices}

\subsection*{Method of Elimination}

{\em Elimination} is the technique most commonly used by
computer software to solve systems of linear equations.  It finds a solution $\vb x$ to  ${\mb A} {\vb x} = \vb b$ whenever the matrix $\mb A$ is
invertible.  In the example used in class,
$$\mb A = \threebythree 1 2 1 3 8 1 0 4 1 \mbox{ and } \vb b = \threebyone {2}{12}{2}.$$

The number $1$ in the upper left corner of $\mb A$ is called the {\em
  first pivot}.  We recopy the first row, then multiply the numbers in
it by an appropriate value (in this case $3$) and subtract those
values from the numbers in the second row.  The first number in
the second row becomes $0$.  We have thus {\em eliminated} the $3$ in row 2 column 1.

The next step is to perform another elimination to get a 0 in row 3 column 1; here this is already the case.

The {\em second pivot} is the value $2$ which now appears in row 2 column
2.  We find a multiplier (in this case $2$) by which we multiply the 
second row to
eliminate the $4$ in row 3 column 2.  The {\em third pivot} is
then the $5$ now in row 3 column 3.

We started with an invertible matrix $\mb A$ and ended with an {\em
  upper triangular} matrix $\mb U$; the lower left portion of $\mb U$
is filled with zeros.  Pivots 1, 2, 5 are on the diagonal of $\mb U$.

$$\mb A = \threebythree 1 2 1 3 8 1 0 4 1 \longrightarrow \threebythree 1 2 1 0 2 {-2} 0 4 1 \longrightarrow \mb U = \threebythree 1 2 1 0 2 {-2} 0 0 5$$

We repeat the multiplications and subtractions with the 
vector $\vb b\nolinebreak[4]=\nolinebreak[4]\small \threebyone 2{12}2$.  For example, we multiply the 2 in the first position by 3 and subtract from 12 to get 6
in the second position.  When calculating by hand we can do this efficiently by {\em augmenting} the
matrix ${\mb A}$, appending the vector $\vb b$ as a fourth or final
column.  The method of elimination transforms the equation $\mb A \vb
x = \vb b$ into a new equation $\mb U
\vb x = \vb c$. In the example above,
$\mb U = \small \threebythree 1 2 1 0 2 {-2} 0 0 5$ comes from $\mb A$ and $\vb c = \small \threebyone {2}{6}{-10}$ comes from $\vb b.$

The equation $\mb U \vb x = \mb c$ is easy to solve by {\em back
  substitution}; in our example, $z=-2$, $y=1$ and $x = 2$.  This is
also a solution to the original system $\mb A \vb x = \vb b$.

\bigskip

The {\em determinant} of $\mb U$ is the product of the pivots.  We will see this again.

Pivots may not be 0.  If there is a zero in the pivot position, we
must exchange that row with one below to get a non-zero value in
the pivot position.

If there is a zero in the pivot position and no non-zero value below
it, then the matrix $\mb A$ is not invertible.  Elimination can not be used to find a unique solution to the
system of equations -- it doesn't exist.

\subsection*{Elimination Matrices}

The product of a matrix (3x3) and a column vector (3x1) is a column vector (3x1) that is a linear combination of the columns of the matrix.

The product of a row (1x3) and a matrix (3x3) is a row (1x3) that is a linear combination of the rows of the matrix.

\smallskip

We can subtract 3 times row 1 of matrix $\mb A$ from row 2 of $\mb A$ by calculating the matrix product:
$$\threebythree 1 0 0 {-3} 1 0 0 0 1 \threebythree 1 2 1 3 8 1 0 4 1 = \threebythree 1 2 1 0 2 {-2} 0 4 1.$$
The {\em elimination matrix} used to eliminate the entry in row $m$ column $n$ is denoted $\mb{E_{mn}}$.
The calculation above took us from $\mb A$ to $\mb E_{21} \mb A$.  The three elimination steps leading to $\mb U$ were: $\mb{ E_{32} (E_{31} (E_{21} A)) = U}$, where $\mb{E_{31} = I}$.  Thus $\mb E_{32}(\mb E_{21} \mb A) = \mb U$.

Matrix multiplication is {\em associative}, so we can also write $\mb{ (E_{32}E_{21}) A = U}$.  The product $\mb{E_{32} E_{21}}$ tells us how to get from $\mb A$ to $\mb U$.  The {\em inverse} of the matrix $\mb{E_{32} E_{21}}$ tells us how to get from $\mb U$ to $\mb A$.

If we solve $\mb U \vb x = \mb E \mb A \vb x
= \mb E \vb b$, then it is also true that $\mb A \vb x = \vb b$.  This
is why the method of elimination works:  all steps can be reversed.

\smallskip

A {\em permutation matrix} exchanges two rows of a matrix; for example,
$$\mb P = \threebythree 0 1 0 1 0 0 0 0 1.$$
The first and second rows of the matrix $\mb{PA}$ are the second and first rows of the matrix $\mb A$.  The matrix $\mb P$ is constructed by exchanging rows of the identity matrix.

To exchange the columns of a matrix, multiply on the right (as in $\mb{A P}$) by a permutation matrix.

Note that matrix multiplication is not {\em commutative}:  $\mb{PA \neq AP}$.

\subsection*{Inverses}

We have a matrix:
$$\mb E_{21}  = \threebythree 1 0 0 {-3} 1 0 0 0 1$$
which subtracts 3 times row 1 from row 2.  To ``undo'' this operation we must add 3 times row 1 to row 2 using the inverse matrix:
$$\mb E_{21}^{-1} = \threebythree 1 0 0 3 1 0 0 0 1.$$
In fact, $\mb{E_{21}^{-1} E_{21} = I}$.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\end{document}