\documentclass{article}
\input{../macros.tex}

\begin{document}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section*{Lecture 4:  Factorization into $\mb A = \mb L\mb U$}

One goal of today's lecture is to understand Gaussian elimination in terms of matrices; to find a matrix $\mb L$ such that $\mb A = \mb L \mb U$.  We start with some useful facts about matrix multiplication.

\subsection*{Inverse of a product}

The inverse of a matrix product $\mb{AB}$ is $\mb B^{-1} \mb A^{-1}$.

\subsection*{Transpose of a product}

We obtain the {\em transpose} of a matrix by exchanging its rows and
columns.  In other words, the entry in row $i$ column $j$ of $\mb A$
is the entry in row $j$ column $i$ of $\mb A^{T}$.

The transpose of a matrix product $\mb A \mb B$ is $\mb B^{T} \mb
A^{T}$.  For any invertible matrix $\mb A$, the inverse of $\mb A^{T}$
is $\left(\mb A^{-1}\right)^{T}$.

\subsection*{$\mb{A = LU}$}

We've seen how to use elimination to convert a suitable
matrix $\mb A$ into an upper triangular matrix $\mb U$.  This leads to
the factorization $\mb A = \mb L \mb U$, which is very helpful in
understanding the matrix $\mb A$.

Recall that (when there are no row exchanges) we can describe the
elimination of the entries of matrix $\mb A$ in terms of
multiplication by a succession of elimination matrices $\mb E_{ij}$,
so that $\mb A \rightarrow \mb E_{21} \mb A \rightarrow \mb E_{31}\mb
E_{21} \mb A \rightarrow \cdots \rightarrow \mb U$.  In the two by two
case this looks like:
$$\begin{array}{cccc}
\mb E_{21} & \mb A & & \mb U\\
\twobytwo 1 0 {-4} 1 & \twobytwo 2 1 8 7 & = & \twobytwo 2 1 0 3 .
\end{array}$$
We can convert this to a factorization $\mb A = \mb L \mb U$ by
``canceling'' the matrix $\mb E_{21}$; multiply by its inverse to get $\mb E_{21}^{-1} \mb E_{21}
\mb A = \mb E_{21}^{-1} \mb U$.
$$\begin{array}{cccc}
\mb A & & \mb L & \mb U\\
\twobytwo 2 1 8 7 & = & \twobytwo 1 0 4 1 & \twobytwo 2 1 0 3 .
\end{array}$$
The matrix $\mb U$ is upper triangular with pivots on the
diagonal.  The matrix $\mb L$ is {\em lower triangular} and has ones
on the diagonal.  Sometimes we will also want to factor out a diagonal matrix whose entries are the pivots:
$$\begin{array}{ccccc}
\mb A & & \mb L & \mb D & \mb U'\\
\twobytwo 2 1 8 7 & = & \twobytwo 1 0 4 1 & \twobytwo 2 0 0 3 & \twobytwo 1 {1/2} 0 1 .
\end{array}$$

In the three dimensional case, if $\mb E_{32} \mb E_{31} \mb E_{21}
\mb A = \mb U$ then $\mb A = \mb E_{21}^{-1} \mb E_{31}^{-1} \mb
E_{32}^{-1} \mb U = \mb L \mb U$.

For example, suppose $\mb E_{31}$ is the identity matrix and $\mb
E_{32}$ and $\mb E_{21}$ are as shown below:
$$\begin{array}{cccc}
\mb E_{32} & \mb E_{21} & & \mb E\\
\threebythree 1 0 0 0 1 0 0 {-5} 1 & \threebythree 1 0 0 {-2} 1 0 0 0 1 & = & \threebythree 1 0 0 {-2} 1 0 {10} {-5} 1 .
\end{array}$$ 
The $10$ in the lower left corner arises because we subtracted twice the first row from the second row, then subtracted five times the new second row from the third.

The factorization $\mb A = \mb L \mb U$ is preferable to the statement
$\mb E \mb A = \mb U$ because the combination of row subtractions does
not have the effect on $\mb L$ that it did on $\mb E$.  Here $\mb L = \mb E^{-1} = \mb
E_{21}^{-1} \mb E_{32}^{-1}$:
$$\begin{array}{cccc}
\mb E_{21}^{-1} & \mb E_{32}^{-1} & & \mb L\\
\threebythree 1 0 0 2 1 0 0 0 1& \threebythree 1 0 0 0 1 0 0 5 1 & = & \threebythree 1 0 0 2 1 0 0 5 1 .
\end{array}$$
Notice the $0$ in row three column one of $\mb L = \mb E^{-1}$, where $\mb E$ had a $10$.
If there are no row exchanges, the multipliers from the elimination matrices are copied directly into $\mb L$.

\subsection*{How expensive is elimination?}

Some applications require inverting very large matrices.  This is done using a computer, of course.  How hard will the computer
have to work?  How long will it take?

When using elimination to find the factorization $\mb A = \mb L \mb U$
we just saw that we can build $\mb L$ as we go by keeping track of row
subtractions.  We have to remember $\mb L$ and (the matrix which will
become) $\mb U$; we don't have to store $\mb A$ or $\mb E_{ij}$ in
the computer's memory.

How many operations does the computer perform during the
elimination process for an $n \times n$ matrix?  A typical operation
is to multiply one row and then subtract it from another, which
requires on the order of $n$ operations.  There are $n$ rows, so the
total number of operations used in eliminating entries in the first
column is about $n^2$.  The second row and column are shorter; that product costs about $(n-1)^2$
operations, and so on. The total number of operations needed to
factor $\mb A$ into $\mb L \mb U$ is on the order of $n^3$:
$$1^2 + 2^2 + \cdots + (n-1)^2 + n^2 = \sum_{i=1}^n i^2 \approx \int_0^n x^2 \, dx = \frac{1}{3} n^3.$$

While we're factoring $\mb A$ we're also operating on $\vb b$.  That
costs about $n^2$ operations, which is hardly worth counting compared
to $\frac{1}{3} n^3$.

\subsection*{Row exchanges}

What if there are row exchanges?  In other words, what happens if
there's a zero in a pivot position?

To swap two rows, we multiply on the left by a permutation matrix.
For example, $$\mb P_{12} = \threebythree 0 1 0 1 0 0 0 0 1$$ swaps
the first and second rows of a $3 \times 3$ matrix.  The inverse of any
permutation matrix $\mb P$ is $\mb P^{-1} = \mb P^T$.

There are $n!$ different ways to permute the rows of an $n\times n$
matrix (including the permutation that leaves all rows fixed) so
there are $n!$ permutation matrices.  These matrices form a {\em
  multiplicative group}.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\end{document}