\documentclass{article}
\input{../macros.tex}

\begin{document}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section*{An overview of key ideas}

This is an overview of linear algebra given at the start of a course
on the mathematics of engineering.

\bigskip

Linear algebra progresses from vectors to matrices to subspaces.

\subsection*{Vectors}

What do you do with vectors?  Take combinations.

We can multiply vectors by scalars, add, and subtract.  Given vectors
$\vb u$, $\vb v$ and $\vb w$ we can form the {\em linear combination }
$x_1 \vb u + x_2 \vb v + x_3 \vb w = \vb b$.

An example in $\R^3$ would be:
$$\vb u = \threebyone 1{-1}0, \vb v = \threebyone 01{-1}, \vb w =
\threebyone 001.$$ The collection of all multiples of $\vb u$ forms a line through the origin.  The collection
of all multiples of $\vb v$ forms another line.  The collection of all
combinations of $\vb u$ and $\vb v$ forms a plane.  Taking {\em all combinations} of some vectors creates a {\em subspace}.

We could continue like this, or we can use a matrix to add in all
multiples of $\vb w$.  

\subsection*{Matrices}

Create a matrix $\mb A$ with vectors $\vb u$, $\vb v$ and $\vb w$ in
its columns:
$$\mb A = \threebythree 100{-1}100{-1}1.$$
The product:
$$\mb A \vb x = \threebythree 100{-1}100{-1}1 \threebyone
{x_1}{x_2}{x_3} = \threebyone{x_1}{-x_1+x_2}{-x_2+x_3}$$ equals the
sum $x_1 \vb u + x_2 \vb v + x_3 \vb w = \vb b$.  The product of a
matrix and a vector is a combination of the columns of the matrix.
(This particular matrix $\mb A$ is a {\em difference matrix} because
the components of $\mb A \vb x$ are differences of the
components of that vector.)

When we say $x_1 \vb u + x_2 \vb v + x_3 \vb w = \vb b$ we're thinking
about multiplying numbers by vectors; when we say $\mb A \vb x = \vb
b$ we're thinking about multiplying a matrix (whose columns are $\vb u$, $\vb v$ and $\vb w$) by the numbers.  The calculations are the same, but our
perspective has changed.

For any input vector $\vb x$, the output of the operation
``multiplication by $\mb A$'' is some vector $\vb b$:
$$\mb A \threebyone 149 = \threebyone 135.$$
A deeper question is to start with a vector $\vb b$ and ask ``for what
vectors $\vb x$ does $\mb A\vb x = \vb b$?''  In our example, this
means solving three equations in three unknowns.  Solving:
$$\mb A \vb x = \threebythree 100{-1}100{-1}1 \threebyone
{x_1}{x_2}{x_3} = \threebyone{x_1}{x_2-x_1}{x_3-x_2} = \threebyone
{b_1}{b_2}{b_3}$$
is equivalent to solving:
\begin{eqnarray*}
x_1 &=& b_1\\
x_2-x_1 &=& b_2\\
x_3-x_2 &=& b_3.
\end{eqnarray*}
We see that $x_1 = b_1$ and so $x_2$ must equal $b_1 + b_2$.  In
vector form, the solution is:
$$\threebyone {x_1}{x_2}{x_3} = \threebyone {b_1}{b_1+b_2}{b_1+b_2+b_3}.$$
But this just says:
$$\vb x = \threebythree 100110111 \threebyone {b_1}{b_2}{b_3},$$ or
$\vb x = \mb A^{-1} \vb b$.  If the matrix $\mb A$ is invertible, we
can multiply on both sides by $\mb A^{-1}$ to find the unique solution
$\vb x$ to $\mb A \vb x = \vb b$.  We might say that $\mb A$
represents a transform $\vb x \rightarrow \vb b$ that has an inverse
transform $\vb b \rightarrow \vb x$.

In particular, if $\vb b = \small \threebyone 000$ then $\vb x =
\small \threebyone 000$.

\bigskip

The second example has the same columns $\vb u$ and $\vb v$ and
replaces column vector $\vb w$:
$$\mb C = \threebythree 1 0 {-1} {-1} 1 0 0 {-1} 1.$$
Then:
$$\mb C \vb x = \threebythree 10{-1}{-1}100{-1}1 \threebyone
{x_1}{x_2}{x_3} = \threebyone{x_1-x_3}{x_2-x_1}{x_3-x_2}$$ and our
system of three equations in three unknowns becomes circular.

Where before $\mb A \vb x = \vb 0$ implied $\vb x = \vb 0$, there are
non-zero vectors $\vb x$ for which $\mb C \vb x = \vb 0$.  For any
vector $\vb x$ with $x_1 = x_2 = x_3$, $\mb C \vb x = \vb 0$.  This is
a significant difference; we can't multiply both sides of $\mb C \vb x
= \vb 0$ by an inverse to find a non-zero solution $\vb x$.

The system of equations encoded in $\mb C \vb x = \vb b$ is:
\begin{eqnarray*}
x_1-x_3 &=& b_1\\
x_2-x_1 &=& b_2\\
x_3-x_2 &=& b_3.
\end{eqnarray*}
If we add these three equations together, we get:
$$0 = b_1 + _2 + b_3.$$ This tells us that $\mb C \vb x = \vb b$ has a
solution $\vb x$ only when the components of $\vb b$ sum to $0$.  In a
physical system, this might tell us that the system is stable as long
as the forces on it are balanced.

\subsection*{Subspaces}

Geometrically, the columns of $\mb C$ lie in the same plane (they are
{\em dependent}; the columns of $\mb A$ are {\em independent}).  There
are many vectors in $\R^3$ which do not lie in that plane.  Those
vectors cannot be written as a linear combination of the columns of
$\mb C$ and so correspond to values of $\vb b$ for which $\mb C \vb x
= \vb b$ has no solution $\vb x$.  The linear combinations of the
columns of $\mb C$ form a two dimensional {\em subspace} of $\R^3$.  

This plane of combinations of $\vb u$, $\vb v$ and $\vb w$ can be
described as ``all vectors $\mb C \vb x$''.  But we know that the
vectors $\vb b$ for which $\mb C \vb x = \vb b$ satisfy the condition
$b_1 + b_2 + b_3 = 0$.  So the plane of all combinations of $\vb u$
and $\vb v$ consists of all vectors whose components sum to $0$.

If we take all combinations of:
$$\vb u = \threebyone 1{-1}0, \vb v = \threebyone 01{-1}, \mbox{ and }
\vb w = \threebyone 001$$ we get the entire space $\R^3$; the equation
$\mb A \vb x = \vb b$ has a solution for every $\vb b$ in $\R^3$.  We
say that $\vb u$, $\vb v$ and $\vb w$ form a {\em basis} for $\R^3$.

 A {\em basis} for $\R^n$ is a collection of $n$ independent vectors in $\R^n$.  Equivalently, a basis is a collection of $n$ vectors whose combinations cover
the whole space.  Or, a collection of vectors forms a basis whenever a
matrix which has those vectors as its columns is invertible.

A {\em vector space} is a collection of vectors that is closed under
linear combinations.  A {\em subspace} is a vector space inside
another vector space; a plane through the origin in $\R^3$ is an
example of a subspace.  A subspace could be equal to the space it's
contained in; the smallest subspace contains only the zero vector.

The subspaces of $\R^3$ are:
\begin{itemize}
\item the origin,
\item a line through the origin,
\item a plane through the origin,
\item all of $\R^3$.
\end{itemize}

\subsection*{Conclusion}

When you look at a matrix, try to see ``what is it doing?''

Matrices can be rectangular; we can have seven equations in three unknowns.  Rectangular matrices are not invertible,
but the symmetric, square matrix $\mb A^T \mb A$ that often appears
when studying rectangular matrices may be invertible.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\end{document}