\documentclass{article}
\input{../macros.tex}

\begin{document}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section*{Lecture 33: Left and right inverses; pseudoinverse}

Although pseudoinverses will not appear on the exam, this lecture will
help us to prepare.

\subsection*{Two sided inverse}

A {\em 2-sided inverse} of a matrix $\mb A$ is a matrix $\mb A^{-1}$
for which $\mb A \mb A^{-1} = \mb I = \mb A^{-1} \mb A$.  This is what
we've called the {\em inverse} of $\mb A$.  Here $r= n =
m$; the matrix $\mb A$ has full rank.

\subsection*{Left inverse}

Recall that $\mb A$ has full column rank if its columns are
independent; i.e. if $r=n$.  In this case the nullspace of $\mb A$
contains just the zero vector.  The equation $\mb A \vb x = \vb b$
either has exactly one solution $\vb x$ or is not solvable.  

The matrix $\mb A^T \mb A$ is an invertible $n$ by $n$ symmetric
matrix, so $(\mb
A^T A)^{-1} \mb A^T \mb A = \mb I$.  We say $\mb A^{-1}_{\text{left}} = (\mb A^T A)^{-1} \mb A^T$
 is a {\em left inverse} of $\mb A$.  (There may be other left
inverses as well, but this is our favorite.)  The fact that $\mb A^T \mb A$ is
invertible when $\mb A$ has full column rank was central to our
discussion of least squares.

Note that $\mb A \mb A^{-1}_{\text{left}}$ is an $m$ by $m$ matrix
which only equals the identity if $m=n$.  A rectangular matrix can't
have a two sided inverse because either that matrix or its transpose
has a nonzero nullspace.

\subsection*{Right inverse}

If $\mb A$ has full row rank, then $r = m$.  The nullspace of $\mb
A^T$ contains only the zero vector; the rows of $\mb A$ are
independent.  The equation $\mb A \vb x = \vb b$ always has at least
one solution; the nullspace of $\mb A$ has dimension $n-m$, so there
will be $n-m$ free variables and (if $n>m$) infinitely many
solutions!

Matrices with full row rank have right inverses $\mb
A^{-1}_{\text{right}}$ with $\mb A \mb A^{-1}_{\text{right}} = \mb I$.  The nicest one of these is $\mb A^T(\mb A\mb
A^T)^{-1}$.  Check:  $\mb A$ times $\mb A^T (\mb A \mb AT)^{-1}$ is
$\mb I$.

\subsection*{Pseudoinverse}

An invertible matrix ($r=m=n$) has only the zero vector in its
nullspace and left nullspace.  A matrix with full column rank $r=n$
has only the zero vector in its nullspace.  A matrix with full row
rank $r=m$ has only the zero vector in its left nullspace.  The
remaining case to consider is a matrix $\mb A$ for which $r < n$ and
$r < m$.

If $\mb A$ has full column rank and $\mb A^{-1}_{\text{left}} = (\mb
A^T \mb A)^{-1}\mb A^T$, then $$\mb A \mb A^{-1}_{\text{left}} = \mb A
(\mb A^T \mb A)^{-1} \mb A^T = \mb P$$ is the matrix which projects
$\R^m$ onto the column space of $\mb A$.  This is as close as we can get to the product $\mb A \mb M = \mb I$.

Similarly, if $\mb A$ has full row rank then $\mb
A^{-1}_{\text{right}}\mb A = \mb A^T(\mb A\mb A^T)^{-1} \mb A$ is the
matrix which projects $\R^n$ onto the row space of $\mb A$.

\smallskip

It's nontrivial nullspaces that cause trouble when we try to invert
matrices.  If $\mb A \vb x = \vb 0$ for some nonzero $\vb x$, then
there's no hope of finding a matrix $\mb A^{-1}$ that will reverse
this process to give $\mb A^{-1} \vb 0 = \vb x$.

\bigskip

The vector $\mb A \vb x$ is always in the column space of $\mb A$.  In
fact, the correspondence between vectors $\vb x$ in the ($r$
dimensional) row space and vectors $\mb A \vb x$ in the ($r$
dimensional) column space is one-to-one.
%On the other hand, if $\vb x$ is in the nullspace of $\mb A$, then
%$\mb A \vb x = \vb 0$.  
In other words, if $\vb x \neq \vb y$ are
vectors in the row space of $\mb A$ then $\mb A \vb x \neq \mb
A \vb y$ in the column space of $\mb A$.  (The proof of this would
make a good exam question.)

\subsubsection*{Proof that if $\vb x \neq \vb y$ then $\mb A \vb x \neq \mb
A \vb y$}

Suppose the statement is false.  Then we can find $\vb x \neq \vb y$
in the row space of $\mb A$ for which $\mb A \vb x = \mb A \vb y$.
But then $\mb A (\vb x - \vb y) = \vb 0$, so $\vb x -\vb y$ is in the nullspace of $\mb A$.  But the row space of
$\mb A$ is closed under linear combinations (like subtraction), so
$\vb x - \vb y$ is also in the row space.  The only vector in both the
nullspace and the row space is the zero vector, so $\vb x- \vb y = \vb
0$.  This contradicts our assumption that $\vb x$ and $\vb y$ are not
equal to each other.

We conclude that the mapping $\vb x \mapsto \mb A \vb x$ from row
space to column space is
invertible.  The inverse of this operation is called the {\em
  pseudoinverse} and is very useful to statisticians in their work
with linear regression -- they might not be able to guarantee that
their matrices have full column rank $r=n$.

\subsubsection*{Finding the pseudoinverse $\mb A^+$}

The {\em pseudoinverse} $\mb A^+$ of $\mb A$ is the matrix for which
$\vb x = \mb A^+ \mb A \vb x$ for all $\vb x$ in the row space of $\mb
A$.  The nullspace of $\mb A^+$ is the nullspace of $\mb A^T$.

We start from the singular value decomposition $\mb A = \mb U \mb \Sigma
\mb V^T$.  Recall that $\Sigma$ is a $m$ by $n$  matrix whose entries are zero except for the singular values $\sigma_1, \sigma_2, ..., \sigma_r$ which appear on the diagonal of its
first $r$ rows.  The matrices $\mb U$ and $\mb V$ are orthonormal and
therefore easy to invert.  We only need to find a pseudoinverse for
$\mb \Sigma$.

The closest we can get to an inverse for $\mb \Sigma$ is an $n$ by $m$
 matrix $\mb \Sigma^+$ whose first $r$ rows have $1/\sigma_1, 1/\sigma_2, ..., 1/\sigma_r$ on the diagonal.  If $r=n =m$ then
$\mb \Sigma^+ = \mb \Sigma^{-1}$.  Always, the product of $\mb
\Sigma$ and $\mb \Sigma^+$ is a square matrix whose first $r$ diagonal
entries are $1$ and whose other entries are $0$.

If $\mb A = \mb U \mb \Sigma \mb V^T$ then its pseudoinverse is $\mb
A^+ = \mb V \Sigma^+ \vb U^T$.  (Recall that $\mb Q^T = Q^{-1}$ for
orthogonal matrices $\mb U$, $\mb V$ or $\mb Q$.)

We would get a similar result if we included non-zero entries in the
lower right corner of $\mb \Sigma^+$, but we prefer not to have extra
non-zero entries.

\subsection*{Conclusion}

Although pseudoinverses will not appear on the exam, many of the
topics we covered while discussing them (the four subspaces, the SVD,
orthogonal matrices) are likely to appear.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\end{document}