\documentstyle[psbox,ps,11pt,fullpage]{article}

\input epsf

\begin{document}
\header{MAS.622}{4}
\problem{1}

Our observation ${\bf x}$ is the values $y_1, y_2, y_3$  ($y_4$ is uniquely determined, given $N$) and our
parameters ${\bf \theta}$ are $P_1, P_2, P_3$.
\par
Our log likelyhood of observing ${\bf x}$ is therefore:
\begin{equation}
\ln(p({\bf x}|{\bf \theta})) = \Sigma_{i=1}^{N} \ln(i) - \Sigma_{i=1}^{y_1} \ln(i) 
...
- \Sigma_{i=1}^{y_4} \ln(i) + y_1 \ln(P_1) + y_2 \ln(P_2) + y_3 \ln(P_3) + (y_4) \ln(1-P_1-P_2-P_3)
\end{equation}

So, 

\begin{equation}
{\bf \nabla_{\theta}} \ln(p({\bf x}|{\bf \theta})) = \left(\begin{array}{c}\frac{y_1}{P_1} - \frac{y_4}{1-P_1-P_2-P_3}\\\frac{y_2}{P_2} - \frac{y_4}{1-P_1-P_2-P_3}\\\frac{y_3}{P_3} - \frac{y_4}{1-P_1-P_2-P_3}\\\end{array}\right)
\end{equation}

setting this to $0$, and substituing $N = y_1+y+2+y_3+y_4$ we get

\begin{equation}
\hat{P}_i = \frac{y_i}{N}
\end{equation}

\problem{2}

\part{a}

Given a large amount of labeled data, the maximum likelyhood estimate
of $\hat{\mu}$ will be $1$, as the maximum likelyhood estimate of the
mean is the sample mean, and for a large amount of data this will
converge to the true mean.

\part{b}

The decision boundary from this estimate will be at $x = .5$.

\part{c}

Bayes Formula gives us,

\begin{equation}
P(\omega_1|x) = \frac{p(x|\omega_1)}{\Sigma_{j=1}^2 p(x|\omega_j)} = \alpha
\frac{e^{-\frac{1}{2}x^2}}{e^{-\frac{1}{2}x^2} + e^{-\frac{1}{2}(\frac{x-1}{10^6})^2}}
\end{equation}

This yields a new decision boundary of $\pm 5.2562$.  Between these
values, it should classify as $\omega_1$, and outside these values,
$\omega_2$.

\part{d}

A larger $\mu = 10.5$ will cause a decision boundary at the above
value, $x = 5.2562$, which will give a lower error than the maximum
likelyhood classifier.

\part{e}

Knowing the underlying distribution makes a big difference in the
quality and values of a discriminator.


\problem{4}

\part{a}

The maximum likelyhood estimates for $\sigma_a^2$ and $\sigma_b^2$ are
simply the sample variances, which in our particular case are
$\sigma_a^2 = \sigma_b^2 = .25$.  Treating these as the true
variances, the posterior on $\theta$ is

\begin{equation}
p(\theta|{\cal D}) = p(a_1|\theta) p(a_2|\theta) p(b_1|\theta) p(b_2|\theta)
\end{equation}

which appears in Figure \ref{p4a}.

\begin{figure}[hbpt]
\begin{center}
\mbox{\epsfysize=3in \epsfbox{p4a.eps}}
\end{center}
\caption{Posterior distribution of theta, taking the sample variances to be the true variances of the measurements}
\label{p4a}
\end{figure}

\part{b}

We're interested in the posterior on $\theta$, so we want to calculate:

\begin{equation}
\int \int P(\theta, \sigma_a^2, \sigma_b^2 | {\cal D}) {\rm d}\sigma_a^2 {\rm d}\sigma_b^2
\end{equation}

The term we are integrating over can be decomposed by independence
relationship into terms we know:

\begin{equation}
P(\theta, \sigma_a^2, \sigma_b^2 | {\cal D}) \propto P({\cal D}|\theta, \sigma_b^2, \sigma_a^2) P(\theta, \sigma_b^2, \sigma_a^2)\\
\propto P({\cal D}_a|\theta, \sigma_a^2) P(\sigma_a^2|\theta) P({\cal D}_b|\theta, \sigma_b^2) P(\sigma_b^2|\theta) P(\theta)
\end{equation}

This allows us to seperate the integrals as:

\begin{equation}
\int P({\cal D}_a|\theta, \sigma_a^2) \frac{1}{\sigma_a^2} {\rm d}\sigma_a^2
\int P({\cal D}_b|\theta, \sigma_b^2) \frac{1}{\sigma_b^2} {\rm d}\sigma_b^2
\propto T(1.5, .125, 3) T(3.5, .125, 3),
\end{equation}

\part{c}

This posterior appears in Figure \ref{p4c}, and is strikingly
different from the posterior from part a.

\begin{figure}[hbpt]
\begin{center}
\mbox{\epsfysize=3in \epsfbox{p4c.eps}}
\end{center}
\caption{Posterior distribution of theta, using a non imformative prior}
\label{p4c}
\end{figure}


\problem{5}

\part{a}

If we draw two fishes from the pond, they are not at all independent.
Given that we have two experts, one of whom is correct, and we draw a
fish from the pond, the weight of that fish will lend more credence to
one expert or the other.  \par $p(w_2|w_1) \ne p(w_2)$ because $w_2$
is more likely to be similar in weight to $w_1$ than not, given the
stated distribution of fish.

\part{b}

The probabilities for each combination will be calculated in the form:

\begin{equation}
p(c_1={\rm foo}, c_2={\rm bar}) = p(c_1={\rm foo}|e_1)p(c_2={\rm bar}|e_1)p(e_1) + p(c_1={\rm foo}|e_2)p(c_2={\rm bar}|e_2)p(e_2)
\end{equation}


\centerline{
\begin{tabular}{c|c|c|}
& $c_2 = \mbox{carp}$ & $c_2 = \mbox{bass}$ \\
%\cline{2-3}
\hline
$c_1 = \mbox{carp}$ & .0070 & .0160\\
%\cline{2-3}
\hline
$c_1 = \mbox{bass}$ & .0116 & .0029\\
%\cline{2-3}
\hline
\end{tabular}
}

The most probable classification is $c_1 = {\rm carp}$, $c_2 = {\rm
bass}$.  The least probable classification is $c_1 = {\rm bass}$, $c_2 = {\rm
bass}$

\part{c}

If your distribution parameters are estimated from training data, and
then classify new points based on that estimated distribution, you
have potential to get far worse performance than if you classify new
points based on the previous data, rather than an inferred
distribution.

\end{document}
