Matthew Gray

MAS.622 Problem Set 4 March 17, 1998

Problem 1

Our observation x is the values y_{1}, y_{2}, y_{3} (y_{4} is uniquely determined, given N) and our parameters θ are P_{1}, P_{2}, P_{3}. Our log likelyhood of observing x is therefore:

ln(p(x|θ)) = Σ_{i=1}^{N} ln(i)−Σ_{i=1}^{y_{1}} ln(i)...−Σ_{i=1}^{y_{4}} ln(i)+y_{1} ln(P_{1})+y_{2} ln(P_{2})+y_{3} ln(P_{3})+(y_{4}) ln(1−P_{1}−P_{2}−P_{3})

(1)

So,

∇_{θ} ln(p(x|θ)) = ( \begin{matrix}\frac{y_{1}}{P_{1}} − \frac{y_{4}}{1−P_{1}−P_{2}−P_{3}} \\ \frac{y_{2}}{P_{2}} − \frac{y_{4}}{1−P_{1}−P_{2}−P_{3}} \\ \frac{y_{3}}{P_{3}} − \frac{y_{4}}{1−P_{1}−P_{2}−P_{3}} \end{matrix} )  (2)

setting this to 0, and substituing N = y_{1} + y + 2 + y_{3} + y_{4} we get

P̂_{i} = \frac{y_{i}}{N}  (3)

Problem 2

Part a

Given a large amount of labeled data, the maximum likelyhood estimate of μ̂ will be 1, as the maximum likelyhood estimate of the mean is the sample mean, and for a large amount of data this will converge to the true mean.

Part b

The decision boundary from this estimate will be at x = .5.

Part c

Bayes Formula gives us,

P (ω_{1}|x) = \frac{p(x|ω_{1})}{Σ_{j=1}^{2}p(x|ω_{j})} = α\frac{e^{− \frac{1}{2} x^{2}}}{e^{− \frac{1}{2} x^{2}} + e^{− \frac{1}{2} ( \frac{x−1}{10^{6}} )^{2}}}  (4)

This yields a new decision boundary of ±5.2562. Between these values, it should classify as ω_{1}, and outside these values, ω_{2}.

Part d

A larger μ = 10.5 will cause a decision boundary at the above value, x = 5.2562, which will give a lower error than the maximum likelyhood classifier.

Part e

Knowing the underlying distribution makes a big difference in the quality and values of a discriminator.

Problem 4

Part a

The maximum likelyhood estimates for σ_{a}^{2} and σ_{b}^{2} are simply the sample variances, which in our particular case are σ_{a}^{2} = σ_{b}^{2} = .25. Treating these as the true variances, the posterior on θ is

p(θ|D) = p(a_{1}|θ)p(a_{2}|θ)p(b_{1}|θ)p(b_{2}|θ)  (5)

which appears in Figure 1.

Figure 1: Posterior distribution of theta, taking the sample variances to be the true variances of the measurements

Part b We’re interested in the posterior on θ, so we want to calculate:

∫ ∫ P (θ, σ_{a}^{2}, σ_{b}^{2}|D)dσ_{a}^{2}dσ_{b}^{2}  (6)

The term we are integrating over can be decomposed by independence relationship into terms we know:

P (θ, σ_{a}^{2}, σ_{b}^{2}|D) ∝ P (D|θ, σ_{b}^{2}, σ_{a}^{2})P (θ, σ_{b}^{2}, σ_{a}^{2}) ∝ P (D_{a}|θ, σ_{a}^{2})P (σ_{a}^{2}|θ)P (D_{b}|θ, σ_{b}^{2})P (σ_{b}^{2}|θ)P (θ) (7)

This allows us to seperate the integrals as:

∫ P (D_{a}|θ, σ_{a}^{2})\frac{1}{σ_{a}^{2}} dσ_{a}^{2} ∫ P (D_{b}|θ, σ_{b}^{2})\frac{1}{σ_{b}^{2}} dσ_{b}^{2} ∝ T (1.5, .125, 3)T (3.5, .125, 3),  (8)

Part c

This posterior appears in Figure 2, and is strikingly different from the posterior from part a.

Figure 2: Posterior distribution of theta, using a non imformative prior

Problem 5

Part a

If we draw two fishes from the pond, they are not at all independent. Given that we have two experts, one of whom is correct, and we draw a fish from the pond, the weight of that fish will lend more credence to one expert or the other. p(w_{2}|w_{1}) ≠ p(w_{2}) because w_{2} is more likely to be similar in weight to w_{1} than not, given the stated distribution of fish.

Part b

The probabilities for each combination will be calculated in the form:

p(c_{1} = foo, c_{2} = bar) = p(c_{1} = foo|e_{1})p(c_{2} = bar|e_{1})p(e_{1}) + p(c_{1} = foo|e_{2})p(c_{2} = bar|e_{2})p(e_{2}) (9)
c_{2} = carp c_{2} = bass
c_{1} = carp  .0070  .0160
c_{1} = bass  .0116  .0029

The most probable classification is c_{1} = carp, c_{2} = bass. The least probable classification is c_{1} = bass, c_{2} = bass

Part c

If your distribution parameters are estimated from training data, and then classify new points based on that estimated distribution, you have potential to get far worse performance than if you classify new points based on the previous data, rather than an inferred distribution.