6.804/9.66/9.660  Computational Cognitive Science

Hi all,

I'll be holding extra office hours tomorrow, at 10 am, in 68-253.

 --Yarden

Hi all,

Sorry for the short notice - but I have to move my recitation.  Instead of tonight, I'll be holding extra office hours in addition to yesterday's on Friday morning, 10 am. Also, remember that Steve's recitation will be held tomorrow. Please email us if you have any questions or would like to discuss your final projects.
Thanks, --Yarden

Time for the pesky e-mails!

Please take a few moments and fill out your on line course evals for 9.66 and your other eligible classes, we really do care and read your comments.

BCS wants to hear it all, the good, the bad and the ugly!

Hi all,

We wanted to clarify the finite mixture part of pset 5. Recall that the Gibbs sampling scheme for any mixture model looks like this:

for every data point n
  for every cluster k
    compute the probability of reassigning point n to cluster k given data
    sample reassignment of point n to a cluster with this probability

In the infinite mixture model, you have to consider the K clusters that exist in the current iteration of the sampler, as well the K+1 cluster -- i.e., the probability of reassigning a point to a new cluster. In the finite case, you only consider K many clusters, where K is fixed and known in advance.

The probability of reassigning point n to cluster k given data can be written using Bayes rule, as a product of the likelihood (probability of point n given its assignment to cluster k) and the prior (prior probability of assigning point n to to cluster k).
In both the infinite and finite mixture cases, the likelihood is the same -- it is simply the familiar Beta-Bernoulli likelihood (integrating out the Bernoulli parameter theta) that was covered in class and that you derived on pset 4.

The priors for the infinite and finite mixture cases are different, however. In the infinite mixture case, the prior is the CRP, as shown in the lecture notes.

In the finite mixture case, the prior is different. Recall that the prior is p(reassigning point n to cluster k). To compute this probability, we have to marginalize the mixing weights of the k clusters. If the vector of mixing weights is theta, this probability is:

p(reassigning point n to cluster k) = integral over theta[ p(assigning point n to cluster k | theta)p(theta) ]

This integral is similar to the integral you computed in pset 4 where the coin weight of a Beta-Bernoulli model is integrated out.  Except in this case, we are doing it in the multivariate version of Beta-Bernoulli, which is the Dirichlet-Multinomial model (our prior on the mixing weights p(theta) is a Dirichlet distribution.)

The analytic solution of this integral is described in a technical report by Griffiths and Ghahramani, which was posted to the Stellar materials section on infinite mixture models. (The relevant equation is equation (10)).

Hi all,

In light of thanksgiving, there won't be a recitation tonight -- if you have questions about the pset, please email the TAs.  I will be available to meet with people who have questions this Friday as well -- just email to setup a time.

Thanks, --Yarden