9.60  Machine-Motivated Human Vision

An example of Deep Dream

As mentioned in class, this music video uses a Deep Dream-like algorithm to certain parts of the video. It's pretty interesting to watch:

https://www.youtube.com/watch?v=dJ1VorN9Cl0

The way it works that a DNN processes an individual frame from the video (which is processed in a series of daisy chained functions, ultimately culminating in labels at the end). Then a human artist selects some neuron in a DNN it would like to optimize for - for example, it looks like the artist chooses a "dog" category neuron for many of the frames.

The human then asks the network to optimize the *pixels* of the image such that the activation for this neuron is increased. This is possible because you can take the derivative of the neuron with respect to the pixels, so he can find out what direction to "move" in pixel space to slightly increase the output of the neuron. This happens a bunch of times.

What results is an altered image that "evolves" dog-like features. It "grows" these features out of parts of the image that sort of look like a dog already (i.e., near a local maxima in pixel space for a dog feature). This is similar to the perception of visual hallucinations, no? To do this for a video, one can apply this process to individual frames independently (as far as I know, this is how it was done. What might be another way to do this?)

This is also how adversarial images are produced. The idea is that you can run just a few steps of the optimization for some category *not* originally in an image. The image doesn't change much, but in practice the output of the neuron changes dramatically, leading to an erroneous prediction.

Announced on 15 March 2019  9:01  a.m. by Michael J Lee

Slides from today uploaded & upcoming tutorials

Hey all, the slides from today are uploaded on Stellar under Materials.

At some point in the coming weeks I'll be uploading tutorials on some things you'll need to do to perform your final project.

I've already uploaded one tutorial on using MATLAB to generate features and perform transfer learning (among other things that probably aren't necessary for your project, like running the "Deep Dream" algorithm on a deep net).

I will also provide more information on using Mechanical Turk soon. The plan is to make sure most of your time on the final project is spent on careful hypothesis generation, experiment design, and data analysis - not messing around with annoying details on Mechanical Turk. So no worries on learning how to use Turk from scratch - stay tuned for more details soon.

Announced on 07 March 2019  2:56  p.m. by Michael J Lee

Welcome - and classroom changes starting next week

Hello all,

Welcome to 9.60! I'm looking forward to a semester of discussing and learning about one of the most exciting perspectives in neuroscience today - the view that the brain (and specifically vision) can be understood as a computing system, and potentially reverse engineered.

In general, feel free to send me a quick email about any issue or question you have with the class (e.g. course expectations, homework help, grading, logistics etc) at mil@mit.edu

As Professor Sinha mentioned today in class, our Thursday class will be held in the same location, 46-3015.

However, starting from Tuesday 2/12/2019, our class will be held in 46-4062.

See you then!

Michael

Announced on 05 February 2019  3:43  p.m. by Michael J Lee