6.870  Multimodal User Interfaces

This is an instance of the Advanced Topics in Computer Vision series.

Prerequisites: 6.034
6.170 will be useful but other programming experience is a satisfactory alternative. This course is distinct in subject matter from 6.831 (User Interface Design and Implementation) and is an appropriate follow-on. This course has revised content and may be repeated for credit for students who have previously taken 6.870. (The above pre-reqs are correct; the listing in the course catalog is out of date.)

This course will cover the design, implementation, and evaluation of intelligent and multi-modal user interfaces. It will involve a combination of papers from the original literature and hands-on exercises to familiarize students with the basic technologies for handling speech, vision, and pen-based interaction, and enable them to explore architectures for combining modalities in a variety of ways.

There are substantial readings from original literature that go with each lecture; students will be expected to summarize and present a significant paper, and will have the opportunity to implement example interfaces in a series of course projects.

Specific topics will be chosen by the instructors based on class interest and may include:

• Gaze tracking and gaze-aware interfaces
• Gesture recognition
• Mobile image-based interfaces
• Scene context recognition
• Multimodal conversational models
• Audiovisual speech recognition
• Pen-based interaction
• Perceptually grounded human-robot interaction
• Multimodal fusion and adaptation algorithms
• Methods for combining channels and/or mutual disambiguation