6.835  Intelligent Multimodal User Interfaces

6.170 will be useful but other programming experience is a satisfactory alternative. This course is distinct in subject matter from 6.831 (User Interface Design and Implementation) and is an appropriate follow-on.
 
Units: 3-0-9
Open to and appropriate for advanced undergraduates and graduate students.
 
This course will cover the design, implementation, and evaluation of intelligent and multi-modal user interfaces.  It will involve a combination of papers from the original literature and hands-on exercises to familiarize students with the basic technologies for handling speech, vision, and pen-based interaction, and enable them to explore architectures for combining modalities in a variety of ways.
 
There are substantial readings from original literature that go with each lecture. Students will have the opportunity to implement example interfaces in a series of course mini-projects, as well as a final project of their own design.
 
Topics will include: 
 
    • Gaze tracking and gaze-aware interfaces
    • Gesture recognition
    • Mobile image-based interfaces
    • Multimodal conversational models 
    • Audiovisual speech recognition
    • Pen-based interaction
    • Multimodal fusion and adaptation algorithms
    • Methods for combining channels and/or mutual disambiguation