6.835 Intelligent Multimodal User Interfaces
Spring 2015
Instructor: Randall Davis
TAs: Brian W Copeland, Jeremy Kenneth Scott
Lecture: TR11-12.30 (4-231)
Information:
This course will cover the design, implementation, and evaluation of intelligent and multimodal user interfaces. Students will become familiar with the basic technologies for handling speech, vision, and pen-based interaction, and will practice integrating these technologies into multimodal applications.
There are substantial readings from the original literature that go with each lecture. Students will have the opportunity to implement example interfaces in a series of mini-projects, as well as a final project of their own design.
Topics will include:
• Sketch understanding
• Vision-based interaction, including face, emotion, and gesture
recognition
• Speech-based interaction
• Haptic interaction
• Multimodal conversational models
• Multimodal fusion and adaptation algorithms
Prerequisites:
Some serious programming experience is important. You will need to
be able to use Matlab and do some web programming in
JavaScript/HTML, but at a level that you can probably learn during
the course.
This course is distinct in subject matter from 6.831 (User Interface Design and Implementation) and is an appropriate follow-on.
Open to and appropriate for advanced undergraduates and graduate students.
Announcements
Equipment return
Hi all - please remember to bring your Kinect and Leap sensors to class today. I will also be in 32-262 today from 3-4pm accepting equipment returns and ready to help if anyone wants to discuss their final reports.If these times don't work for you, please reach out and arrange an equipment return time with me.
Announced on 14 May 2015 7:53 a.m. by Jeremy Kenneth Scott
Final presentations
Hi all - please send me slides in any format (other than Keynote) by 10am on the day you're presenting. You will have 6 minutes to present.Also, please refer to Prof. Davis' comments on presentation
content on
Piazza:
"And please keep in mind what I said today in class: no need
to remind people what you're working on, beyond a single phrase
(Leap-enabled piano, weight lifting coach, etc.). Focus your talk
on things like: what kind of performance level you were able to
achieve (ie what can it do); what are the limits and why; what did
you get from the user study; and overall what lessons did you
learn?"
Get in touch with any questions and looking forward to hearing about your evaluations!
Announced on 11 May 2015 8:42 a.m. by Jeremy Kenneth Scott
Office hours, evaluations, and grades
Hi all,I'll be holding office hours this Friday from 11am-12pm in 32-262 if anyone wants to talk through their projects. Feel free to reach out before then with questions, as I will not be in class on Thursday.
In case anyone missed the lecture on user study design, here are the slides and the user study template. Remember that we are asking you to perform (1) a performance evaluation and (2) a usability evaluation, if the latter seems to make sense for your project.
Also, here is a pointer to Prof. Miller's content on experimental analysis. The T Test will probably be useful for many of you.
Lastly, I've updated your grades on Gradebook. I'll be
posting comments on your Stellar homework submission pages later
today. Let me know if you have any questions or concerns.
Announced on 06 May 2015 12:40 p.m. by Jeremy Kenneth Scott
Revised description of class Tue May 7 -- What You (Almost) Missed: The Hidden 6.835 Lectu
Announced on 01 May 2015 4:26 p.m. by Randall Davis
About the final presentation and term paper
Presentations and Final Projects: Spring 15
· Projects, papers and slides from your final presentation are due at 6PM on Friday May 15th . Turn in all of them in the usual way on Stellar.
· Projects should be in in a zip file a form that is runnable by us if at all possible, i.e., unless they require specialized hardware other than a Leap or Kinect. In that unlikely case, please consult with us about what to turn in. We’ll work with you to get something reasonable that we can still try out. In all cases be sure to include a README file with instructions for setting up and running your code.
· A crucial thing about the project is what you learned from it, not whether it worked perfectly. Good performance that you can’t explain isn’t worth much, while disappointing performance that arises for an interesting reason can be quite important.
· Content: the paper should cover at least the following items
o What is the task? Motivate it – why is it interesting?
o Describe the system – what does it do? Show a real example of input and output.
o Describe how it works. Note that this should be at the conceptual and architectural level, not at the level of code. Describe what ideas, techniques, and tools you used (e.g., for speech understanding, generation., etc.)
o Describe how well it works. What experiments did you do? What did it do well and what did it do badly? Provide examples of both outstanding performance and underwhelming performance. Do the best you can to explain why it had this performance.
o What in the implementation turned out to be more difficult than expected and why? (e.g., the Leap sensor was insufficiently accurate in some way, which you should describe). What worked easily?
o Describe what you would do next to improve the system, to take its performance to the next level.
· Logistics
o Papers should also be turned in by hardcopy by the same deadline, in Prof. Davis’s office (32-237). We ask you to print it because then each of you does 10 minutes worth of work, rather than the course staff a few hours’ worth of work, plus this avoids the inevitable difficulty of getting fonts and/or figures to print properly.
o Expected length is 10-15 pages, covering the material listed. Papers will be judged on content, insight, etc., not length. If you can be brief and substantive, fine; just make sure you address the issues above. Conversely, 20 rambling pages will not be well received.
o Papers are individual work. Collaboration on the project should be indicated in the write-up: indicate who did what in building and testing the system. But you are expected to do independent write-ups. There will be obvious overlap in content, given that you are describing the same thing, but you should not subdivide the write-up work and each do only part.
· The basic content for both the presentation and the papers is the same. Obviously the talk will cover these briefly, while the paper is expected to go into more detail.
· Given the opportunities you’ve already had for presentations this term, the final presentations should be focused. They should remind people about the task and motivation, and provide a real example, but should focuson what you’ve learned, and any surprises/insights about performance and implementation (as above). A live demo is fine if you want to, but is not required; screenshots or a video are also ok. Explain what you learned, what came out of your user study, etc.
· As noted, after your talk please turn in on Stellar a copy of your slides.
· You must turn in any borrowed hardware to Professor Davis, Jeremy, or the course secretary in person. Replacing these is time consuming and expensive, so we will unabashedly hold your grade hostage until we get your hardware back.
Announced on 28 April 2015 6:58 p.m. by Randall Davis