Spring 2017, CS-180H: Greensheet

San José State University
College of Science / Department of Computer Science
CS-180H, Individual Studies for Honors, Section 1, Spring 2017

Course and Contact Information

Instructor:	Robert Bruce
Office Location:	Engineering, ENGR-246
Telephone:
Email:
Office Hours:	Tuesdays 4pm-6pm, or by appointment
Class Days/Time:	Thursdays, 2pm-4pm
Classroom:	ENGR-246
Prerequisites:	At least junior standing as computer science major. GPA of 3.5 or higher in the major and department chair consent.

Course Description

Senior project on advanced topics in computer science as determined by the instructor. Written paper or oral presentation of the project required. Intended for students graduating with departmental honors.

Project title

Real-time motion capture applied to 3D virtual model

Project description

In this project, the student will develop a markerless motion capture system to project facial expressions onto a virtual 3D animated avatar in real-time. This project requires knowledge of systems programming, 2D and 3D computer graphics, Maya animation/modelling software, and computer vision.

Learning Outcomes

Upon successful completion of this course, students will be able to:

Course Learning outcome: Use the Active Appearance Modelling (AAM) algorithm from OpenFace to locate and project 2D facial features onto a 3D animated avatar in Maya in real-time.
Course Learning outcome: Use the C++ Maya plugin Application Programmer’s Interface (API) for direct manipulation and access to 3D computer graphics model models in Maya.
Course Learning outcome: Use shared memory and semaphores for synchronized interprocess communication (i.e. avoiding race conditions) between OpenFace and the Maya API.
Course Learning outcome: Creating a real-time markerless motion capture system.
Course Learning outcome: Apply software engineering principles through the entire development lifecycle from design, implementation, testing, release, and maintenance.

Required Texts/Readings

Textbook

Complete Maya Programming (Volume I) by David A. D. Gould. ISBN: 978-1558608351. A copy of this book is available in Lab-246 (ENGR-246).

Complete Maya Programming (Volume II): An In-depth Guide to 3D Fundamentals, Geometry, and Modeling by David Gould. ISBN: 978-0120884827. A copy of this book is available in Lab-246 (ENGR-246).

Advanced programming in the UNIX environment by W. Richard Stevens and Stephen A. Rago. A copy of this book is available in Lab-246 (ENGR-246).

Course Requirements and Assignments

SJSU classes are designed such that in order to be successful, it is expected that students will spend a minimum of forty-five hours for each unit of credit (normally three hours per unit per week), including preparing for class, participating in course activities, completing assignments, and so on. More details about student workload can be found in University Policy S12-3 at http://www.sjsu.edu/senate/docs/S12-3.pdf.

Note that University policy F69-24 at http://www.sjsu.edu/senate/docs/F69-24.pdf states that "Students should attend all meetings of their classes, not only because they are responsible for material discussed therein, but because active participation is frequently essential to insure maximum benefit for all members of the class. Attendance per se shall not be used as a criterion for grading."

Deliverables

Foundation OpenFace / Maya relationship	Establish shared memory OpenFace dumps 68 data points to shared memory Interprocess Communication (IPC) Maya plug-in (IPC)
Stage 2 Rigging in Maya	Develop fully rigged model Develop expression-driven rigs (Cheeks move when mouth corners move)
Stage 3 Maya plugin core	Manipulating bones using plugin Correcting for head roll to get true horizontal/vertical delta distances Apply delta distances to bones
Stage 4 User Interface	Start/stop bone manipulation Choose bones to be manipulated/mapped, assign OpenFace data to map to which bone

Grading Policy

This class is graded credit / no credit.

Grading Rubric

Deliverable	10 points	18 points	25 points
Foundation	Establishes shared memory. Writes 68 data points into memory.	Establishes shared memory. Locks shared memory using mutex or semaphore. Writes 68 data points into shared memory. Frees memory lock using mutex or semaphore after writing 68 data points.	Establishes shared memory. Locks shared memory using mutex or semaphore. Writes 68 data points into shared memory. Frees memory lock using mutex or semaphore after writing 68 data points.
Stage 2	Develops partially rigged model to manipulate skull in yaw, pitch, and roll.	Develops fully rigged model to manipulate skull (yaw, pitch, roll), eyes (pupils), eyebrows, nose, mouth (lips and jaw).	Develops fully rigged model to manipulate skull (yaw, pitch, roll), eyes (pupils), eyebrows, nose, mouth (lips and jaw). Develops expression-driven rigs for left and right cheeks.
Stage 3	Compute delta distance for yaw, pitch, roll from previous to current frame. Compute delta distances for 68-data points from previous to current frame.	Apply delta distance for yaw, pitch, and roll to skull rig. Apply delta distance from 68 data points in OpenFace to eyebrows, eyelids, and eyes.	Apply delta distance for yaw, pitch, and roll to skull rig. Apply delta distances from 68 data points in OpenFace to eyebrows, eyelids, eyes, upper lip, lower lip, jaw. Drive left and right cheeks from delta distances applied to mouth corners (expression-driven rig).
Stage 4	Add UI control buttons to start/stop bone manipulation. Buttons have no functionality yet.	Add UI control to record (start/stop) delta distances applied to rig for skull (yaw, pitch, roll).	Add UI control to record (start/stop) delta distances applied to rig for skull (yaw, pitch, roll), eyebrows, eyes, upper/lower mouth, jaw, and cheeks.

Total possible points for this project is 100. To receive credit for this class one needs to score 70 points or above.

Classroom Protocol

Regular class attendance is highly recommended. Students are responsible for knowing all materials covered through in-class lectures and assigned readings. Please be mindful of fellow students and the instructor by not talking on mobile phones during instruction. Students are expected to leave the class quietly in the event they must use their mobile phones.

University Policies

Per University Policy S16-9, university-wide policy information relevant to all courses, such as academic integrity, accommodations, etc. will be available on Office of Graduate and Undergraduate Programs' Syllabus Information web page at http://www.sjsu.edu/gup/syllabusinfo/

CS-180H / Individual Studies for Honors, Spring 2017, Course Schedule

Course Schedule

Week	Date	Topics, Readings, Assignments, Deadlines
1	Thursday, January 26	Lecture: Overview for building a real-time motion capture pipeline
1	Thursday, February 2	Lecture: Shared memory, Semaphores, and Inter-process communication
2	Thursday, February 9	Lecture: Introduction to Pthreads (multi-threading for speed)
2	Thursday, February 16	Lecture: Mapping 2D shapes onto a 3D avatar
3	Thursday, February 23	Lecture: Introduction (overview) of the Maya C++ API
3	Thursday, March 2	Lecture: Dissection of a boilerplate (minimal code) for a Maya C++ plugin.
4	Thursday, March 9	Lecture: Mesh manipulation (vertices) in the Maya C++ API
4	Thursday, March 16	Lecture: Introduction to rigging in Maya
5	Thursday, March 23	Lecture: Lip segmentation and tracking - rigging tips and tricks.
5	Thursday, March 30	SPRING BREAK: NO CLASS TODAY
6	Thursday, April 6	Lecture: Mesh deformation C++ API: Manipulating vertices in Maya.
6	Thursday, April 13	Lecture: Writing multi-threaded C++ Maya plugins
7	Thursday, April 20	Lecture: Developing Maya user interfaces plugins in QT.
7	Thursday, April 27	Lecture: Maya plugin File I/O.
8	Thursday, May 4	Lecture: Introduction to Directed Acyclic Graphs (DAG) for Maya scene descriptions.
8	Thursday, May 11	Project demonstration. DUE: Final Project.