Chris Pollett > Students >
Michelle Yuan

    ( Print View )

    [Bio]

    [CS297Proposal]

    [Del1]

    [Del2]

    [Del3]

    [CS297Report]

    [CS298Proposal]

    [CS298Presentation]

    [CS298Report]

                           

























CS298 Proposal

Transmitting Avatar Emotions over the Web

Jing Yuan Michelle_Yuan@yaho.com

Advisor: Dr. Chris Pollett

Committee Members: Dr. Ho Kuen Ng, Dept. of Mathematics and Computer Science (Ng@cs.sjsu.edu), Dr. Sin-Min Lee, Dept. of Mathematics and Computer Science (lee@cs.sjsu.edu)

Abstract:

Over the years psychologists have developed a number of encoding schemes for recording human facial expressions and to record hand gestures [E41], [EF69], [EF78], [M92]. Recently, there have been a number of AI projects, which attempt to use these recording schemes to write programs, which can recognize human emotions. The early example in development of facial animation technology began in 1992 by the Cambridge Research Laboratory and resulted in DECface [W1]. Several books and conference papers on Facial Animation research have appearanced [WL94A], [WL94B], [WL95], [FK96], [WLM98], [W92]. Imaginative applications of animated graphical faces are found in sophisticated human-computer interfaces, interactive games, multimedia titles, and, as always, in a broad variety of production animations. There has also been some work on an XML-based language, HumanML, for encoding such emotional and gesture information. Zong, Dohi, and Ishizuka [ZHMI00] have proposed an XML-based language MPML (multimodal presentation markup language), which is SMIL compliant for giving character agents emotions while they present things like PowerPoint slides. W3C organization also provides X3D that is XML based language for displaying 3D graphic.

In this project, we propose to develop a network poker game with 3D-characters being able to express emotional information based on their cards. The technologies we will use for our project are XML, X3D, ECMAscript, and XSLT. Each client chooses 3D characters for the various players from a selection of characters available on his/her local machine, and, via a form associated player.xml files for those 3D characters are uploaded to the server. This uploaded file contains details about which emotions these 3D players support. The game then starts. The server receives emotion messages sent by each client (these are coded in an XML language of our design) and sends the message to the other clients that whose 3D character for that player have the ability to display the emotion specified by the message. Message passing will all be done using HTTP, using tricks with frames. A hidden send frame is used to send GET requests to the server with an emotion message and a hidden receive frame is used to poll the server periodically via the HTTP-REFRESH technique to get information about other clients' emotions. When a client receives an emotion message, it is used to apply a XSLT transformation to the player the emotion is supposed to appear on to make the emotion appear.

CS297 Results

  • Researched and studied VRML and created VRML a 3D mock-up of my apartment.
  • Researched and studied the XML, XSLT language and created a DTD for ShML (a shape Markup language).
  • Researched and studied the paper [ZHMI00] and current variants human mark-up languages such as Multimodal Presentation Markup and HumanML etc.
  • Researched and studied [EF78] to understand various human facial behaviors and gestures in order to translate XML application to 3D human facial animation.
  • Implemented an XML based application to express human facial behaviors and its DTD.
  • Implemented an XML based emotion passing scheme for several character agents with different abilities such as eye blink, eyebrow up, and eyebrow down.

Proposed Schedule

Week 1-2 (8/26 - 9/8)Improve the face designs (more human face look like) of several characters
Week 3-5 (9/9 . 9/29)Add more facial animation actions (XSLT transforms)
Week 6-7 (9/30 . 10/13)Complete the design of the proposed network poker game: client-server set up.
Week 8-9 (10/14 - 10/27)Final coding of the whole project
Week 10-11 (10/28 . 11/10)Write the report
Week 12 (11/11 - 11/17)Give draft to committee, clean up code
Week 13 (11/18 - 11/24)Prepare final presentation
Week 14 (11/25 - 12/01)Finalize write-up
Week 15 (12/2 - 12/8)Oral defense

Key Deliverables:

  • Software
    • Define an XML language for specifying what emotions a 3D character supports.
    • Define an XML language for transmitting emotions.
    • Implement a variety of X3D characters and XSLT transforms to display emotions on them.
    • Set up network poker game (client-server) to test applying emotion messages to character through the web.
  • Report
    • The background information and the motivation behind the study of our research topic.
    • A detailed description of the proposed XML DTDs, X3D characters, and the style-sheet transforms on these characters.
    • A detailed description of the proposed network poker game set up and apply emotion message to character through the web.

Innovations and Challenges

We are proposing two new DTDs for emotions, which we hope might be useful in network gaming. Using XSLT is a novel approach to applying emotions to X3D characters. It is also challenging to get such transformations to work. Our approach uses existing open standards and might provide a new way to do network games that avoids proprietary systems such as DirectPlay (part of DirectX) from Microsoft. The technique of message passing in HTTP we intend to use is also novel and difficult to implement.

References:

1. [E41] Efron, D. Gesture and Environment. King's Crown Press. 1941.

2. [EF69] Ekman, P. and Friesen, W.V. The repertoire of nonverbal behavioural categories: Origins, usage, and coding. Semiotica 1:49-98.

3. [EF78] Ekman, P. & Friesen, W. V. The Facial Action Coding System (FACS): A technique for measurement of facial action. Consulting Psychologists Press. 1978.

4. [M92] David McNeill. Hand and Mind. University of Chicago Press. 1992. [W3C97] Extensible Markup Language (XML). www.w3.org/XML.

5. [W3C99] XSL Transformations (XSLT) Version 1.0. www.w3.org/TR/xslt.

6. [ZHMI00] Zong, Y., Hiroshi, D. and Mitsuru, I. Emotion expression functions Attached to Multimodal Presentation Markup Language MPML. 4th International Conference on Autonomous Agents. 2000.

7. [ECMA99] Standard ECMA-262 ECMAScript Language Specification 3rd ed. http://www.ecma.ch/ecma1/stand/ecma-262.htm. ECMA. 1999.

8. [W1] http://crl.research.compaq.com/projects/facial/facial.html

9. [WL94A] Waters, K. and Levergood, T. DECface: An automatic Lip-Synchronization Algorithm for Synthetic Faces, CRL Technical Report 93/4, September 1994

10. [WL94B] Waters, K. and Levergood, T. An Automatic Lip-Synchronization Algorithm for Synthetic Faces, ACM Multimedia, San Francisco, pages 149-156, October 1994

11. [WL95] Waters, K., Levergood, T. DECface: A System for Synthetic Face Applications, Journal of Multimedia Tools and Applications, Kluwer Academic Publishers, Vol. 1, No. 4, pages 349-366. Nov. 1995

12. [FK96] Computer Facial Animation, Fred I. Parke and Keith Waters. A. Peters Ltd., Boston Massachusetts, ISBN 1-56881-014-8 Hardcover, pp. 450, 1996. [software available for download]

13. [WLM98] Waters, K, Rehg, J, Loughlin, M, Kang, S, Computer Vision for Human-Machine Interaction, Roberto, C, and Pentland, A (eds)., Cambridge University press, pages 83-96, 1998

14. [W92] Waters, K. Modeling Three-Dimensional Facial Expressions, Processing Images of Faces, Bruce and Burton eds., Ablex, New Jersey, pages 202-227, 1992