CS298 Proposal
Transmitting Avatar Emotions over the Web
Jing Yuan Michelle_Yuan@yaho.com
Advisor: Dr. Chris Pollett
Committee Members: Dr. Ho Kuen Ng, Dept. of Mathematics and Computer Science
(Ng@cs.sjsu.edu), Dr. Sin-Min Lee, Dept. of Mathematics and Computer Science (lee@cs.sjsu.edu)
Abstract:
Over the years psychologists have developed a number of encoding schemes for recording human facial
expressions and to record hand gestures [E41], [EF69], [EF78], [M92]. Recently, there have been a number
of AI projects, which attempt to use these recording schemes to write programs, which can recognize human
emotions. The early example in development of facial animation technology began in 1992 by the Cambridge
Research Laboratory and resulted in DECface [W1]. Several books and conference papers on Facial Animation
research have appearanced [WL94A], [WL94B], [WL95], [FK96], [WLM98], [W92]. Imaginative applications of
animated graphical faces are found in sophisticated human-computer interfaces, interactive games,
multimedia titles, and, as always, in a broad variety of production animations. There has also been some
work on an XML-based language, HumanML, for encoding such emotional and gesture information. Zong, Dohi,
and Ishizuka [ZHMI00] have proposed an XML-based language MPML (multimodal presentation markup language),
which is SMIL compliant for giving character agents emotions while they present things like PowerPoint
slides. W3C organization also provides X3D that is XML based language for displaying 3D graphic.
In this project, we propose to develop a network poker game with 3D-characters being able to express
emotional information based on their cards. The technologies we will use for our project are XML, X3D,
ECMAscript, and XSLT. Each client chooses 3D characters for the various players from a selection of
characters available on his/her local machine, and, via a form associated player.xml files for those 3D
characters are uploaded to the server. This uploaded file contains details about which emotions these 3D
players support. The game then starts. The server receives emotion messages sent by each client (these
are coded in an XML language of our design) and sends the message to the other clients that whose 3D
character for that player have the ability to display the emotion specified by the message. Message
passing will all be done using HTTP, using tricks with frames. A hidden send frame is used to send GET
requests to the server with an emotion message and a hidden receive frame is used to poll the server
periodically via the HTTP-REFRESH technique to get information about other clients' emotions. When a
client receives an emotion message, it is used to apply a XSLT transformation to the player the emotion
is supposed to appear on to make the emotion appear.
CS297 Results
- Researched and studied VRML and created VRML a 3D mock-up of my apartment.
- Researched and studied the XML, XSLT language and created a DTD for ShML (a shape Markup
language).
- Researched and studied the paper [ZHMI00] and current variants human mark-up languages such
as Multimodal Presentation Markup and HumanML etc.
- Researched and studied [EF78] to understand various human facial behaviors and gestures in
order to translate XML application to 3D human facial animation.
- Implemented an XML based application to express human facial behaviors and its DTD.
- Implemented an XML based emotion passing scheme for several character agents with different
abilities such as eye blink, eyebrow up, and eyebrow down.
Proposed Schedule
Week 1-2 (8/26 - 9/8) | Improve the face designs (more human face look like) of several characters
|
Week 3-5 (9/9 . 9/29) | Add more facial animation actions (XSLT transforms)
|
Week 6-7 (9/30 . 10/13) | Complete the design of the proposed network poker game: client-server
set up.
|
Week 8-9 (10/14 - 10/27) | Final coding of the whole project
|
Week 10-11 (10/28 . 11/10) | Write the report
|
Week 12 (11/11 - 11/17) | Give draft to committee, clean up code
|
Week 13 (11/18 - 11/24) | Prepare final presentation
|
Week 14 (11/25 - 12/01) | Finalize write-up
|
Week 15 (12/2 - 12/8) | Oral defense
|
Key Deliverables:
- Software
- Define an XML language for specifying what emotions a 3D character supports.
- Define an XML language for transmitting emotions.
- Implement a variety of X3D characters and XSLT transforms to display emotions on them.
- Set up network poker game (client-server) to test applying emotion messages to character through
the web.
- Report
- The background information and the motivation behind the study of our
research topic.
- A detailed description of the proposed XML DTDs, X3D characters, and the
style-sheet transforms on these characters.
- A detailed description of the proposed network poker game set up and apply
emotion message to character through the web.
Innovations and Challenges
We are proposing two new DTDs for emotions, which we hope might be useful in network gaming. Using
XSLT is a novel approach to applying emotions to X3D characters. It is also challenging to get such
transformations to work. Our approach uses existing open standards and might provide a new way to do
network games that avoids proprietary systems such as DirectPlay (part of DirectX) from Microsoft.
The technique of message passing in HTTP we intend to use is also novel and difficult to implement.
References:
1. [E41] Efron, D. Gesture and Environment. King's Crown Press. 1941.
2. [EF69] Ekman, P. and Friesen, W.V. The repertoire of nonverbal behavioural categories: Origins,
usage, and coding. Semiotica 1:49-98.
3. [EF78] Ekman, P. & Friesen, W. V. The Facial Action Coding System (FACS): A technique for
measurement of facial action. Consulting Psychologists Press. 1978.
4. [M92] David McNeill. Hand and Mind. University of Chicago Press. 1992. [W3C97] Extensible Markup
Language (XML). www.w3.org/XML.
5. [W3C99] XSL Transformations (XSLT) Version 1.0. www.w3.org/TR/xslt.
6. [ZHMI00] Zong, Y., Hiroshi, D. and Mitsuru, I. Emotion expression functions Attached to
Multimodal Presentation Markup Language MPML. 4th International Conference on Autonomous Agents. 2000.
7. [ECMA99] Standard ECMA-262 ECMAScript Language Specification 3rd ed.
http://www.ecma.ch/ecma1/stand/ecma-262.htm. ECMA. 1999.
8. [W1] http://crl.research.compaq.com/projects/facial/facial.html
9. [WL94A] Waters, K. and Levergood, T. DECface: An automatic Lip-Synchronization Algorithm for
Synthetic Faces, CRL Technical Report 93/4, September 1994
10. [WL94B] Waters, K. and Levergood, T. An Automatic Lip-Synchronization Algorithm for Synthetic
Faces, ACM Multimedia, San Francisco, pages 149-156, October 1994
11. [WL95] Waters, K., Levergood, T. DECface: A System for Synthetic Face Applications, Journal of
Multimedia Tools and Applications, Kluwer Academic Publishers, Vol. 1, No. 4, pages 349-366. Nov. 1995
12. [FK96] Computer Facial Animation, Fred I. Parke and Keith Waters. A. Peters Ltd., Boston
Massachusetts, ISBN 1-56881-014-8 Hardcover, pp. 450, 1996. [software available for download]
13. [WLM98] Waters, K, Rehg, J, Loughlin, M, Kang, S, Computer Vision for Human-Machine Interaction,
Roberto, C, and Pentland, A (eds)., Cambridge University press, pages 83-96, 1998
14. [W92] Waters, K. Modeling Three-Dimensional Facial Expressions, Processing Images of Faces, Bruce
and Burton eds., Ablex, New Jersey, pages 202-227, 1992
|