CS297 Proposal

I'm there for you - Sign Language Assistant.

Charulata Lodha (charulatamahesh.lodha@sjsu.edu)

Advisor: Dr. Chris Pollett

Description:

We are developing a prototype computer vision system to help the deaf and mute communicate in a shopping setting. Our goal system would use video feeds to recognize American Sign Language (ASL) gestures and notify shop clerks of deaf and mute patrons' intents. Our prototype will operate on videos created in Unity of 3D humanoid models in a shop setting performing ASL signs. This semester we wrote four short programs to help us towards development of our final system.

Features:

-Real time ASL hand gesture detection and interpretation from ASL to English speech.

-Easily Integrable with any camera software.

-Removes the dependability of human translators and makes communication easy for all people.

Why :

Between 6 and 8 million people in the United States have some form of language impairment [1]. American Sign Language (ASL) is the leading minority language in the U.S. after the "big four": Spanish, Italian, German, and French.

Schedule:

Week 1: Aug 19 - Aug 25	Kickoff meeting and idea discussion.
Week 2: Aug 25 - Sept 1	Project Overview, Audience Analysis, Deliverables Finalization.
Week 3: Sept 1 - Sept 8	Learn about PyTorch and OpenCV. Read research papers on Sign Language Detection.
Week 4: Sept 8 - Sept 15	Study various CNN architectures. Study ASL for Alphabets.
	STAGE 1 - RESEARCH ABOUT ASL and CNN COMPLETED
Week 5: Sept 15 - Sept 22	Study OpenCV motion detection. Gather other sign language datasets. Code a neural network for Static Sign Language Alphabets Detection.
Week 6: Sept 22 - Sept 29	Study LeNet-5. Experiment with CNN parameters and architecture to improve the accuracy.
Week 7: Sept 29 - Oct 06	Deliverable 1 : 'Static ASL Detector' due on Oct 06.
	STAGE 2 - AI MODEL DEVELOPMENT COMPLETED
Week 8: Oct 6 - Oct 13	Learn Unity to create video of a human figure doing some actions in 3D using inbuilt animations.
Week 9: Oct 20 - Oct 27	Learn to do rigs configuration and create animation on a humoid avators.
	STAGE 3 - UNITY LEARNING COMPLETED
Week 10: Oct 27 - Nov 3	Work on Deliverable 2. Create animations for scratch.
Week 11: Nov 3 - Nov 17	Deliverable 2 due on Nov 17.
Week 11: Nov 17 - Nov 24	Deliverable 3 and Skeleton Detection Research Paper Presentation due on Nov 24.
	STAGE 4 - UNITY MOVIE and ANIMATION CREATION COMPLETED
Week 13: Nov 24 - DEC 1	Work on deliverable 4: Identify the bone joints of the humanoid avator. Deliverable 4 due on dec 4.
	STAGE 5 - COMPLETED
Week 14: DEC 8	CS 297 FINAL REPORT Submission.

Deliverables:

The full project will be done when CS298 is completed. The following will be done by the end of CS297:

1. American Sign Language Alphabets Recognition Using Lenet-5

2. Sign Language Gesture Creation In A Shop Setting Using Animations In Unity

3. Humanoid Gesture Synthetic Dataset Creation In Unity

4. Bone Key Points Detection Using OpenPose And Studying Skeleton-Based Action Recognition

5. CS 297 Report

References:

[1] Statistics on Voice, Speech, and Language - NIDCD

[2] Hundred basic ASL signs that are frequently used between parents and their young children

[3] Humanoid Avatars