CS298 Proposal

Improved Hand's Free Text Entry System

Gaurav Gupta (gaurav.gupta@sjsu.edu)

Advisor: Dr. Chris Pollett

Committee Members: Dr. Katerina Potika, Prof. Kevin Smith

Abstract:

An input device is a hardware device which is used to send input data to the computer, which is further utilized to control and interact with the computer system. Contemporary input mechanism can be categorized in several ways depending on the medium the input is provided, for example audio-based input, video-based input, input in the form of images. Few examples of contemporary input devices include Keyboard, Mouse, Siri or Alexa (voice-based input devices for Apple devices and Amazon Echo devices), Touchscreens (included on mobile interfaces and many others), video-based input devices like used in self-driving cars where continuous frames of images are provided as input. The objective of this project is to come up with a solution that provides an input entry mechanism based on head movements. Input entry based on head movements will help people with disability to interact with the computers easily.

CS 297 Results:

Developed a plugin for Gedit text editor to toggle content.
Develped a tool to interact with camera on Mac.
Detected head movements based on Lucas-Kanade principal.
Mapped text entry to head movements based on Lucas-Kanade principal.

Schedule:

Week 1,2: Feb. 6 - Feb. 20	Deliverable 1: Intermediate layer to write text on Atom editor which is generated from Python program.
Week 3, 4, 5: Feb. 21 - Mar. 13	Deliverable 2: Develop a Python based model to detect face and gestures in continuous frame of image.
Week 6, 7: Mar. 14 - Mar. 27	Deliverable 3: Propose a system design so that accessing features on the system is convenient.
Week 8, 9, 10: Mar. 28 - Apr. 17	Deliverable 4: Generate text on the basis of head movement.
Week 7: Apr 18 - May. 1	Deliverable 5: CS 298 report and presentation.

Key Deliverables:

Design:

Propose a design for the system so that it helps in accessing the features of the system conveniently.

Software:

Intermediate layer to write text on Atom editor which is generated from Python program.
Tool to detect face and gestures in continuous frame of image.
Generate text on the basis of head movement.

Experiments:

Compare performance based on the technique suggested by Nowosielski [1][2] with mine.

Provide analysis on the features used in the model and number of letters that could be typed in one minute using both models.
Report:
- CS 298 Report
- Presentation

Innovations and Challenges

Provide a convenient and rapid mechanism to generate text input based on head movement.
Provide a smart mechanism to access key features of the system using facial gestures.
There are two research papers that provide a solution for generating text from head movements, challenge over here is to suggest a solution which proposes an improvement on the prior implemented techniques.

References:

[2017] 3-Steps Keyboard: Reduced Interaction Interface for Touchless Typing with Head Movements. Adam Nowosielski. in Proceedings of the 10th International Conference on Computer Recognition Systems CORES 2017, Polanica Zdroj, Poland, 2017. [Online] Available: https://link.springer.com/chapter/10.1007/978-3-319-59162-9_24.
[2017] Two-Letters-Key Keyboard for Predictive Touchless Typing with Head Movements. Adam Nowosielski. 17th International Conference, CAIP 2017, Ystad, Sweden, Proceedings, Part I. [Online] Available: https://link.springer.com/chapter/10.1007/978-3-319-64689-3_6.
[2017] Face Detection Based on Skin Color and AdaBoost Algorithm.C. Lv, T. Zhang, C. Lin. in Control and Decision Conference (CCDC) 2017, Chongqing, China. [Online] Available: http://ieeexplore.ieee.org/document/7978729/.
[2017] A Face Detection Algorithm based on Adaboost and new Haar-Like Feature. S. Ma, L. Bai in 7th IEEE International Conference on Software Engineering and Service Science (ICSESS) 2017, Beijing, China [Online] Available: http://ieeexplore.ieee.org/abstract/document/7883152/.
[1998] Neural Network-Based Face Detection. H.A. Rowley, S. Baluja IEEE Transaction on Pattern Analysis and Machine Intelligence, vol 20, no.1 pp. 23-38, 1998 [Online] Available: http://ieeexplore.ieee.org/document/655647/.
Wikipedia, "TensorFlow," [Online]. Available:https://en.wikipedia.org/wiki/TensorFlow
Google, "TensorFlow," [Online]. Available:https://www.tensorflow.org/
Google Brain Team, "TensorFlow Object Detection API," [Online]. Available:https://www.github.com/tensorflow/models/tree/master/research/object_detection/
"Tensorflow Detection Model Zoo,"[Online]. Available:https://www.github.com/tensorflow/models/blob/master/research/object_detection/g3doc/detection_model_zoo.md
"Tensorflow Tutorial for Object Detection API,"[Online]. Available:https://pythonprogramming.net/introduction-use-tensorflow-object-detection-api-tutorial/
Wikipedia "Atom (text Editor),"[Online]. Available:https://en.wikipedia.org/wiki/Atom_(text_editor)
"Atom editor home page,"[Online]. Available:https://atom.io/
J. Kaufman, "google-1000-english,"[Online]. Available:https://github.com/first20hours/google-10000-english
Z. Zhang, P. Luo, C. C. Loy, X. Tang, "Learning Social Relation Traits from Face Images," in International Conference on Computer Vision (ICCV) 2015, Santiago, Chile. [Online]. Available:https://mmlab.ie.cuhk.edu.hk/projects/socialrelation/index.php