Chris Pollett > Students > Gupta

    Print View

    [Bio]

    [Blog]

    [CS 297 Proposal]

    [Deliverable 1: Gedit-Python Extension]

    [Deliverable 2: Camera Capture Tool]

    [Deliverable 3: Detect Head Movement]

    [Deliverable 4: Map Text to Head Movement]

    [CS 297 Report - PDF]

    [CS 298 Proposal]

    [CS 298 Report - PDF]

    [CS 298 IHFTES Slides - PDF]

























CS298 Proposal

Improved Hand's Free Text Entry System

Gaurav Gupta (gaurav.gupta@sjsu.edu)

Advisor: Dr. Chris Pollett

Committee Members: Dr. Katerina Potika, Prof. Kevin Smith


Abstract:

An input device is a hardware device which is used to send input data to the computer, which is further utilized to control and interact with the computer system. Contemporary input mechanism can be categorized in several ways depending on the medium the input is provided, for example audio-based input, video-based input, input in the form of images. Few examples of contemporary input devices include Keyboard, Mouse, Siri or Alexa (voice-based input devices for Apple devices and Amazon Echo devices), Touchscreens (included on mobile interfaces and many others), video-based input devices like used in self-driving cars where continuous frames of images are provided as input. The objective of this project is to come up with a solution that provides an input entry mechanism based on head movements. Input entry based on head movements will help people with disability to interact with the computers easily.


CS 297 Results:

  • Developed a plugin for Gedit text editor to toggle content.
  • Develped a tool to interact with camera on Mac.
  • Detected head movements based on Lucas-Kanade principal.
  • Mapped text entry to head movements based on Lucas-Kanade principal.

Schedule:

Week 1,2: Feb. 6 - Feb. 20 Deliverable 1: Intermediate layer to write text on Atom editor which is generated from Python program.
Week 3, 4, 5: Feb. 21 - Mar. 13 Deliverable 2: Develop a Python based model to detect face and gestures in continuous frame of image.
Week 6, 7: Mar. 14 - Mar. 27 Deliverable 3: Propose a system design so that accessing features on the system is convenient.
Week 8, 9, 10: Mar. 28 - Apr. 17 Deliverable 4: Generate text on the basis of head movement.
Week 7: Apr 18 - May. 1 Deliverable 5: CS 298 report and presentation.


Innovations and Challenges

  • Provide a convenient and rapid mechanism to generate text input based on head movement.
  • Provide a smart mechanism to access key features of the system using facial gestures.
  • There are two research papers that provide a solution for generating text from head movements, challenge over here is to suggest a solution which proposes an improvement on the prior implemented techniques.

References:

  1. [2017] 3-Steps Keyboard: Reduced Interaction Interface for Touchless Typing with Head Movements. Adam Nowosielski. in Proceedings of the 10th International Conference on Computer Recognition Systems CORES 2017, Polanica Zdroj, Poland, 2017. [Online] Available: https://link.springer.com/chapter/10.1007/978-3-319-59162-9_24.
  2. [2017] Two-Letters-Key Keyboard for Predictive Touchless Typing with Head Movements. Adam Nowosielski. 17th International Conference, CAIP 2017, Ystad, Sweden, Proceedings, Part I. [Online] Available: https://link.springer.com/chapter/10.1007/978-3-319-64689-3_6.
  3. [2017] Face Detection Based on Skin Color and AdaBoost Algorithm.C. Lv, T. Zhang, C. Lin. in Control and Decision Conference (CCDC) 2017, Chongqing, China. [Online] Available: http://ieeexplore.ieee.org/document/7978729/.
  4. [2017] A Face Detection Algorithm based on Adaboost and new Haar-Like Feature. S. Ma, L. Bai in 7th IEEE International Conference on Software Engineering and Service Science (ICSESS) 2017, Beijing, China [Online] Available: http://ieeexplore.ieee.org/abstract/document/7883152/.
  5. [1998] Neural Network-Based Face Detection. H.A. Rowley, S. Baluja IEEE Transaction on Pattern Analysis and Machine Intelligence, vol 20, no.1 pp. 23-38, 1998 [Online] Available: http://ieeexplore.ieee.org/document/655647/.
  6. Wikipedia, "TensorFlow," [Online]. Available:https://en.wikipedia.org/wiki/TensorFlow
  7. Google, "TensorFlow," [Online]. Available:https://www.tensorflow.org/
  8. Google Brain Team, "TensorFlow Object Detection API," [Online]. Available:https://www.github.com/tensorflow/models/tree/master/research/object_detection/
  9. "Tensorflow Detection Model Zoo,"[Online]. Available:https://www.github.com/tensorflow/models/blob/master/research/object_detection/g3doc/detection_model_zoo.md
  10. "Tensorflow Tutorial for Object Detection API,"[Online]. Available:https://pythonprogramming.net/introduction-use-tensorflow-object-detection-api-tutorial/
  11. Wikipedia "Atom (text Editor),"[Online]. Available:https://en.wikipedia.org/wiki/Atom_(text_editor)
  12. "Atom editor home page,"[Online]. Available:https://atom.io/
  13. J. Kaufman, "google-1000-english,"[Online]. Available:https://github.com/first20hours/google-10000-english
  14. Z. Zhang, P. Luo, C. C. Loy, X. Tang, "Learning Social Relation Traits from Face Images," in International Conference on Computer Vision (ICCV) 2015, Santiago, Chile. [Online]. Available:https://mmlab.ie.cuhk.edu.hk/projects/socialrelation/index.html