Chris Pollett > Students > Gupta

    Print View

    [Bio]

    [Blog]

    [CS 297 Proposal]

    [Deliverable 1: Gedit-Python Extension]

    [Deliverable 2: Camera Capture Tool]

    [Deliverable 3: Detect Head Movement]

    [Deliverable 4: Map Text to Head Movement]

    [CS 297 Report - PDF]

    [CS 298 Proposal]

























CS298 Proposal

Improved Hand's Free Text Entry System

Gaurav Gupta (gaurav.gupta@sjsu.edu)

Advisor: Dr. Chris Pollett

Committee Members: Dr. Katerina Potika, Prof. Kevin Smith


Abstract:

An input device is a hardware device which is used to send input data to the computer, which is further utilized to control and interact with the computer system. Contemporary input mechanism could be categorized in several ways depending on the medium the input is provided, for example audio based input, video based input, input in the form of images. Few examples of contemporary input devices include: Keyboard, Mouse, Siri or Alexa (voice based input devices for Apple devices and Amazon Echo devices), Touchscreens (included on mobile interfaces and many others), Video based input devices like used in self driving cars where continuous frames of images are provided as input. The objective of this project is to come up with a solution that provides an input entry mechanism based on head movements. Input entry based on head movements would help people with disability to interact with the computing devices easily.


CS 297 Results:

  • Developed a plugin for Gedit Editor to toggle content
  • Develped a tool to interact with camera on Mac
  • Detected Head Movements based on Lucas-Kanade principal
  • Mapped text entry to head movements based on Lucas-Kanade principal

Schedule:

Week 1,2: Feb. 6 - Feb. 20 Deliverable 1: Program to update file on Atom editor from the output generated from Python program
Week 3, 4, 5: Feb. 21 - Mar. 13 Deliverable 2: Develop a Python based neural network model to detect faces in continous frames of image.
Week 6, 7: Mar. 14 - Mar. 27 Deliverable 3: Propose a system design so that accessing features on the system is convinient.
Week 8, 9, 10: Mar. 28 - Apr. 17 Deliverable 4: Generate text on the basis of head movement.
Week 7: Apr 18 - May. 1 Deliverable 5: CS 298 report and presentation

Key Deliverables:

  • Design:
    • Propose a design for the system so that it helps in accessing the features of the system conviniently.
  • Software:
    • Intermediate layer to write text on Atom editor which is generated from Python program
    • Tool to detect facial region with the help of neural network
    • Generate text on the basis of head movement
  • Report:
    • CS 298 Report
    • Presentation

Innovations and Challenges

  • Provide an innovative, convenient and rapid mechanism to generate text input based on head movement.
  • Provide a meaningful design such that selecting text character or suggested word could be done smartly.
  • There are two research papers that provide a solution for generating text from head movements, challenge over here is to suggest a solution which proposes an improvement on the prior implemented techniques.

References:

  1. [2017] 3-Steps Keyboard: Reduced Interaction Interface for Touchless Typing with Head Movements. Adam Nowosielski. in Proceedings of the 10th International Conference on Computer Recognition Systems CORES 2017, Polanica Zdroj, Poland, 2017. [Online] Available: https://link.springer.com/chapter/10.1007/978-3-319-59162-9_24.
  2. [2017] Two-Letters-Key Keyboard for Predictive Touchless Typing with Head Movements. Adam Nowosielski. 17th International Conference, CAIP 2017, Ystad, Sweden, Proceedings, Part I. [Online] Available: https://link.springer.com/chapter/10.1007/978-3-319-64689-3_6.
  3. [2011] Optimization of Head Movement Recognition Using Augmented Radial Basis Function Neural Network. Mitchell Yuwono, A.M. Ardi Handojoseno, H.T. Nguyen. 33rd Annual International Conference of the IEEE EMBS. [Online] Available: http://ieeexplore.ieee.org/document/6090760/.
  4. [2011] Head Movement Recognition Based on LK Algorithm and Gentleboost. Liu Jian-zheng, Zhao Zheng. Networked Computing and Advanced Information Management (NCM) [Online] Available: http://ieeexplore.ieee.org/document/5967551/.
  5. [2012] Head Movement Recognition Based on Lucas- Kanade Algorithm. Zheng Zhao, Yuchuan Wang, Shengbo Fu. International Conference on Computer Science and Service System [Online] Available: http://ieeexplore.ieee.org/document/6394889/.