Chris Pollett > Students >
Lodha
    ( Print View)

    [Bio]

    [Blog]

    [ Unity Hands-on ]

    [CS 297 Proposal]

    [Deliverable 1 ]

    [Deliverable 2]

    [Deliverable 3]

    [Deliverable 4]

    [Skeleton Based Action Recognition-PDF]

    [CS 297 Report-PDF]

    [CS298 Proposal]

    [Faster-RCNN-PDF]

    [CS298 Final Report-PDF]

    [CS298 Final Presentation-PDF]

Project Blog CS 298

Week 9 - April 13, 2021

Minutes

  • Shown the demo of LeNet5 Model for 3 classes.
  • Finalized the ASL classes : Hello, Help, Dollar, Washroom, Food, Okay, Hand Wave.
  • Discussed about OpenPose Model to get the 2D co-ordinates of the human skeleton in any input video.
  • Discussed about training the model with just 2d co-ordinate data and make RG images instead of RGB images and draw insights.
  • Work on interface that has 5 webpages as part of the detected action's response.

To-Do

  • Develop Unity Dataset with 3d co-ordinates generated in Unity.
  • Train the model with any one class with RG images instead of RGB.
  • Gather dataset for Help, Dollar, Washroom ASL.
  • Train the LeNet5 Model for all the finalised classes.
  • Develop a basic UI for responding to the detected signs.

Week 8 - April 6, 2021

Minutes

  • Shown the demo of SVM Model for 3 classes.
  • Discussed about different CNN model to use for training the model : SqueezeNet and LeNet 5.

To-Do

  • Develop a CNN Based model for these 3 classes.
  • Train the model for Unity generated video dataset.

SPRING BREAK : March 30 - April 5 2021

  • Studied Transfer Learning.
      • Important Points:
    • When the amount of training data is not sufficient to adjust all parameters which causes an overfitting. In this case, transfer learning and fine-tuning are used.
    • Transfer-learning (or shallow retraining) v/s Fine-tuning (or deep retraining)
References : (PDF) Leveraging Pre-trained CNN Models for Skeleton-Based Action Recognition. Available from: https://www.researchgate.net/publication/337450352_Leveraging_Pre-trained_CNN_Models_for_Skeleton-Based_Action_Recognition [accessed Apr 06 2021].

Week 7 - March 24, 2021

Minutes

  • Generation of csv file having 60 class lables and 3d joint points of the generated temporal image dataset to be used for CNN.
  • Keras based Le-Net model creation.
  • Discussion on Le-Net architecture for recognition.

To-Do

  • Support Vector Machine (SVM) Image Classification for 3 classes to analyse the precision and recall of this temporal dataset for prediction.
  • Start training on basic CNN model.

Week 6 - March 16, 2021

Minutes

  • Demo of RGB images geneartion from 3D joint points.
  • Pre-processing for CNN : Resize and Normalise the images.
  • Discussion about using CNN to do image recognition on these RGB image dataset.

To-Do

Week 5 - March 9, 2021

Minutes

  • Successful transformation of motioncapture sequences into a simple spatio-temporal RGB image-like representation.

To-Do

Week 4 - March 2, 2021

Minutes

  • Took the skeleton related files from NTURGB and parsed it to create 3 matrix : R,G,B
  • NTURGB Datasets

To-Do

Week 3 - Feb 23, 2021

Minutes

  • Presentation on Faster RCNN model.
  • Detailed discussion about Window Proposal Network from Action Recognition Paper.
  • Study and understand few ASL action recognition datasets from this list of ASL Datasets

To-Do

  • Try to create image from 3d joints.
  • Start trying the Faster RCNN model with window proposal network.

Week 2 - Feb 16, 2021

Minutes

  • Discussion about the categorizing 3d coordinates into various body parts.
  • Discussion about Window Proposal Network from Action Recognition Paper

To-Do

  • Crop the Okay Sign video dataset.
  • Try Open Pose Model for our cropped video dataset
  • Study "Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks".
  • Create presentation for Faster RCNN.
  • Study about processing video dataset into frames to pass it into the model.

Week 1 - Feb 9, 2021

Minutes

  • Kickoff meeting and idea discussion.
  • Discusses about the plan for the semester : Build Action Recognition Model, Creation of User Interface, Working on Dataset.

To-Do

  • Start working on Action Recognition Model

Week 0 - Feb 2 - 5, 2021

Minutes

No Meeting

  • Cs298 Proposal Submission
  • Got Cs298 Add Code

Project Blog CS297

Week 15 - Dec 15, 2020

Minutes

  • CS297 Final Report Submission and "CR" received.

Week 14 - Dec 8, 2020

Minutes

  • CS297 Final Report review.

Week 13 - Dec 1, 2020

Minutes

  • Submission of Deliverable 4 : video dataset of Bone Points Detection for Okay Sign.
  • Cs297 Draft Report review.

Week 12 - Nov 24, 2020

Minutes

  • Discussed the progress so far.
  • Deliverable 4. discussion about Bone Points Detection on the videos of Okay Sign.
  • Presentation on Action Recognition Model.
  • Final Submission on Dec 8.

To-Do:

  • Try to make Okay Sign a better.
  • Upload Deliverable 2.
  • Read about Skeleton Detection
  • Try to use script for video creation

Week 12 - Nov 17, 2020

Minutes

  • Discussed the progress so far.
  • Instructions to start writing report.
  • Proposal Review. Draft 1 due on Dec 1.
  • Deliverable 4. discussion about Skeleton Detection instead of model on Okay Sign.
  • Final Meeting would be on Dec 8, 2020.
  • Final Submission on Dec 8.

To-Do:

  • Try to make Okay Sign a better.
  • Upload Deliverable 2.
  • Read about Skeleton Detection
  • Try to use script for video creation

Week 11 - Nov 10, 2020

Minutes

  • Animation Clip of avator doing wave. Work on hands in the animation. They should look good for better accuracy.

To-Do:

  • Work on hands in the animation. They should look good for better accuracy.
  • Work on it to make it do Okay sign.

Week 10 - Nov 3, 2020

Minutes

  • Shown about Humoind Avator Rigs Configuration and how I created a OK Sign.
  • Shown bookshelf created from a signle table and placed in the room near humoid avator.
  • 5 signs that we may take for humoid figure : Dollar, Name, Thank You, Help, Home.
  • Discussion about Deliverable 3 expectation of the Unity Movie from various angle.
  • Discussion about Final Deliverable - OK Sign or Not OK Sign Detector
    • Deliverable 4 is to create a model capable of detecting the hand gesture sign (OK Sign) of human.

To-Do:

  • Create Animation for the OK Sign.
  • Deliverable 3 : Unity Moview - Generate 5 videos from diffierant angles to capture the actions of humanoid to be used for model training.
  • Submit Deliverable 2.

Week 9 - Oct 27, 2020

Minutes

  • Shown demo of my Unity projects where a human (Avator: Elizabet Warren) & a cartoon human avator figure was placed in a Washroom having a Mirror, Sink, Door and side table in a Room. Elizabet was talking to the character by using hands & face moveemnt. The cartoon character was waving back to her.
  • Discussion about creating a shop where 1 shelves would be placed and 1 human figure doing an OK sign as part of deliverable 2.

To-Do:

  • Learn about Rigs
  • Create the room where the bookshelf and avators are placed.
  • Learn how to make the avator do the OK Sign.
  • Finaly create the unity project by placing the humanoid figure doing OK sign in a room.
  • Think about atleast 5 signs that we can take up this humoid figure

Week 8 - Oct 20, 2020

MidTerm Week Break

Week 7 - Oct 15, 2020

Minutes

  • Shown demo about a Unity projects where a human figure was placed in a 3D space and was capable of doing some actions on press of key : Juml, Move Left/Right, Move Forward/Backward in a 3d space.
  • Start Deliverable 2: Discussion about creating a shop where 2 shelves would be placed and 1 human figure doing an OK sign as part of deliverable 2.

To-Do:

  • Do a tutorial on Unity.
  • Create a project with 1 human doing OK sign.

    • Week 6 - Oct 6, 2020

    Minutes

    • Shown demo about a Unity projects where a human figure was placed in a 3D space and was capable of doing some actions on press of key : Juml, Move Left/Right, Move Forward/Backward.
    • Start Deliverable 2: Discussion about creating a shop where 2 shelves would be placed and 1 human figure doing an OK sign as part of deliverable 2.

    To-Do:

    • Do a tutorial on Unity.
    • Start Deliverable 2: Create a project with 1 human doing OK sign in 3d space.

      • Week 5 - Sept 29, 2020

      Minutes

      • Shown demo of LeNet5 Model for ASL Alphabets Detection.
      • Submitted Deliverable 1.
      • Discussion about next milestone and to get handson on Unity.

      To-Do:

      • Do a tutorial on Unity.
      • Learn about Object Placements in a 3d Space.

      Week 4 - Sept 22, 2020

      Minutes

      • Discussion about the intial CNN model we have: how many conv2d, max pooling layer, activation, epoch size .
      • Discussion about LeNet-5

      To-Do:

      • Try and experiment with the CNN layers and its parameters
      • Study about LeNet-5 architecture
      • Create a document/ppt about my understanding of terminologies of CNN

      Week 3 - Sept 15, 2020

      Minutes

      • Review of the proposal
      • Discussion about the developing a CNN for static American Sign Language detection using PyTorch and/or OpenCV.
      • Finalised the alphabets of ASL for detection project. Dropped J & Z as it needs motion detection.

      To-Do:

      • Code a basic CNN to detect static ASL.
      • Create a short presenation on Motion Detection using OpenCV.
      • Gather datasets for other sign languages.

      Week 2 - Sept 1, 2020

      Minutes

      • Project Overiew, Auidence Analysis, Deliverables Finalization.
      • Plan for the Project and list of deliverables Finalised.

      To-Do:

      • Questions :
        • What is the best way to give feedback?
        • What kind of security cameras will support motion detection?
      • PyTorch & OpenCV Study
      • Read research papers on Sign Language Detection.
      • Update the website with timelines & task in CS297 Proposal webpage.

      Week 1 - Aug 25, 2020

      Minutes

      • Kickoff meeting and idea discussion.
      • Discusses about the demand of American Sign Language using Statistics from NIDCD.

      To-Do:

      • Find 4 concrete things that will help in project.
      • Write draft CS297 proposal
      • Update the website with Bio and Idea Description.