Chris Pollett > Students >
Kulkarni
    ( Print View)

    [Bio]

    [Blog]

    [CS297 Proposal]

    [CS297 - Deliverable 1]

    [CS297 - Deliverable 2]

    [CS297 - Deliverable 3]

    [CS297 - Deliverable 4]

    [CS297 Report - PDF]

    [CS298 Proposal]

    [CS298 - Deliverable 1]

    [CS298 - Report -PDF]

CS297 Proposal

AI assisted User Interface Development

Siddharth Kulkarni (siddharth.kulkarni@sjsu.edu)

Advisor: Dr. Chris Pollett

Description:

The User interface design process involves translating project requirements from creative explorations in the form of mockups to working prototypes. My Deep Learning powered solution aims to streamline overall user interface development by generating prototypes directly from design sketches.

Schedule:

Week 1: Sept. 2 to Sept. 8Discuss idea with Professor, get feedback, plan steps ahead, Read pix2code paper
Week 2: ` Sept. 9 to Sept. 15Research Recurrent Neural Network for Drawing Classification
Week 3: Sept. 16 to Sept. 22Learn Tensorflow basics, Use Google QuickDraw image data set on RNN Model
Week 4: Sept. 23 to Sept. 29Deliverable #1: Tensorflow implementation of Drawing Classification
Week 5: Sept. 30 to Oct. 6Learn how Convolutional neural networks work
Week 6: Oct. 7 to Oct. 13Research with ml4a library
Week 7: Oct. 14 to Oct. 20Openframeworks environment setup and tutorial
Week 8: Oct. 21 to Oct. 27Doodle Classifier Classification installation, getting example to work
Week 9: Oct. 28 to Nov. 3Doodle Classifier fixing bugs and other progress
Week 10: Nov. 4 to Nov. 10no meeting (Training on circle, squares and triangles)
Week 11: Nov. 11 to Nov. 17Deliverable #2: Hand drawn shape classification using Doodle Classifier
Week 12: Nov. 18 to Nov. 24no meeting (Training model to learn Components)
Week 13: Nov. 25 to Dec. 1Deliverable #3: Model trained to recognize UI Components
Week 14: Dec. 2 to Dec. 8Deliverable #4: Program to send detections to web server using OSC
Week 15: Dec. 9 to Dec. 15Deliverable #5: CS297 Report, discuss steps ahead

Deliverables:

The full project will be done when CS298 is completed. The following will be done by the end of CS297:

1. Tensorflow implementation of Drawing Classification using RNN

2. Model capable of Drawing Classification using Covnet

3. Model trained to recognize UI Components

4. Program to convert UI sketches into code

5. CS297 Report

References:

[2017] pix2code: Generating Code from a Graphical User Interface Screenshot. Tony Beltramelli. EICS '18 Proceedings of the ACM SIGCHI Symposium on Engineering Interactive Computing Systems. 2018.

[2018] A Neural Representation of Sketch Drawings David Ha and Douglas Eck. International Conference on Learning Representations

[2015] An Introduction to Convolutional Neural Networks. O'Shea, Keiron and Nash, Ryan. ArXiv e-prints. 2015

[1996] Long short-term memory. Neural computation S. Hochreiter and J. Schmidhuber. 1997.

[2014] Long-term recurrent convolutional networks for visual recognition and description J. Donahue, L. Anne Hendricks, S. Guadarrama, M. Rohrbach, S. Venugopalan, K. Saenko, and T. Darrell. In Proceedings of the IEEE conference on computer vision and pattern recognition