Project Blog

Feb 19, 2019

Presented existing system metrics with demo with more components

Tasks for next meeting

  • Task #1: Work towards deliverable 1

Feb 12, 2019

Showed proposal edits along with detailed schedule

Tasks for next meeting

  • Task #1: Make slides on metric of existing system
  • Task #2: Demo of existing systen with more components

Feb 5, 2019

Reviewed work done in CS297, revisted Deliverable 4, thought about various ideas for this semester (CS298)

Ideated possible archicture for end-to-end system with openframeworks(C++), web-server (NodeJS), client (angular), and database (mongoDB). New design language to facilitate add on library for other developers to build plugins such as:

  • Size (X x Y)
  • Color #------ (hex color scheme)
  • Custom CSS such as border-color, shadow, font-size ...

Tasks for next meeting

  • Task #1: Improve key deliverables software
  • Task #2: More detailed schedule
  • Task #3: Metric for performance of CS297 deliverable

December 4, 2018

Presented Deliverable #4

Learned OSC protocol: Open Sound Control (OSC) is a protocol for communication among computers, sound synthesizers, and other multimedia devices that is optimized for modern networking technology. Bringing the benefits of modern networking technology to the world of electronic musical instruments, OSC's advantages include interoperability, accuracy, flexibility, and enhanced organization and documentation.

Used osc-js library for JavaScript applications (UMD module for Node, Browser etc.) with address pattern matching and timetag handling. Sends messages via UDP, WebSocket or both (bridge mode) and offers a customizable Plugin API for network protocols.

  • Task #1: Write up CS297 report first draft, meet Dec 5 (office hours)
  • Task #2: Get all links w3/accessibility validated
  • Task #3: Deliverable 4 with component rendering
  • Task #4: Upload all deliverables on this website (<100MB)

November 27, 2018

Presented Deliverable #3, Discuss future steps

  • Task #1: Prepare and upload Deliverable 4
  • Task #2: Start working on CS297 Final report draft

November 20, 2018

No meeting

  • Uploaded Deliverable #2
  • Started building custom UI library

November 13, 2018

Showed Deliverable 2 demo.

  • Task #1: Upload Deliverable #2
  • Task #2: Work on and upload Deliverable 3

November 6, 2018

No meeting


October 30, 2018

Discussed deliverable 2 improvements.

  • Task #1: Show deliverable #2 with circles squares
  • No meeting on Nov 20(Tuesday)

October 23, 2018

Discussed deliverable 1 improvements, Unsuccesful demo of Doodle Classifier on simple Images, fix is to use deprecated version 9 of openframeworks and not version 10.

  • Task #1: Clean up links on Blog, Deliverable 1.
  • Task #2: Upload deliverable 1 on github.
  • Task #3: Show Deliverable 2 demo.

October 16, 2018

Spoke about Doodle Classifier and deliverable ideas.

  • Task #1: Complete uploading Deliverable 1, add header description
  • Task #2: Use Doodle Classifier to classifiy American Sign Language.

October 9, 2018

Presented SketchRNN example with rotated sheep

  • Task #1: Upload Deliverable 1, show example.
  • Task #2: Read up and present Doodle Classifier

October 2, 2018

  • Task #1: Make Progress/Complete Deliverable 1: tensorflow/magenta implementation of Drawing Classification
  • Task #2: Read up on CNN, discuss strategies for next week

Questions to ask

  • Example of CNN's being used for audio or times series data processing
  • 3D conv. maps might be useful for handling a sequence of images (video) or volumetric processing.

September 25, 2018

  • Task #1: Better understanding of the math in the Paper
  • Task #2: Train the SketchRNN model to rotate a given image

Research findings

  • I first studied the training scripts provided by the Magenta team to train a sketch-rnn model to learn new data. I was looking for part of the code where the delta x and delta y was being input from the numpy arrays. The reason for this is that so that I could swap these columns to get the desired rotation effect. However on careful review, I found that it would be easier to swap these columns in the dataset which is kind of "hacky"
  • To train the model I first needed a dataset containing train/validation/test examples in the 3-stroke format. Each example in the dataset is stored as list of coordinate offsets such as the following  x,y, and a binary value representing whether the pen is lifted away from the paper.
  • I made a python script to convert the numpy arrays in a pre-existing dataset and interchange the first two columns which are x, y
  • My next step was to now use this new .npz consisting of 3 .npy files which are train.npy, valid.npy and test.npy and train a new sketch RNN model. I had a lot of problems setting up magenta especially with use of tensorflow-gpu. However, Virtual-env to the rescue and I was able to setup the learning for the sketch-rnn using GPU.
  • Once training is completed I will convert the model weights into a sketch RNN js readable JSON format. Jupyter Notebook
  • Run sketch-RNN demo code but this time with my trained model, hopefully we see rotated sheeps.

Interesting reading

  • Numpy: a fundamental package for scientific computing with Python, contains support for large, multidimensional arrays and matrices, along with a large collection of higth-level mathematical functions to operate on these arrays.
  • https://magenta.tensorflow.org/ Magenta is a research project exploring the role of machine learning in the process of creating art and music. Primarily this involves developing new deep learning and reinforcement learning algorithms for generating songs, images, drawings, and other materials. But it's also an exploration in building smart tools and interfaces that allow artists and musicians to extend (not replace!) their processes using these models. Magenta was started by some researchers and engineers from the Google Brain team, but many others have contributed significantly to the project

September 25, 2018

  • Task #1: Present Sketch RNN paper.
  • Task #2: Setup dev environment for Sketch RNN
  • Task #3: Show working example if possible with fixed classes

Interesting articles


September 11, 2018

  • Presented Pix2code, other discussions.
  • Task #1: Need simpler proposal topic description.
  • Task #2: Understand feature vector representation, word2vec
  • Task #3: Present RNN for drawing Classification Paper/Google Article
  • Task #4: Setup Tensorflow Environment on my laptop, show basic demo

Representing things in machine learning

Feature Vector Representation

  • An instance 'x' represents a specific object (thing)
  • Often represented by a N-dimensional feature vector
  • Each dimension is called a feature. Continuous or discrete.
  • "x" is a dot in the N-dimensional feature space
  • Source

Word2Vec

  • Word2vec is a two-layer neural net that processes text and trained to reconstruct linguistic contexts of words.
  • Its input is a text corpus and its output is a set of vectors: feature vectors for words in that corpus.
  • Used for learning vector representations of words, called "word embeddings".
  • Word vectors are positioned in the vector space such that words that share common contexts in the corpus are located in close proximity to one another in the space.

Interesting articles/videos


September 4, 2018

  • Discussed thesis idea, scope and possiblities.
  • Task #1: Need more concrete and defined deliverables
  • Task #2: Read and present pix2code.
  • Task #3: Get more reference research papers.

Interesting articles