HW#3 --- last modified October 26 2021 00:32:23.

Solution set.

Due date: Nov 1

Files to be submitted:
  Hw3.zip

Purpose: To learn about CNNs and neural net frameworks. To realize non image data can often be transformed into something that looks image-like.

Related Course Outcomes:

The main course outcomes covered by this assignment are:

CLO4 -- Be able to select neural network layers type to build a network suitable for various learning tasks such as object classification, object detection, language processing, planning, policy selection, etc.

CLO5 -- Be able to select an appropriate regularization technique for a given learning task.

CLO6 -- Be able to code and train with a library such as Tensorflow or Pytorch a multi-layer neural network.

CLO7 -- Be able to measure the performance of a model, determine if more data in needed, as well as how to tune the model.

Description:

For this homework, I want you to build and train a simple multi-layer neural network in Keras and TensorFlow. Your network will be used to classify collisions based on 1D particle interactions into one of four classes: NO_COLLISION, INELASTIC, ELASTIC, INELASTIC_DECAY.

Imagine that we have a particle detector for a 1-dimensional world. (There are lots of stories about 2D-worlds such as Flatland or Planiverse.) Particles are shot at each other from particle guns at opposite ends of the detector. The detector has 64 sensors arranged in a line between the guns. These sensors out 1 if a particle is currently passing and 0 otherwise. Particles can vary in length from 1 sensor long to 5 sensors long. Similarly, the guns can shoot particles out at velocities between 1 and 5 sensor traversals/time step. The guns are synchronized to fire within at most 5 times steps of each other. An experiment consists in watching a sequence of 64 time steps from the firing of the first gun to fire. The states of the 64 sensors at each time step can be laid out in sequence to form a 64 x 64 image.

You will write two programs as part of your homework: mini_geant.py (name comes from Géant Particle Simulator Software) and interact_net.py.

mini_geant.py is used to generate 1D experiment images and will be run from the command line with a syntax like:

python mini_geant.py num_experiments min_start_difference max_start_difference min_size max_size min_velocity max_velocity output_folder

To generate an experiment, it selects, each particle's length uniformly at random between the min_size and max_size, it selects the particles speeds from between the min_velocity and max_velocity, and it selects the difference between the starting times of the left and right particle according to min_start_difference and max_start_difference. It then computes a sequence of steps of the particles motions from either end of the sensor according to their velocities. If any portion of both particles occupies the same sensor position at the same time a collision occurs. As we are making up the physics of our world, this is equally like to be a collision of any of the four types: INELASTIC, ELASTIC, INELASTIC_DECAY. In an inelastic collision, the two particles become one particle of size the sum of their sizes and of velocity= (vel_1*size_1 + vel_2*size_2)/(size_1+size_2). In an ELASTIC collision, particle 1's velocity after collision is (size_2*vel_2)/size_1 and particle 2's velocity is (size_1 * vel_1)/size_2. In the case of INELASTIC_DECAY, when the particles initially collide like in the INELASTIC case. Then after an amount of time uniformly chosen between min_start_difference and max_start_difference, the single particle splits into two particles with velocities according the ELASTIC case. After simulating for 64 time steps, mini_geant.py uses the 64 x 64 array it has created to make an image in output_folder with name experiment_no_collision_type.png. For example, an image name might be 55_INELASTIC.png to indicate it was the 55th experiment and the type of collision that occurred was inelastic.

The program interact_net.py is supposed to be a neural net written using Keras and Tensorflow that can correctly classify 1D particle interactions. This program's command line signature should look like:

python interact_net.py mode model_file data_folder

Here mode is either train, in which case your program will use all the data for training; 5fold, in which case your program will do 5-fold cross-validation training and testing; or test, in which case your program will use the provided model and the data folder to do testing only. model_file is a file to read or write the trained weights of your program to. In the case of either train or 5fold mode your program should write a model to this file. For 5fold, as you really generate five models during this process, just write the last one. For test, the neural net in your program should use weights in model_file when performing testing. The data_folder is the name of a folder that your program will look for .png files in. Your program should train on all png in the data_folder (you don't need to train on files in sub folders).

Here are some additional requirements on interact_net.py:

1. The neural net trained/used in testing by your program should be implement in Keras and Tensorflow.
2. It should consist of between two and five CNN layers (your choice of maxpooling), the followed by a dense layer and a softmax layer.
3. Hidden layers should use rectified linear units.
4. Models should be trained using stochastic gradient descent, but no momentum. The mini-batch size should be a configurable constant set at the top of your program. Your program when it reads a string should determine its label before giving it to Tensorflow for training.
5. You can assume all the data in a folder can be read in to memory, so you can read everything in first. As your program processes data, it should write messages to the console indicating how much progress it has made at least every thousand items processed (make sure 1000 is some multiple of the mini-batch size). It should also write a message "Processing complete!" followed by the total number of items trained on and the total number of items tested on.
6. If the mode is either 5fold or test your program should output the overall accuracy over the test data, the rates from the confusion matrix, the total runtime for training, and the total runtime for testing.

Given the above data sets I want you to design and conduct experiments which follow the guidelines for experiments from the Sep 29 Lecture and which answer the following questions:

1. What is the effect of training set size on how good the trained model is? How does this compare to the number of weights that your model has?
2. What is the difference between training on data chosen completely at random versus on well chosen examples? What is the difference in choosing the testing data at random versus data that is likely to be equally representative of the classes that we are training for?
3. What is the difference in accuracy in using cross-validation for testing versus using separate test data.

Put your experiment writes up in Hw3.pdf which you should include in your Hw3.zip file.