Introduction

Last week, we finished our first pass discussion of training algorithms such as BFGS and Stochastic Gradient Descent.
We discussed why ReLU and its variants are often used for the activation layer for hidden nodes in multi-layer networks.
We then went over the backpropagation algorithm used to train multi-layer neural networks.
We talked about how frameworks to implement neural networks often make use of a computational graph where quantities such as partial derivatives are often memo-ized.
We talked about two common approaches to handling derivatives in such graphs: symbol-to-number differentiation and symbol-to-symbol differentiation.
We then introduced Tensorflow as a framework for handling neural nets and Keras as a convenient front-end to Tensorflow.
We haven't said too much yet about either of these pieces of software except how to install them and to suppress warning if not using the GPU for training.
Today, we are going to look at Keras in more detail. Before we do, let me mention that often in the internet or in book examples of Tensorflow make use of Tensorflow 1, so these examples might need to be altered to get to work in the more recent Tensorflow 2.

Keras Overview

The Keras API can be thought of as consisting of three major components:
1. Core Module - as part of training and testing a neural net, we need to say where it gets its data from, what optimizer it uses, what regularizer (if any is in use), what loss function its uses, and what it should use for metrics for testing and visualization. These activities are all part of the Core modules.
2. Model - these represent whole neural networks. In building a model, we typically want to decide if neural network layers are just a sequential composition or if they are arranged in a more bespoke functional composition.
3. Layer - an individual layer in a network can be one of several different kinds (dense, convolutional, lstm, etc), each with different configuration attributes.

Creating Model

To create a simple model in Keras, we import the relevant modules, and then instantiate a Sequential object and add layers to our model:

from keras.models import Sequential 
from keras.layers import Dense, Activation, Dropout
import os
os.environ['TF_CPP_MIN_LOG_LEVEL']='2' #suppress no GPU warnings

num_classes = 10 #if using MNIST digits 0-9
model = Sequential()  
model.add(Dense(512, activation = 'relu', input_shape = (784,)))
model.add(Dropout(0.2)) 
model.add(Dense(512, activation = 'relu'))
model.add(Dropout(0.2)) 
model.add(Dense(num_classes, activation = 'softmax'))

Remarks on Code

The above code presents a simple model that might be useful as a starting place for handwritten character recognition using the MNIST (Modified National Institute of Standards and Technology) database of digits 0-9.
Once we have instantiated model = Sequential(), we use its add method to add the layer's we want.
Notice for the first layer we need to say how it will receive inputs ( input_shape = (784,) ). The number 784 = 28*28 the image size of our digits, where we have reshaped them so they are all in a line.
In the above example, we have two layer types: Dense and Dropout.
A Dense layer is fully connected with its inputs. The first component of the Dense constructor says how many neurons we want in the layer.
So for the first layer in our network, we want 512 neurons, each with 784 inputs. So after this layer we have 512 outputs.
A Dropout layer by default has the same number of inputs as outputs. During training, it either copies and scales the inputs to the outputs or with the probability `r` given in the constructor sets the output to 0. The scaling factor is `1/(1 - r)` for the non-zero'd outputs. Dropout layers are often used to prevent overfitting.
In our example, above `r = 0.2` and the second layer would have 512 inputs, 512 outputs.
Notice we can specify the activation type of our neural net layer using the activation attribute. In the above, we use 'relu' for the 1st and 3rd layer, and 'softmax' for the 5th layer. Other possibilities are: linear, tanh, sigmoid, etc.

Quiz

Which of the following is true?

1-hot vectors are useful for training softmax layers.
Stochastic Gradient Descent is based on the Newton Raphson algorithm for finding extrema.
Training for an epoch would involve a full pass over the training data at most one time.

Initializers, Regularizers, Constraints

The model of a couple slides back used the default setting for how weights are set-up.
Three important constructor parameters used to control weight are:
1. Initializers these are used to specify the initial values for weights.
2. Regularizers these are often used to penalize in the loss function having weights which are too large or too small.
3. Constraints these are used to clamp the weight to within a certain range.

We can specify these constructor properties with code like:

from keras.models import Sequential 
from keras.layers import Activation, Dense 
from keras import initializers 
from keras import regularizers 
from keras import constraints
model = Sequential() 
model.add(Dense(512, input_shape=(784,), kernel_initializer = 'he_uniform', 
   kernel_regularizer = None, kernel_constraint = 'MaxNorm', activation = 'relu'))

The above says use the built-in he_uniform initializer. This initializer comes from a paper of He, et al 2015, where they say weights should initially be randomly chosen in the range [-limit, limit] where limit is `sqrt(6 / mbox(fan_in_of_layer))`.
It says do not use a regularizer, but do constrain the weights to less than limit.

It is completely possible to make your own initializers, regularizers, and constraints. For example, we can do:

my_init_zero = initializers.Zeros() 
    #when layer needs an input value this initializer always outputs 0.
my_init_one = initializers.Ones()
    #when layer needs an input value this initializer always outputs 0.
my_init_half = initializers.Constant(value = 0.5)
    #when layer needs an input value this initializer always outputs 0.5.
my_init_normal = initializers.RandomNormal(mean=0.0, stddev = 0.05, seed = None) 
    #normal distribution
model.add(Dense(512, input_shape=(784,), kernel_initializer = my_init_zero))

More on Models

Once we have built a model, we can see its components by looking at its .layers attribute:

>>>exec(open("simple_model.py").read()) #code from before
>>>model.layers
[<keras.layers.core.Dense object at 0x107826880>, 
<keras.layers.core.Dropout object at 0x1567bd9d0>,
<keras.layers.core.Dense object at 0x1567bdbb0>, 
<keras.layers.core.Dropout object at 0x15681c040>,
<keras.layers.core.Dense object at 0x15681c5e0>]

Similarly, model.inputs, and model.outputs would provide information about the inputs and outputs of the model.
There are set_weights, and get_weights methods that let you get and set weight from a numpy array.

If you want pretty-printed information you can use model.summary() :

>>> model.summary()
Model: "sequential"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
dense (Dense)                (None, 512)               401920    
_________________________________________________________________
dropout (Dropout)            (None, 512)               0         
_________________________________________________________________
dense_1 (Dense)              (None, 512)               262656    
_________________________________________________________________
dropout_1 (Dropout)          (None, 512)               0         
_________________________________________________________________
dense_2 (Dense)              (None, 10)                5130      
=================================================================
Total params: 669,706
Trainable params: 669,706
Non-trainable params: 0
_________________________________________________________________

Using a Model

The following four methods of a Keras model are among the most important:
1. compile - used to set up the learning process of the model
2. fit - used to train the model on a training data set
3. evaluate - used to train the model on a test data set
4. predict - used to get a prediction from the model on a new data item
We will go over these in detail next day, since predict is somewhat easy let's see how to use it on a pre-trained model.

Using a Previously Trained Model to Make Prediction

Keras comes with several previously trained models and built-in datasets.

To end today, let's look at code to use these:

from tensorflow.keras.applications.resnet import ResNet101
from tensorflow.keras.preprocessing import image
from tensorflow.keras.applications.resnet import preprocess_input, decode_predictions
import numpy as np
import os
os.environ['TF_CPP_MIN_LOG_LEVEL']='2'

#we are using a pre-built 101 layer network whose weights 
#were trained on ImageNet dataset
model = ResNet101(weights='imagenet')

#a picture of me
img_path = 'myphoto.jpg'
img = image.load_img(img_path, target_size=(224, 224))
x = image.img_to_array(img)
x = np.expand_dims(x, axis=0)
#get the image in the correct format
x = preprocess_input(x)

#make a prediction
preds = model.predict(x)
# decode the results into a list of tuples (class, description, probability)
# (one such list for each sample in the batch)
print('Predicted:', decode_predictions(preds, top=3)[0])

Keras and Tensorflow

Outline