Introduction

On Monday we were discussing how to build neural nets using the Keras API on top of Tensorflow.
We said the Keras API has three main parts: Core Modules, the Model API, and the Layer API.
We showed how to add Layers of different types (Dense, Dropout) to build a neural net using the Sequential class in the Model API.
We said how to configure a Layer's activation type, initializers, regularizers, and constraints.
We verbally went over the MaxPool2D and Conv2D layer types (not on last day's slides).
We showed how we can see a summary of the model we built as well as how to get and set its weights.
Finally, we said how Keras comes with a collection of pre-trained networks and training data, and we demo'd how to use one these models to make predictions of image content.
Today, we go over the steps from compiling, training, and testing a model.

Compiling a Model

Now that we've created a model, the next step is to compile it.
Compilation is where we specify what loss function we'll use, what optimization method we'll use, and how we will measure the accuracy of our model when we are done training.

Below is our code so far:

from keras.models import Sequential
from keras.layers import Dense, Activation, Dropout
from keras import losses
from keras import optimizers
from keras import metrics

import os
os.environ['TF_CPP_MIN_LOG_LEVEL']='2'

num_classes = 10 #if using MNIST digits 0-9
model = Sequential()
model.add(Dense(512, activation = 'relu', input_shape = (784,)))
model.add(Dropout(0.2))
model.add(Dense(512, activation = 'relu'))
model.add(Dropout(0.2))
model.add(Dense(num_classes, activation = 'softmax'))
model.compile(loss = 'mean_squared_error',
   optimizer = 'sgd', metrics = [metrics.categorical_accuracy])

As you can see we are using 'mean_squared_error' for the loss function, stochastic gradient descent (sgd) for the optimization method, and we are using accuracy at predicting the correct category for the metric for how good our model is.
Other choices for loss function include categorical_crossentropy, kullback_leibler_divergence, etc.
Other choices for optimizer include adam, nesterov_adam, etc.
Other choices for metrics include top_k_categorical_accuracy, cosine_proximity, etc.

Fitting a model

A model's fit function is used to actually perform training:

model.fit(x_train, y_train, batch_size = some_num,
  epochs = some_num2, validation_data = (x_val, y_val))

Here x_train and y_train are numpy arrays that depend on the net being trained. For example, if we are using 28 x 28 grayscale images, then x_train might have shape (28, 28) or (784,) depending on whether we flattened it or not.
y_train might be an array of binary numbers for a binary classifier or if we have a fixed list of categories, it could be 1-hot numbers in that range.
batch_size is the size of batches to use for stochastic gradient descent.
epochs is the number of forward/backward passes over the whole data set to perform.
validation_data is what data to use for testing.

Generic MNIST Digit training example

Below is complete code to do the most generic kind of data using the MNIST digit dataset that comes with Keras:

import keras
import tensorflow as tf
from keras.datasets import mnist
from keras.models import Sequential
from keras.layers import Dense, Dropout
from tensorflow.keras.optimizers import RMSprop
import numpy as np

import os
os.environ['TF_CPP_MIN_LOG_LEVEL']='2'

(x_train, y_train), (x_test, y_test) = mnist.load_data()

# xtrain is originally 28x28 grayscale uint8's between 0-255
x_train = x_train.reshape(60000, 784)
x_test = x_test.reshape(10000, 784)
x_train = x_train.astype('float32')
x_test = x_test.astype('float32')
x_train /= 255
x_test /= 255

y_train = tf.keras.utils.to_categorical(y_train, 10)
y_test = tf.keras.utils.to_categorical(y_test, 10)

model = Sequential()
model.add(Dense(512, activation='relu', input_shape = (784,)))
model.add(Dropout(0.2))
model.add(Dense(512, activation = 'relu'))
model.add(Dropout(0.2))
model.add(Dense(10, activation = 'softmax'))
model.compile(loss = 'categorical_crossentropy',
   optimizer = RMSprop(),
   metrics = ['accuracy'])

# if we don't want to see training progress info set verbose to 0
history = model.fit(x_train, y_train,
   batch_size = 128, epochs = 20, verbose = 1,
   validation_data = (x_test, y_test))

Example Training Output, Evaluation

During training we had verbose set 1, so Keras provides quite detail output:

Epoch 1/20
469/469 [==============================] - 4s 7ms/step 
   - loss: 0.2480 - accuracy: 0.9228
   - val_loss: 0.1335 - val_accuracy: 0.9592
Epoch 2/20
469/469 [==============================] - 3s 7ms/step
   - loss: 0.1022 - accuracy: 0.9692
   - val_loss: 0.1041 - val_accuracy: 0.9686
Epoch 3/20
...
Epoch 20/20
469/469 [==============================] - 3s 7ms/step
   - loss: 0.0164 - accuracy: 0.9957
   - val_loss: 0.1116 - val_accuracy: 0.9843

The loss and accuracy above are for the training data, and val_loss, and val_accuracy are for the test data.

After training you can also evaluate your model on test data using:

score = model.evaluate(x_test, y_test)
#this will default to verbose = 1, for verbose = 0
score = model.evaluate(x_test, y_test, verbose = 0)
print('Test loss:', score[0])
print('Test accuracy:', score[1])

In-Class Exercise

Make a Numpy array of 10 data item with values between 1-5.
Verify that to_categorical is doing what claimed.
Please post your solution to the Oct 27 In Class Exercise Thread.

Doing Cross Validation

To do cross validation, we want to split our training data into different subsets using one portion for the training the other for the validation.

We can either code this ourselves or we can use sci-kit learn:

from sklearn.cross_validation import StratifiedKFold
#...
def create_model():
  # code to create model

def train_and_evaluate(model, data[train], labels[train], 
  data[test], labels[test)):
  # code to train and fit model
#...
x_values, y_values  = load_model()
five_fold = StratifiedKFold(n_splits=5)
for train, test in five_fold.split(x_values, y_values):
  model = None #get rid of results of previous training
  model = create_model()
  running_totals = train_evaluate(model, x_values[train], y_values[train],
    x_values[test], y_values[test])
#use running totals to compute averages losses and accuracy

Prediction, Loading Saving Models

If we wanted to do predictions on the model just trained, we can do as we did last day in our Resnet example and use the predict method.

For example, we can see how our train model would make a prediction on the first training data item:

>>> from matplotlib import pyplot as plt
>>> pixels = xtrain[0].reshape((28,28))
>>> plt.imshow(pixels, cmap='gray')
>>> plt.show() #this draw the 0th training item so can see is a 5
>>> x = x_train[0]
>>> x = np.expand_dims(x, axis=0)
>>> model.predict(x)
array([[0.0000000e+00, 4.7968505e-27, 3.7603754e-31, 8.6074351e-07,
        0.0000000e+00, 9.9999917e-01, 2.5081769e-35, 2.4804149e-33,
        9.4426353e-29, 6.9787635e-27]], dtype=float32)

Notice 5 is also has predicted with the highest probability above.

To save a model for future use we can use the model's save method. This saves the weight either in a tensorflow format (several files) or a single file h5 format:

model.save("my_model_path")
  #this would save in tensorflow format
model.save("my_model_path.h5")
  #would detect .h5 extension and save as h5 format
# or could do model.save(path, save_format='tf') #or h5

To load a model, we can do:

from keras.models import load_model
del model
model = load_model('my_model_path')

More on Layer Types

We already had a slide where we talked about the Dense and Dropout layer types.
For the homework, we also need to make use of a convolution layer and it probably makes sense to have max pool layers.

A 2D convolutional layer might be added to our model with a line like:

model.add( Conv2D(32, kernel_size = (5, 5), strides = (1, 1),
   activation ='relu'))

The first arguments tells the number of filters to create. If you are working with images, you can think of the output of one filter as being one image where the image has been process by a single filter. So 32 above would be 32 images out, each of which has had a different 5x5 filter applied to 5x5 subportions of the original image where we step between applications of the filter to the left and right according to the stride.
Notice the above layer would have 32*5*5=800 weights we are training even though the output might be roughly 32*original_image size.
So convolutional layers require way fewer neurons than dense layers typically.
As a rule of thumbs we want to keep the number of neurons we train at each layer relatively constant, so after increasing the number of output images, we typically want a downsampling layer to reduce the image sizes going in to the next layer. We can use a max pool layer to do this:
```
model.add( MaxPooling2D((2, 2)))
```
Such a layer breaks an 2D array of inputs into an (x,y) sub-portion (2x2 in the above) and outputs only the maximum values of those x*y points.
So a 2x2 maxpooling layer would reduce a 2D array size by 4.

Regularization for Deep Learning

We would like our trained neural nets to perform well not only on the training data, but also on new inputs.
Strategies to reduce the test data (generalization) error are known collectively as regularization.
Regularization strategies might involve putting constraints or penalties on parameter (weight) values.
These might improve training performance, but might also be used to encode prior information we have about what the trained values should look like.

Parameter Norm Penalties

Parameter Norm Penalties are a popular technique for regularization that comes from linear regression in statistics.
Suppose we have a loss function: `J(vec{theta}, vec{mathbb(x)}, mathbb{y})`.
We create a new loss function where we add on to this a function that increase with the weights: `\tilde{J}(vec{theta}, vec{mathbb(x)}, mathbb{y}) = J(vec{theta}, vec{mathbb(x)}, mathbb{y}) + \alpha Omega(vec{theta})`.
`alpha` in the above is a hyperparameter controlling the amount of regularization.
Some popular choices for `Omega` are:
- `1/2||vec{theta}||_2^2` - using this is called `L_2`-parameter regularization also known as ridge regression or Tikhonov regularization.
- `||vec{theta}||_1` - using this is called `L_1`-parameter regularization.
In both cases, using these regularizations will make the resulting loss function larger as the length of the weight vector gets larger.
When all the components of the weight vector all small, the regularization term will tend to be small compared to the loss function.
So we are favoring small weight solutions to our training problem. This will tend to avoid overfitting from large weights with weird cancellations.
`L_1`-parameter regularization tends to result in more of the weights getting 0 values after training. I.e., the weight vector becomes more sparse.
On the other hand, the behavior of this kind of regularization is harder to analyze.

Regularization for Deep Learning

We would like our trained neural nets to perform well not only on the training data, but also on new inputs.
Strategies to reduce the test data (generalization) error are known collectively as regularization.
Regularization strategies might involve putting constraints or penalties on parameter (weight) values.
These might improve training performance, but might also be used to encode prior information we have about what the trained values should look like.

Parameter Norm Penalties

Parameter Norm Penalties are a popular technique for regularization that comes from linear regression in statistics.
Suppose we have a loss function: `J(vec{theta}, vec{mathbb(x)}, mathbb{y})`.
We create a new loss function where we add on to this a function that increase with the weights: `\tilde{J}(vec{theta}, vec{mathbb(x)}, mathbb{y}) = J(vec{theta}, vec{mathbb(x)}, mathbb{y}) + \alpha Omega(vec{theta})`.
`alpha` in the above is a hyperparameter controlling the amount of regularization.
Some popular choices for `Omega` are:
- `1/2||vec{theta}||_2^2` - using this is called `L_2`-parameter regularization also known as ridge regression or Tikhonov regularization.
- `||vec{theta}||_1` - using this is called `L_1`-parameter regularization.
In both cases, using these regularizations will make the resulting loss function larger as the length of the weight vector gets larger.
When all the components of the weight vector all small, the regularization term will tend to be small compared to the loss function.
So we are favoring small weight solutions to our training problem. This will tend to avoid overfitting from large weights with weird cancellations.
`L_1`-parameter regularization tends to result in more of the weights getting 0 values after training. I.e., the weight vector becomes more sparse.
On the other hand, the behavior of this kind of regularization is harder to analyze.

Regularization and Under-Constrained Problems

One common problem in neural net training and in linear regression (curve fitting) is we have too many parameters that are being trained for as compared to the amount of data we have to train.
You can think of a parameter norm penality:
`\tilde{J}(vec{theta}, vec{mathbb(x)}, mathbb{y}) = J(vec{theta}, vec{mathbb(x)}, mathbb{y}) + \alpha Omega(vec{theta})`.
as enforcing an additional constraint on the data. I.e., that the weights/learned parameters must be smaller.
These kind of constraints typically reduce the dimensionality of the space we are looking for weights in and make it more likely we can find a solution.
The book mentions how this idea of a penalty term as a constraint to an optimization problem is a useful tool in the design of many algorithms, not just in neural net training, but also in other areas of learning and in linear algebra.
For example, suppose one wants to have a function that behaves like a matrix inverse for the case where a matrix `X` is potentially non-square and singular.
To make the matrix square and symmetric, we can use `X^TX`.
As `X^TX` might be singular, we add a term `\alpha I`, where `I` is the identity matrix, giving `X^TX + \alpha I` which is invertible (for most `alpha`).
We can then take our whole approximate inverse to be the Moore-Penrose pseudo-inverse:
`X^+ = lim_{\alpha -> 0+}(X^{T}X + alpha I)^{-1}X^T`.
The limit in `alpha` can be thought of algorithmically using some kind of weight decay protocol when performing a task like linear regression.
Notice this idea also showed up in the BFGS algorithm when we wanted to compute a learning rate for our partials.

Data Augmentation

The easiest way to improve our learning algorithms and make them behave better under more situations is to train them with more data.
If the amount of data is limited, we can try to create useful fake data, and add it to the data set.
For some tasks like image classification, we know the data should be invariant under a wide variety of transformations.
For example, for digit recognition, a tiny `A` should be classified as an `A`, just as much as a larger sized `A`.
When generating the fake data, we should be careful not to apply transformations that don't preserve the classes we want to label. For example, a 180 degree rotation maps a `b` to a `q`, and so might not be appropriate for augmented data in image classification.
Data augmentation can cause problems when trying to compare training algorithms: If algorithm `A` was trained with noise and `B` without, and `A` performed better, you can't say whether it performed better because it was a better algorithm.

Noise Robustness

We can improve generalization by adding noise to various parts of our training set up:
- Inputs Perturbation: Add noise to the inputs (say drawn from a Gaussian) before using the inputs for training. (This is another kind of data augmentation.)
- Weights Perturbation:
  - Suppose are trying to train an `l`-layer perceptron network to compute `hat y: \vec{x} -> RR` using with a least squares cost function:
    `J =E_{p(x,y)}[(hat{y}(\vec{x}) - y)^2]`,
    and using data `{ (\vec{x}^{(1)}, y^{(1)}), ..., (\vec{x}^{(m)}, y^{(m)})}`.
  - We could add a perturbation `\epsilon_{\mathbf W}` to the weights drawn from a multivariate Gaussian centered at `vec{0}` of variance `eta I`.
  - The new objective function is thus:
    `tilde{J}_W = E_{p(x,y, \epsilon_{\mathbf W} ) }[(hat{y}_{ \epsilon_{\mathbf W} }(\vec{x}) - y)^2]`
    `\qquad = E_{p(x,y, \epsilon_{\mathbf W})}[hat{y}_{ \epsilon_{\mathbf W}}(\vec{x})^2 - 2y \cdot hat{y}_{ \epsilon_{\mathbf W}}(\vec{x}) + y^2]`
  - For small `eta`, minimization of `tilde{J}_W` is thus equivalent to an additional regularization term:
    `eta E_{p(x,y, \epsilon_{\mathbf W})}[||\grad_{\vec{W}}hat{y}(\vec{x})||^2].`
  - This encourages the parameters of to go to regions of the parameter space where small perturbations of the weights have a smaller influence on the output. I.e., making the trained model less sensitive to slight variations in the weights and so more robust.
- Outputs Perturbation: Real data tends to have errors. We can model this by assuming for some `epsilon >0` that each data item is labeled correctly `1 -\epsilon` of the time, and incorrectly otherwise. This can then be used to code up a modified cost function incorporating this correction.

Training and Evaluation using Keras, Regularization, Data Augmentation

Outline

Introduction

Compiling a Model

Fitting a model

Generic MNIST Digit training example

Example Training Output, Evaluation

In-Class Exercise

Doing Cross Validation

Prediction, Loading Saving Models

More on Layer Types

Regularization for Deep Learning

Parameter Norm Penalties

Regularization for Deep Learning

Parameter Norm Penalties

Regularization and Under-Constrained Problems

Data Augmentation

Noise Robustness