Outline

• Using Tensorboard
• In-Class Exercise
• GANs

Introduction

• On Monday, we finished discussing common kinds of preprocessing that might be used for computer vision tasks, namely, we discussed contrast normalization and data augmentation.
• We then talked about neural net approaches to speech recognition using restricted Boltzman machines.
• We then looked at neural net approaches to natural language processsing (NLP).
• Our starting point was n-gram language models, and then we looked at word embeddings.
• We said word embeddings could be used together with an encoder-decoder network to do natural language translation.
• Today, we begin by looking at an important tool to visualize training before considering a popular neural net architecture known as GANs.

Using Tensorboard

• Tensorboard is a visualization tool that comes with Tensorflow that lets you see graphically how training and validation of your neural net is going.
• It works based on log event data that the fit method of your model can be told to generate.
• The tensorboard program consists of a small web server whose default landing page has graphs based on the log data it is configured to server that was written by Tensorflow.
• Since Tensorboard draws graphs for whatever is currently stored in its log directory, if you are training a large model and it is taking minutes or hours, you can visualize how the the training is improving by simply refreshing your Tensorboard web page.
• Let's see how this works by modifying our mnist toy example from a couple weeks back to set it up to write logs...

Tensorboard MNIST Example

import keras
import tensorflow as tf
from keras.datasets import mnist
from keras.models import Sequential
from keras.layers import Dense, Dropout
from tensorflow.keras.optimizers import RMSprop
import numpy as np
import os
import time

os.environ['TF_CPP_MIN_LOG_LEVEL']='2'

#top level directory to log output of training
root_logdir = os.path.join(os.curdir, "my_logs")

#function to compute a path to a sub folder to hold
#logs for a given run
def per_run_logdir():
    run_id = time.strftime("run_%Y_m_%d-%H_%M_%S")
    return os.path.join(root_logdir, run_id)
run_logdir = per_run_logdir()

#create a new callback that will do logging
my_tensorboard_callback = keras.callbacks.TensorBoard(run_logdir)

(x_train, y_train), (x_test, y_test) = mnist.load_data()

# xtrain is originally 28x28 grayscale uint8's between 0-255
x_train = x_train.reshape(60000, 784)
x_test = x_test.reshape(10000, 784)
x_train = x_train.astype('float32')
x_test = x_test.astype('float32')
x_train /= 255
x_test /= 255

y_train = tf.keras.utils.to_categorical(y_train, 10)
y_test = tf.keras.utils.to_categorical(y_test, 10)

model = Sequential()
model.add(Dense(512, activation='relu', input_shape = (784,)))
model.add(Dropout(0.2))
model.add(Dense(512, activation = 'relu'))
model.add(Dropout(0.2))
model.add(Dense(10, activation = 'softmax'))
model.compile(loss = 'categorical_crossentropy',
   optimizer = RMSprop(),
   metrics = ['accuracy'])

history = model.fit(x_train, y_train,
   batch_size = 128, epochs = 20, verbose = 1,
   validation_data = (x_test, y_test),
   callbacks=[my_tensorboard_callback] #tell fit to use callback defined above
   )

Remarks on and Using the Example

• Each time we run our program from the previous slide a new sub folder of my_logs will be created of the form run_some_timestamp.
• In this subfolder, we will get different events written by Tensorflow:
  
• To get Tensorboard to start serving graphs related to these logs, we run it using the commnad:

  tensorboard --logdir=./my_logs --port=8080
  

• We can then point our browser at: http://localhost:8080/ to see graphs such as:

In-Class Exercise

• Test out Tensorboard on your own machine using the code just given.
• Generate log data by conducting three runs.
• Take a screen shot of the Tensorflow epoch_accuracy graphs.
• How similar are the curves? Why aren't they the same?
• Take a screen shot of the Main Graph of your model.
• Please post your answers and reduced file sized screen shots to the Dec 1 In-Class Exercise Thread.

Generative Adverserial Models

• Suppose we want to learn what is a shirt.
• One way would be to try to train on labeled images some of which are shirts, some of which are not.
• This can often be hard as the space of non shirts is potentially huge.
• Goodfellow et al 2014 proposed a method for doing this called Generative Adversarial Models.
• In it, we co-train two models: a generator which tries to generate examples similar to the training class and a discriminator which tries to distinguish between items generated by the generator and real items.
• The generator starts from a small input space and each time it is run on a different input noise vector.
• The discriminator takes as input either an image output from generator or a real image.

Training a GAN

• Each training iteration of a GAN is divided into two phases:
  • In Phase 1, we train the discriminator. A batch of real images is sampled from the training set and is completed with an equal number of fake images produced by the generator. These are appropriately labeled 0 - fake, or 1 - real. The discriminator is trained on this labeled batch for one step, using the binary cross-entropy loss. The generators weight are not touched.
  • In Phase 2, we train the generator. We use the generator to make a batch of fake images. We run the discriminator on this batch to see which one it labels as real. The generator is then trained with a loss functions which is lower if more of the discriminator outputs are real (i.e., mis-labeled).
• The above picture is from "Hands-On Machine Learning with Scikit-Learn, Keras and Tensorflow," Geron 2019. It shows the output of a generator trained on clothing examples after one epoch of training.

Deep Convolutional GANs

• The original GAN paper only generated small images.
• Alec Radford, et al 2015 based on experimentation proposed modifying the original GAN model so as to allow it to work with deeper networks capable of generating higher resolution images.
• They called their architectures deep convolutional GANs (DCGANS).
• In their paper, they argue that architectures that do the following tend to work better when the number of layers increase:
  • Replace any pooling layers with strided convolutions (in the discriminator) and transposed convolutions (in the generator).
  • Use Batch Normalization in both the generator and the discriminator, except in the generator's output layer and the discriminator's input layer.
  • Remove fully connected hidden layers for deeper architectures.
  • Use ReLU activation in the generator for all layers except the output layer, which should use tanh
  • Use leaky ReLU activation in the discriminator for all layers.

Progressive Growing of GANs

• Karras et al. 2018 propose generating small images at the beginning of training, then gradually adding convolutional layers to both the generator and the discriminator to produce larger and larger images (4 x 4, 8 x 8, 16 x 16, ..., 512 x 512, 1024 x 1024).
• To train 2y x 2y generator given a trained y x y generator, we take the images generators by the y x y generator model and up-sample them to 2y x 2y images.
• At the start of training for the 2y x 2y generator, we have both the 2y x 2y generator and the y x y generator generate 2y x 2y images `I_1`, `I_2` (the latter by upsampling).
• We feed a linear combination on these images to the discriminator `alpha cdot I_1 + (1-alpha)cdot I_2`.
• Initially, `alpha` is close 0, but as training progresses we increase it toward 1.
• A similar process is also done to train the 2y x 2y discriminator using the y x y discriminator.