Outline

• LSTMs
• Quiz
• Neural Network Tuning

Introduction

• Last week, we began talking about recurrent neural networks (RNN).
• We said these networks were useful for handling sequential or temporal data.
• We gave three example architectures for such neural nets:
  1. An RNN that produces an output at each time step and has recurrent connections between hidden units.
  2. An RNN that produces an output at each time step and has recurrent connections only from the outputs at one step to the hidden units at the next.
  3. An RNN with recurrent connections between hidden units, that reads the entire sequence, and then produces a single output
• We talked about how to represent the back connections in an RNN in an unfolded way, and how to do training using back propagation through time.
• We then talked about some other RNN models: bidirectional RNNs and Encoder-Decoder architectures which were useful for speech and text processing.
• Today, we look at one last RNN unit type and how it solves a problems with training RNNs.

Long Term Dependencies

• One problem with training with RNNs is that unfolding them multiple steps can have the effect of matrix powering.
• This was mentioned before when we described some of the challenges of NN Training as an Optimization problem.
• If we assume for the moment that our activation function is the identity, then computing an RNN layer gives a recurrence like:
  `vec{h}^{(t)} = W^{top}vec{h}^{(t-1)}`
• So `vec{h}^{(t)} = (W^t)^{top}vec{h}^{(0)}`.
• If `W` has an eigenvalue decomposition, then `W = Q\Lambda Q^{-1}`, where `Lambda` is a diagonal matrix, so the above becomes:
  `vec{h}^{(t)} = (Q^{-1}\Lambda^t Q)vec{h}^{(0)}`
  and we see if the largest magnitude of W's eigenvalues is greater than 1, the value of `h` will tend to explode, and if it's less than `1` the values will tend to go 0 rapidly.
• This observation explains why it can be hard to train RNNs if `tau` is large.
• Let's look at a kind of RNN gate that addresses this problem.

LSTMs

• Long Short Term Memory gates were first introduced by Hochreiter and Schmidhuber 1997 where they gave six example experimental problems for sequential data where it performed better than standard RNNs, basically, because of avoiding the long term dependency problem mentioned on the last slide.
• Their model was later improved by Gers et al 2000, and others.
• LSTMs have been used for handwriting recognition, speech recognition, handwriting generation, machine translation, image captioning, and parsing.
• To avoid long term dependencies, LSTMs make use of a memory cell which can remember the previous state.
• Unlike normal RNNs, this cell's value is also controlled by forget lines that can roughly reset the memory of the previous state to 0, thus, preventing the powering issue of the previous slide.

LSTMs

• The above diagram shows the parts of an LSTM gate and how it computes.
• We imagine that internally it has a memory cell which can be used to remember the previous state. Its input, forget gates control how this cell is written to, and the value of this cell, the inputs, and the previous outputs control the next output.
• Its major components are input (i), forget (f), cell (c), output (o) gates, the above diagram as equations is given below (Graves 2013): \begin{eqnarray*} \vec{i}_t &=& \sigma_i(\vec{W}_{xi}\vec{x}_t + \vec{W}_{yi}\vec{y}_{t-1} + \vec{W}_{ci} \odot \vec{c}_{t-1} + \vec{b}_i)\\ \vec{f}_t &=& g(\vec{W}_{xf}x_t + W_{yf}\vec{y}_{t-1} + \vec{W}_{cf} \odot \vec{c}_{t-1} + \vec{b}_f)\\ \vec{c}_t &=& \vec{f}_t \odot \vec{c}_{t - 1} + \vec{i}_t \odot \vec{tanh}(\vec{W}_{xc}\vec{x}_t + \vec{W}_{yc}\vec{y}_{t-1} + \vec{b}_c)\\ \vec{o}_t &=& g( \vec{W}_{ox}\vec{x}_t + \vec{W}_{oy}\vec{y}_{t-1} + \vec{W}_{oc}\vec{c}_t + \vec{b}_o)\\ \vec{y}_t &=& \vec{o}_t \odot \vec{tanh}(\vec{c}_t) . \end{eqnarray*}
• In the above `g` is `(1+e^{-x})^{-1}`, `\odot` is Hadamard product, `\sigma_i` is a sigmoid function, `\vec{b}_{i}`, `\vec{b}_{o}`, `\vec{b}_{c}` are trainable bias weights, and matrices of the form `\vec{W}_{-i}`, `\vec{W}_{-o}`, `\vec{W}_{-c}`, `\vec{W}_{-f}` where `-` is `x`,`y`, or `c` are trainable weight matrices for the input, forget, cell, output gates for the respective vectors `\vec{x}`, `\vec{y}`, or `\vec{c}`.

Optimization for RNN training

• Before we leave the topic of RNNs, we consider techniques that prevent the gradient from becoming too large. (LSTMs and other kinds of gated recurrent units can prevent it from becoming too small.)
• To prevent the gradient from becoming too large one can clip the gradient.
• Some ways to do this are:
  1. Clip the parameter gradient from a minibatch element-wise just before the parameter update. (This is simpler).
  2. Clip the norm `||vec{g}||` of the gradient `vec{g}` just before the parameter update using the rule:
     if `||vec{g}|| > v` (where `v` is our gradient threshold), set `vec{g} := (vec{g}v)/||vec{g}||` (This is more complicated, but preserves gradient direction).
  3. Take a random step of size the gradient threshold.
• It turns out each of the above works roughly as well as the other.

Quiz

RNNs in Keras

• Keras has several built-in RNN and Related Layer types:
  1. keras.layers.SimpleRNN - a fully-connected RNN where the output from previous timestep is to be fed to next timestep.
  2. keras.layers.GRU (gated recurrent units) (Cho et al., 2014) - GRUs are like LSTMs, but have slightly fewer parameters.
  3. keras.layers.LSTM - Implements the LSTM gates we just discussed.
  4. keras.layers.Embedding - These aren't really recurrent networks, but are often used with them. An embedding is a trainable layer taking large vectors (for example, a vector indicating which words from a vocabulary appear in a sentence) to a small dimensional space.
  5. keras.layers.Bidirectional. This class is used in conjunction with other RNN layers to make bidirectional networks.
• Below is a simple example of using an LSTM layer:

  model = keras.Sequential()
  # Add an Embedding layer expecting input vocab of size 1000, and
  # output embedding dimension of size 64.
  model.add(layers.Embedding(input_dim=1000, output_dim=64))
  
  # Add a LSTM layer with 128 internal units.
  model.add(layers.LSTM(128))
  
  # Add a Dense layer with 10 units.
  model.add(layers.Dense(10))
  
  model.summary()
  

• Here is an example of a bidirectional network:

  model = keras.Sequential()
  
  model.add(
      layers.Bidirectional(layers.LSTM(64, return_sequences=True), input_shape=(5, 10))
  )
  model.add(layers.Bidirectional(layers.LSTM(32)))
  model.add(layers.Dense(10))
  
  model.summary()
  

• The RNN layer supplied to Bidirectional is added to the network twice, once taking inputs in the forward direction, once in the backward direction.
• The output of the bidirectional layer is the concatenation of the outputs of the two directions.

Neural Nets in Practice

• We now want to turn our attention to how to go about designing, building, and tuning a neural net for some specific purpose.
• The book discusses a design process based on Ng (2015), and gives a running example based on address number transcription on houses passed by the Google Street View cars.
• Google solved this problem using a CNN based on the Street View Car photographs, and used the results to add to the Google Map database the address at the correct location on the maps.
• Ng's process consists of the following:
  • Determine your goals What error metrics should be used? What are your target values for these metrics? Choose these values based on the problem at hand.
  • Establish a working end-to-end pipeline as early as possible Make use of baseline models related to your problem and improve from there.
  • Instrument your system well so you can determine bottlenecks in performance. Diagnose which components are performing worse than expected and whether performance is due to overfitting, underfitting, or a defect in the data or software.
  • Repeatedly make incremental changes. Gather new data, adjust hyperparameters, change algorithms, based on finding from your instrumentation.

Performance Metrics

• Your choice of error metric will guide how you create your neural net as it influences each of the remaining items in Ng's model.
• Because we are using stochastic methods like SGD we usually can't achieve a 0 error rate.
• We are also limited by data and training time.
• One source of figuring out the performance we might target and what metrics to use is based on looking at previous published papers for related problems. This can give you a realistic initial estimate of your error rate.
• We have already seen some metrics to measure performance such as accuracy, confusion matrix components, and ROC curves.
• Sometimes though you will need more advanced metrics:
  • This might be because it is more costly to make one kind of error than another. For example, for email it is worse to falsely label legitimate mail spam, than vice-versa. So we might want to use a weighting of the FPR with the FNR.
  • This might be because we want to detect rare events. Suppose we want to detect a 1 in a million image cancer. It is very easy to achieve a high accuracy by just reporting in all cases the cancer is not present. One way to measure this is using precision (fraction of detections reported by the model that were correct) and recall (the fraction of true events that were detected). There are various ways to combine these into one measure such as PR curve, and `F_1` score := `frac(2 cdot R cdot P)(R+P)`.
  • This might be because, you would like your system to also estimate when it can't make a decision. For example, with Street View it was important to only add the data to Google maps if the system was highly confident the address was correct. In their case, they had a metric coverage which estimated the number of addresses added to the map/total number of addresses detected and they played this value versus accuracy.
  • This might be because of how the system will be used. For example, an AI for ad placement might use click-through rate as a metric.

End-to-End Pipeline, Default Baseline Models

• The next step in Ng's process is to come up with an initial architecture/pipeline for your network.
• First, you should choose the general category of model based on your data:
  • If you want to perform supervised learning with fixed-size vectors as input, use a feedforward network with fully connected layers.
  • If the data has a topological structure such as an image, try a CNN, probably using ReLU or a variant as your activation function.
  • If your input or output is a sequence, use an LSTM net.
• Once you have your base model, a reasonable choice for optimization algorithm is SGD with momentum using a decaying learning rate or Adam.
• You should use some mild form of regularization such as early stopping, dropout, and batch normalization (normalize the values in hidden layer in minibatches around their variance before training with them).
• You might want to use a previously trained model for a task for your new task. For example, use the ImageNet model for a computer vision task.
• Decide whether your problem domain could benefit from training in an unsupervised (deep Q-learning, some kind of word embeddings) or semi-supervised way (images when don't have much labeled data). Otherwise, only use supervised techniques.