Recall a `k`-layer feedforward network `N` can be viewed as a composition of layer functions, `L_1,..., L_k`. On an input `vec{x}`, it computes the function `N(vec{x}) := L_k(L_{k-1}(...L_2(L_1(vec{x})) ...))`.
Motivated by maximum likelihood estimator (MLE) considerations, we said the cross entropy, `C`,
`-E_{vec{y}~hat{p}_{data}}[log p_{mo\de\l}(y|\vec{x};vec{theta})] = - 1/m \sum_{i=1}^m[y^{(i)} \log g(vec{theta}^Tvec{x}^{(i)}) + (1 - y^{(i)}) \log (1 - g(vec{theta}^Tvec{x}^{(i)}))]`
is a good choice of error function.
Using training considerations, we determined the ReLU or leaky ReLU hidden units often work better than those with logistic activations.
Also motivated by MLE considerations, we said that the last layer, `L_k` usually computes a softmax function of its weighted inputs, `soft\m\ax(vec{z})_i = exp(z_i)/(sum_j(exp(z_j)))`.
The cross-entropy serves as a good error function for the last layer as we can directly compare the output of a unit of the last layer with the value it is supposed to give.
We described two update weight techniques to search for weights which minimize cross-entropy:
Newton-like Methods which seek to find zeros of C' using a weight update rule based on multi-variable versions of computing `(C')/(C'')` (I.e., involving the gradient and the Hessian). For example, the Broyden-Fletcher-Goldfarb-Shanno (BFGS) algorithm.
Gradient descent methods which directly update the weight based on `\grad C`.
Cross entropy calculations needed for either method involve cycling over the complete training set.
We begin today by saying how to avoid this via random sampling. This use of "mini-batches" gives rise to the stochastic gradient descent algorithm.
We then need to say how to compute updates for hidden units to an algorithm to train all the neurons in a neural net.
Stochastic Gradient Descent
The cross entropy, `-E_{vec{x}~hat{p}_{data}}[log p_{mo\de\l}(y|\vec{x};vec{theta})]`, can be written as:
`1/m sum_{i=1}^m Loss(\vec{x}^{(i)}, y^{(i)}; vec{theta})`
where `Loss` is the per example loss `- log p_{mo\de\l}(y|\vec{x};vec{theta})`.
So to compute the gradient we need to compute:
`1/m sum_{i=1}^m grad_{theta}Loss(\vec{x}^{(i)}, y^{(i)}; vec{theta})`
This involves cycling over the complete dataset for each derivative component. So has computation cost `O(m)`.
As the gradient is an expectation, it may be approximated by using a small set of examples.
So at each step of gradient descent rather than compute the complete sum, we can sample a minibatch,
`mathbb{B} = {x^{(1)}, ..., x^{(m')}}` drawn uniformly from the training set. Here `m'` is fixed regardless of the size `m` of the dataset.
The estimated gradient is then
`vec{g} = 1/(m') sum_{i=1}^(m') grad_{theta}Loss(\vec{x}^{(i)}, y^{(i)}; vec{theta}).`
We then update according to
`vec{theta} leftarrow vec{theta} - epsilon vec{g}`.
This technique is called Stochastic Gradient Descent (SGD). Robbins Munro (1951) has an early proof of its convergence properties.
If we want to emphasize we are using the original approach of using the whole data set to compute cross entropy, then we say we are using batch gradient descent.
If the mini-batch size is 1, we will say we are using online gradient descent.
Training Terminology
Training a neural net usually involves two passes using the data in a batch/mini-batch:
A forward pass in which the values output by each operation computed by each neuron in a network for each data sample in the batch are computed.
A backward pass in which certain partial derivative of these operations are computed with respect to each of their inputs.
An iteration is a forward and backward pass through one batch/mini-batch of data.
One epoch is a visit to all the training data twice - once forward, once backward.
Visiting All the Data with SGD
You might be wondering: If we are randomly sampling the data set, using SGD, how do we ensure that we see all of the training data?
At the start of an epoch, we compute a random permutation of the training data:
for i in range(1, len(training_data)):
j = random.randint(i+1, len(training_data))
swap(i, j, training_data)
Now we can cycle through the data
for k in range(0, len(training_data), mini_batch_size)]:
current_batch = training_data[k:k+mini_batch_size]
# ... now use the mini-batch for training
In-Class Exercise
Suppose our data set had 6 items: ((1, 1.9), 1), ((2, 4.1), 0), ((3, 5.8), 1), ((4, 7.6), 1), ((5, 10.2), 0),
((6, 11.9), 1) and we were using a mini-batch size of 3.
Simulate computing a permutation of these input.
At what points in the backward pass would we update weights if we were using mini-batches?
Recall a layer `L` in our network computes a function `vec{g}(W vec{z} + vec{b})` where `vec{g}` computes a vector-valued activation, `W` is a weight matrix, `vec{z}` is the output of the previous layer, and `vec{b}` is a bias vector.
To keep things simple we will assume there is a dummy output from the previous layer that always has value 1, and we move the bias into the weight matrix, giving the form `vec{g}(W vec{z})`.
Assume our network has `n` layers.
We write `vec{g}_i` for the vector-valued activation function used in the `i`th layer.
We write `\mathbf{W}` for the tensor representing the weights for all layers in our network. Let `W_i` denote the weights in `\mathbf{W}` for the `i`th layer, let `W_{ij}` denote the weights of the `j`th neuron in this layer, and let `W_{ijk}` denote the weights of its `k`th component.
Recall we defined a tensor to mean a multi-linear map. `\mathbf{W}` acts linearly for each `W_i` so is a multi-linear map.
Let `vec{N}_{i}` denote the outputs of the `i`th layer of our network. We let `N(vec{x},\mathbf{W})` denote the matrix of outputs from all layers computed by our network.
If we have a set of training inputs `vec{mathbb{x}} = {vec{x}^{(1)}, ..., vec{x}^{(m)}}` and outputs `vec{mathbb{y}} = {y^{(1)}, ..., y^{(m)}}`, write `vec{mathbb{N}_i}(vec{mathbb{x}}, \mathbf{W})` for `{N_i(vec{x}^{(1)},\mathbf{W}), ..., N_i(vec{x}^{(m)},\mathbf{W})}` and `vec{mathbb{N}}(vec{mathbb{x}}, \mathbf{W})` for `{N(vec{x}^{(1)},\mathbf{W}), ..., N(vec{x}^{(m)},\mathbf{W})}`.
Let `C(mathbb(y), mathbb(z)) = - 1/m \sum_{i=1}^m[y^{(i)} \log z^{(i)} + (1 - y^{(i)}) \log (1 - z^{(i)})]` be the cross-entropy function.
Let `epsilon` be our learning rate. Let `\mathbb{x}_t`, `\mathbb{y}_t` denote the items in the `t` training mini-batch. Let `\mathbf{W}^{(t)}` denote the weights after `t` updates.
Using stochastic gradient descent we update the weights in the `i`th layer according to:
`W_i^{(t+1)} = W_i^{(t)} + \epsilon (\partial Loss(mathbb(y), vec{mathbb{x}}, \mathbf{W}))/(\partial W_i)|_{{:(vec{mathbb{x}} = vec{\mathbb{x}}_t),(vec{mathbb{y}} = vec{\mathbb{y}}_t), (\mathbf{W} = \mathbf{W}^{(t)}):}}`
Here `(\partial Loss(mathbb(y), vec{mathbb{x}}, \mathbf{W}))/(\partial W_i)` is a matrix of partials `(\partial Loss(mathbb(y), vec{mathbb{x}}, \mathbf{W}))/(\partial W_{ijk})`.
We will abuse notation and depending on the two things on the right hand side of the chain rule, we will treat '`cdot`' as either multiplication, dot product, matrix product, or tensor composition as appropriate.
In the above, it is not hard to see how we could slightly perturb the outputs of layer `i-1` to approximate `(\partial vec{mathbb{N}_i})/(\partial W_i)`.
It is harder to see how to calculate `(\partial C)/(\partial \mathbb{N}_i)`.
Backpropagation
If our network has `n` layers, then we can slightly tweak the output of the last layer to compute `(\partial C)/(\partial \mathbb{N}_n)`.
We can then use the chain rule to compute the partials `(\partial C)/(\partial \mathbb{N}_i)` via the recursion:
`(\partial C)/(\partial \mathbb{N}_{i-1})|_{{:(vec{mathbb{N}_{i-1}} = vec{\mathbb{N}_{i-1}}^{(t)}),(vec{mathbb{y}} = vec{\mathbb{y}}_t), (\mathbf{W} = \mathbf{W}^{(t)}):}}``= (\partial C)/(\partial \mathbb{N}_i)|_{{:(vec{mathbb{N}_i} = vec{\mathbb{N}_i}^{(t)}),(vec{mathbb{y}} = vec{\mathbb{y}}_t), (\mathbf{W} = \mathbf{W}^{(t)}):}}``cdot (\partial \mathbb{N}_i)/(\partial \mathbb{N}_{i-1})|_{{:(vec{mathbb{N}_{i-1}} = vec{\mathbb{N}_{i-1}}^{(t)}), (\mathbf{W} = \mathbf{W}^{(t)}):}}`
The whole procedure for computing the gradient `(\partial Loss(mathbb(y), vec{mathbb{x}}, \mathbf{W}))/(\partial W_i)` of the last two slides is called backpropagation (Rumelhart et al, 1986), but it gets its name because of the above computation.
Computing Partial Derivatives
The proceeding description of the partials needed to compute the update for `W_i^{(t+1)}` is slightly simplified.
For example, we could imagine computing `(\partial \mathbb{N}_i)/(\partial \mathbb{N}_{i-1})` as `(\partial \mathbb{N}_i)/(\partial vec{g}_i) cdot (\partial vec{g}_i)/(\partial \mathbb{N}_{i-1})`.
Individual components of `(\partial \mathbb{N}_i)/(\partial vec{g}_i)` might be useful in computing other partial derivative in the whole algorithm.
To avoid recomputing partial derivatives, most software systems break `N` down into its set of operations (one such operation might, say, compute a component of `vec{g}_i`), and define a computation graph of what needs to be computed before what for both the operations and their associated derivatives.
A computation graph is symbolic, you can think of nodes in this graph having references to functions to call to evaluate that node.
Given a numerical input to such a graph we can evaluate the nodes in the graph.
To get values for derivatives, there are two general approaches:
Use symbol-to-number differentiation (what Torch and Caffe do). Here when we need to compute `(\partial f)/(\partial x_i)` at a point `vec{x}`, we just use `(f(vec{x} + h\cdot hat{e}_i) - f(vec{x}))/h` where `hat{e}_i` is the `i`th unit basis vector and `h` is a small real value like `0.01`. In this case the derivative's the value is returned and memoized if need be.
Use symbol-to-symbol differentiation (what Theano and TensorFlow do). i.e., Here we have a reference to a function computing `(f(vec{x} + h\cdot hat{e}_i) - f(vec{x}))/h` as part of our computation graph. Each node has an eval function for when we need to compute an actual value. This allows higher order derivatives to also be computed, say if we want to use memory limited BFGS as our update mechanism.
TensorFlow and Keras
TensorFlow is a neural network package developed by the Google Brain team at Google.
It was initially released under an Apache Licence in November, 2015, but was based on an earlier system called DistBelief that had been used internally at Google since 2011.
Keras created by Francois Chollet in 2015 originally for Theano is a simplified abstraction layer frontend to Tensorflow for quickly writing deep learning code.
To install TensorFlow for use with python, we can do:
pip install tensorflow
This will get a version of TensorFlow suitable for the experiments in this class, although it probably won't make use of your computer's GPU.
To use the TensorFlow module, you then put something like:
Potential gotcha's in the install process often arise because you have multiple copies of Python installed.
For example, on my Mac, I had the default install, an install that came with MAMP, and an install from Homebrew.
The install can be persnickety about having a compatible version of Numpy.
One thing to be aware of is if you use the simpler instructions like the above, when you do some operations in TensorFlow, you might get a warning like:
2021-10-20 12:12:19.334445: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
To suppress these (and other) warnings, you can add:
import os
os.environ['TF_CPP_MIN_LOG_LEVEL']='2'
to the start of your code.
What TensorFlow Lets Us Do
Tensor Flow allows us to do the following main operations:
