Introduction

We have been talking about multi-layer perceptron networks (MLPs).
To train these networks we need a cost (also known as a loss) function which estimates how far the model is from the target function or probability distribution we are trying to learn.
Motivated by the maximum likelihood estimator, we determined the cost function `-E_{vec{x}~hat{p}_{data}}[log p_{mo\de\l}(y|\vec{x};vec{theta})]` would be a good choice and that we should try to minimize this function, known as the cross-entropy.
We said that in the case where `vec{x}` are chosen based on a Gaussian distribution, and where the target function being learned is linear, this reduces to using the mean squared error function, `1/m sum_i||y_i - N(vec{x}_i)||^2`.
We then looked at possible output layers that our model should have depending on whether we are trying to learn a set of statistics (then said use linear unit layer), a binary classifier (then use a single perceptron with sigmoid activation), or a multi-label classifier (then use softmax layer). The latter two we also motivated using Maximum Likelihood Estimation.
Today, we are going to start by looking at gradient descent and then look at the affect of gradient descent on the choice of activation function for our hidden layers, and finally discuss stochastic gradient descent.

Minimizing Functions

We want to update our neural nets weights so as to minimize our cost function.
From calculus, we know that we can find local extrema (local minima or local maxima) of functions by computing their derivatives, setting them to 0, and solving for the minimized variables.
Recall a point `vec{x}` is a local minima for `f` if for some `epsilon >0` and all points `vec{z}` within distance `epsilon` of `vec{x}, f(vec{x})\leq f(vec{z})`.
In learning we often settle for local minima.
It is the second derivative determines whether an extrema is a minima or maxima.
In the case, of a single variable twice differentiable function, if `f'(x)=0` and `f''(x) geq 0` implies the point `x` is a local minima.
For multivariable function case, if the second derivatives, the Hessian, is positive semi-definite (`vec{x}^TM\vec{x} \geq 0` for all `vec{x} in RR^n` ), then we will be at a local minima.
We will call a function which satisfies this test convex.
If it is not practical to compute the exact form of the derivative then to find extrema, we need to search for them.

Newton Methods For Finding Minima

Recall one way, Newton-Raphson aka Newton's Method (1669), to find the zeros of a function `f(x) = y` is to start with an initial guess of `x_0`, compute the tangent line to `(x_0, f(x_0))`, see the value `x_1` where that intersects the `x`-axis, and use that as a guess for the zero, and repeat if it is not yet close enough.
This gives the update rule:
`x_{n+1} = x_n - f(x_n)/(f'(x_n))`.
Under reasonable conditions this will eventually converge on a zero.
If our cost function `C` were 1 dimensional, one could imagine to minimize `C` we find the zeros of the derivative, `C'` using this method. This would involve `C''` and would assume the function was twice differentiable.
Quasi-Newton methods serve as the basis of one class of minimization techniques in the multi-variable setting.
In the multi-variable setting the above observations mean we need to compute Hessians which often need to be tweaked to be made positive-definite.
One algorithm, Levenberg-Marquardt (1944-1963), often used for surface fitting, requires adding an extra `alpha I` term to the Hessian so that the inverse we need for the Newton update is more likely to behave.
Another class of techniques known as conjugate gradient descent algorithms (Fletcher Reeves 1964, Polak Ribiere 1969) try to avoid computing the inverse Hessian by using the Hessian to compute direction lines to multiply against the gradient.
Nowadays, a simpler, closer to Newton approach known as the Broyden-Fletcher-Goldfarb-Shanno (BFGS) algorithm (four papers, one for each author, all from 1970) has become more popular in the Neural Net setting.

The BFGS Algorithm

The BFGS Algorithm maintains three quantities: `vec{x}_i`, our current best estimate of a solution; `vec{p}_i`, a search direction; and `B_i`, an approximation to the Hessian. We start from an initial guess `vec{x}_0` and an initial approximation of the Hessian `B_0`. To compute the `i+1` values from the `i` values we then do:
1. Solve `B_i vec{p} = - grad f(vec{x}_i)` for `vec{p}_i` using an algorithm to solve linear equations (in class we did Gaussian elimination earlier in the semester).
2. Compute a learning rate (aka step size) for this iteration by searching for `alpha_i = argmin_{alpha} f(vec{x}_i + alpha vec{p}_i)`.
3. Set `vec{x}_{i+1} = vec{x}_{i} + vec{s}_i` where `vec{s}_i = alpha_i vec{p}_i`.
4. Compute `B_{i+1}` by first defining `vec{y}_i = grad f(vec{x}_{i+1}) - grad f(vec{x}_{i})`, and then setting
  `B_{i+1} = B_{i} + (vec{y}_i vec{y}_i^T)/(vec{y}_i^T vec{s}_i) + (B_i vec{s}_i vec{s}_i^T B_i^T)/(vec{s}_i^T B_i vec{s}_i)`
You can think of (1) above as computing an analog of `f(x_n)/(f'(x_n))`.
The linear solving portion of the algorithm above makes it at least quadratic and probably cubic in practice to implement step 1.
There is a variant of this algorithm, limited-BFGS (Nocedal 1980), which is used in practice and avoids the computation of this full `n times n` matrix. For limited-BFGS, the update only takes `O(n)`.

In-Class Exercise

Consider the function `f(x)=(x-2)^2(x-3)^2`. Suppose we state with `x_0 = 5`. Compute the values of `x_1, x_2, x_3, x_4` as determined by Newton-Raphson.
Post your answers to the Oct 18 In-Class Exercise Thread.

Gradient Descent

Instead, if we just want to minimize `C(x)`, we can do the simpler procedure of starting with some `x_0`, compute `C'(x_i)`, and use an update `x_{i+1} = x_i - epsilon C'(x_i)`. I.e., we move in the direction that makes the function less by a rate proportional to the current derivative.
The generalization of this method to many dimensions is called the method of gradient descent, or in the case of minimization, the method of steepest descent (Debye 1909). When one has a function `f: RR^m -> RR`, this method's update rule looks like:
`vec{x}_{n+1} = vec{x}_n - epsilon grad f(vec{x}_n)`.
Here the gradient, `grad f(vec{x}_n)` is a vector of partial derivatives with respect to each basis direction and `-grad f(vec{x}_n)` points in the direction of greatest decrease.
`epsilon >0` here is called the learning rate.
The line search method of choosing the learning rate is to evaluate `f(vec{x} - epsilon grad f(vec{x}))` for several values of `epsilon` and choose the one that results in the smallest objective function value.

Hidden Units

Hidden layers of a neural network are layers not directly connected to the output.
For now, we are assuming the units in our layers are perceptrons and are concerned mainly with what activation function to choose.
Since we are going to train our models by using gradient descent to find a local mimima, it often makes sense to use this as one of the selecting criteria.
A perception computes `g(vec{w}^T\vec{x} + b)`. Let `f(vec{x}; vec{w}, b) = vec{w}^T\vec{x} + b` so its partial derivative with respect to a `w_i` or `b` is given by the chain rule:
`(del g)/ (del f) \cdot (del f)/(del w_i)` and `(del g)/ (del f) \cdot (del f)/(del b)` .

Types of Activation

identity function g(z) = z - the resulting perceptron is called a linear unit. `(del g)/ (del f)` is 1 and so the derivative of the `(del g)/(del w_i)` and `(del g)/(del b)` will be consistent with `f` and convergence will tend to be fast.
`g(z)= max{0,z}` - the resulting perceptron is called a rectified linear unit (ReLU).
- To compute the derivative software packages usual compute either a left-hand or right hand limit. So even though the derivative is not defined at 0, the handed derivatives are and this usually suffice in practice.
- Notice `(del g)/ (del f)` is 1 whenever the unit is active, and so the overall derivatives `(del g)/(del w_i)` and `(del g)/(del b)` will tend to remain large whenever the unit is active resulting in fast convergence.
- Also, the second derivative is 0 almost everywhere, so the function will tend to behave in a way that will achieve a local minimum, more so than a complex activation function.
`g(z; alpha) = max(0, z) + alpha min(0,z)` - when `alpha = -1`, this is called absolute value rectification. A unit with a small value for `alpha` is called a leaky ReLU. If `alpha` is a learned parameter, then a unit with this kind of activation is called a parametric ReLU.
- Units with `g(z; alpha)` have many of the same benefits as ReLU gates as far as their derivatives are concerned but also have a larger domain over which they are non-zero.
- Since this domain determines when gradient descent will work they tend to be slightly more robust.
`g(z)=tanh(z)` or the sigmoid activation - these have the virtue of having everywhere defined derivatives, but as there is only a narrow range where these derivatives differ much from 0, units using these activations tend to train more slowly.

Remarks on Initialization

One thing we haven't mentioned yet is what should be the initial choice of weights we should use when we start to do gradient descent.
If we are using ReLUs then it makes sense that we want the biases `vec{b}` not to be `0` and the weights not to be `0` so that the gradient is non zero and we can get going.
For this reason we often initialize the weight to some random float in a range `[-b, b]` where `b` is usually determined based on the number of neurons in the network (we'll talk more about this later).

Stochastic Gradient Descent

The cross entropy, `-E_{vec{x}~hat{p}_{data}}[log p_{mo\de\l}(y|\vec{x};vec{theta})]`, can be written as:
`1/m sum_{i=1}^m Loss(\vec{x}^{(i)}, y^{(i)}; vec{theta})`
where `Loss` is the per example loss `- log p_{mo\de\l}(y|\vec{x};vec{theta})`.
So to compute the gradient we need to compute:
`1/m sum_{i=1}^m grad_{theta}Loss(\vec{x}^{(i)}, y^{(i)}; vec{theta})`
This involves cycling over the complete dataset for each derivative component. So has computation cost `O(m)`.
As the gradient is an expectation, it may be approximated by using a small set of examples.
So at each step of gradient descent rather than compute the complete sum, we can sample a minibatch, `mathbb{B} = {x^{(1)}, ..., x^{(m')}}` drawn uniformly from the training set. Here `m'` is fixed regardless of the size `m` of the dataset.
The estimated gradient is then
`vec{g} = 1/(m') sum_{i=1}^(m') grad_{theta}Loss(\vec{x}^{(i)}, y^{(i)}; vec{theta}).`
We then update according to
`vec{theta} leftarrow vec{theta} - epsilon vec{g}`.
This technique is called Stochastic Gradient Descent (SGD). Robbins Munro (1951) has an early proof of its convergence properties.
If we want to emphasize we are using the original approach of using the whole data set to compute cross entropy, then we say we are using batch gradient descent.
If the mini-batch size is 1, we will say we are using online gradient descent.

Training Terminology

Training a neural net usually involves two passes using the data in a batch/mini-batch:
1. A forward pass in which the values output by each operation computed by each neuron in a network for each data sample in the batch are computed.
2. A backward pass in which certain partial derivative of these operations are computed with respect to each of their inputs.
An iteration is a forward and backward pass through one batch/mini-batch of data.
One epoch is a visit to all the training data twice - once forward, once backward.

Visiting All the Data with SGD

You might be wondering: If we are randomly sampling the data set, using SGD, how do we ensure that we see of the training data?

At the start of an epoch, we compute a random permutation of the training data:

for i in range(1, len(training_data)):
    j = random.randint(i+1, len(training_data))
    swap(i, j, training_data)

Now we can cycle through the data

for k in range(0, len(training_data), mini_batch_size)]:
    current_batch = training_data[k:k+mini_batch_size]
    # ... now use the mini-batch for training

Minimization Methods, Hidden Units, Stochastic Gradient Descent

Outline

Introduction

Minimizing Functions

Newton Methods For Finding Minima

The BFGS Algorithm

In-Class Exercise

Gradient Descent

Hidden Units

Types of Activation

Remarks on Initialization

Stochastic Gradient Descent

Training Terminology

Visiting All the Data with SGD