Last week, we said a m-neuron perceptron layer `L` in a multi-layer perceptron network can be computed as `vec{g}_m(W vec{x} + vec{b})` where `W` and `vec{b}` each have `m` rows and `vec{g}_m(vec{a})` was the function `(g(a_1), ..., g(a_m))` for some activation function `g`.
If we had a network `N` of `k` layers, `L_1,... L_k`, then on an input `vec{x}`, it computes the function `N(vec{x}) := L_k(L_{k-1}(...L_2(L_1(vec{x})) ...))`.
We would now like to describe how to train `N(x)` in the presence of labeled data.
Recall that when we proved the perceptron convergence theorem, we knew that we were learning some target threshold function with weights `vec{w}^t` and we could define a function of the current weights `vec{w}` to say how far we were from the target.
This function was `||vec{w}^t - vec{w}||^2` and we showed the update rule reduced this.
For our PAC-learning result, we used the potential function `N_t(alpha) = ||alpha w - w_t||^2 + (alpha theta - theta_t)^2`, and for the S-K algorithm we used `||vec{w}^{+} - vec{w}^{-}|| - m(vec{x}'_t)`, but the idea was the same.
These functions are often called loss or cost, or potential functions.
In each case, we are trying to adjust the weights to minimize the potential function.
One class of algorithms to figure out how to adjust the weights are known as gradient descent algorithms.
Before we look at these, let's looks at a couple different cost functions.
Costs Functions
Assume we train using a sequence of examples `(vec{x}^{(1)},y^{(1)}), (vec{x}^{(2)},y^{(2)}), ..., (vec{x}^{(m)},y^{(m)})` and let `N(vec{x})` be the output of our network.
One common choice of cost function is mean squared error (MSE) `1/m sum_i||y^{(i)} - N(vec{x}^{(i)})||^2`.
This measures the average error the networks makes over the training set.
It turns out this choice of cost function can be justified if the underlying function that the data is coming from is a linear function of the
`vec{x}^{(i)}` and the `vec{x}^{(i)}`'s are chosen according to a Gaussian distribution.
To get a more robust cost function for a larger variety of underlying target functions, we make use of the theory of estimators from last day.
Maximum Likelihood Estimation - unlabeled data
Let `mathbb{X}` be a set of examples `vec{x}^{(1)}, ..., vec{x}^{(m)}` drawn independently from sample space `S` according to some unknown data generating distribution `p_{data}(vec{x})`.
Let `p_{mo\d\el}(\vec{x};\vec{theta})` be a parametric family of probability distributions over the same space `S` indexed be `vec{theta}`.
Our goal is to field a `vec{theta}` so that `p_{mo\de\l}(\vec{x};\vec{theta})` is as close to `p_{data}(vec{x})` as possible.
The maximum likelihood estimator (MLE) for `vec{theta}` is defined as:
\begin{eqnarray*}
\vec{\theta}_{ML} &=& argmax_{\vec{\theta}} p_{model}(\mathbb{X}; \vec{\theta})\\
&=& argmax_{\vec{\theta}}\prod_{i=1}^m p_{model}(\vec{x}^{(i)}; \vec{\theta})
\end{eqnarray*}
I.e., we want to find the value of `vec{theta}` that maximizes the product of probabilities of the sequence of items that were observed.
MLE for unlabeled data as an Expectation
The product on the previous slide can be quite small and so is sensitive to underflow round off errors.
Since when the log of this product is maximized when the product is maximized, we can instead use the less round off sensitive definition:
`\vec{\theta}_{ML} = ar\gm\ax_{\vec{\theta}} sum_{i=1}^m log p_{mo\de\l}(\vec{x}^{(i)}; \vec{\theta}).`
Since multiplying the above sum by `1/m` doesn't affect the argmax either, we can do that and then view the result as an expectation:
`\vec{\theta}_{ML} = ar\gm\ax_{\vec{\theta}} E_{vec{x} ~ hat{p}_{data}} (log p_{mo\de\l}(\vec{x}; \vec{\theta})).`
From MLE for unlabeled data to a Cost Function
We can interpret the above as trying to minimize the dissimilarity between the empirical distribution `hat{p}_{data}`, defined by the training set and the model. Ideally, we would like to match the actual distribution `p_{data}`, but we don't have direct access to this.
One measure of this dissimilarity between two distributions is Kullback-Leibler (KL) divergence:
`D_{KL}(P||Q) = E_{x~P}[log( (P(x))/ (Q(x)))] = E_{x~P}[log P(x) - log Q(x)]`
As log of a number is roughly the number of bits in a number, the above roughly measures the expected number of bits of information needed to compute `P(x)` given `Q(x)`.
In the case of MLE, the dissimilarity between `hat{p}_{data}` and `p_{mo\de\l}` is
`D_{KL}(hat{p}_{data}||p_{mo\de\l}) = E_{vec{x}~hat{p}_{data}}[log hat{p}_{data}(\vec{x]) - log p_{mo\de\l}(\vec{x}; \vec{theta})].`
So `\vec{\theta}_{ML}` is the value that minimizes `vec{theta}` the above.
To minimize `D_{KL}(hat{p}_{data}||p_{mo\de\l})` it suffices to minimize:
`-E_{vec{x}~hat{p}_{data}}[log p_{mo\de\l}(\vec{x}; vec{theta})]`
So we can take this as our loss function.
From MLE labeled data to a Cost Function
In the case of labeled data, we need to use conditional probabilities:
`\vec{\theta}_{ML} = ar\gm\ax_\vec{\theta} P(mathbb{Y}|\mathbb{X}; vec{theta})`
In this case, the cost function we want to minimize is:
`-E_{vec{y}~hat{p}_{data}}[log p_{mo\de\l}(y|\vec{x};vec{theta})]`
This cost function is known as the cross entropy.
Cross-Entropy Cost Function - Linear Gaussian Case
Notice for the Gaussian distribution we have:
`p_{mo\de\l}(y|\vec{x};vec{theta}) = Pr(y, hat{y}(\vec{x};vec{theta}), sigma^2) = sqrt(1/(2\pi\sigma^2))exp(-1/(2\sigma^2)(y - hat{y}(\vec{x};vec{theta}))^2)`.
Let `m` again be the number of training examples. Assuming the data items independently, identically chosen from the underlying distribution, we see that when we substitute this into the cross entropy, we get
`- 1/m sum_{i=1}^m log p_{mo\de\l}(y^{(i)}|\vec{x}^{(i)};vec{theta}) = log sigma + 1/2 log (2\pi) + 1/m sum_{i=1}^m (||y^{(i)} - hat{y}^{(i)} ||^2)/(2\sigma^2).`
The first two terms on the right are constant, and minimizing with or without the `2\sigma^2` term has no effect.
So minimizing the above is equivalent to minimizing the mean square error (MSE): `1/m sum_i||y^{(i)} - hat{y}^{(i)}||^2`.
Thus, if we had a neuron whose activation function was the identity function, using MSE might make sense.
Cross-Entropy Cost Function - Logistic Function Case
Suppose the values of `y` are in `{0,1}` as we have a bianry classifier.
Let `vec{theta}` represent the weights coming into a preceptron.
By using the trick of having a dummy input `x_0` which is always 1, we can imagine the perceptron computes some function `g(vec{theta}^Tvec{x})`.
As before, the cross entropy can be calculated as:
`-E_{\vec{y}~\hat{p}_{data}}[\log p_{mo\d\el}(y|\vec{x};\vec{\theta})] = - 1/m \sum_{i=1}^m \log p_{mo\d\el}(y^{(i)}|\vec{x}^{(i)};\vec{\theta}).`
Assume `g` is the logistic function: `g(z) = 1/(1-e^{-z}) = 1/2 + 1/2tanh(z/2)`.
`g(vec{theta}^Tvec{x})` represents the probability of finding `y=1` and `1-g(vec{theta}^Tvec{x})` represents the probability of finding `y=0`.
For the actual data `y` represents probability of the label being 1, and `1-y` represents the probability of the label being 0.
Since we are taking an average anyway, rather than try to compute `\log p_{mo\d\el}(y^{(i)}|\vec{x}^{(i)};\vec{\theta})` directly we can use in the above the easier to calculate:
`E(\log p_{mo\d\el}(y^{(i)}|\vec{x}^{(i)};\vec{\theta})) = y^{(i)} \log g(vec{theta}^Tvec{x}^{(i)}) + (1 - y^{(i)}) \log (1 - g(vec{theta}^Tvec{x}^{(i)}))`.
This gives the cross entropy loss function as:
`- 1/m \sum_{i=1}^m[y^{(i)} \log g(vec{theta}^Tvec{x}^{(i)}) + (1 - y^{(i)}) \log (1 - g(vec{theta}^Tvec{x}^{(i)}))].`
This approximation is also used for other activation functions as well.
Quiz
Which of the following is true?
The linear_alg submodule that comes with a default numpy install can be used to compute matrix inverses.
Leave-p-out is an inexhaustive cross-validation technique.
`\hat{mu}_m = 1/m(sum_{i=1}^m x^{(i)})` is an unbiased estimator of `mu` for the Gaussian distribution.
Output Units
The choice of cost function is often related to the choice of the units in our output layer.
For example, suppose we were trying to learn a vector of average statistics `vec{s}` a baseball team might get based on data about features `vec{h}` in a hidden layer computed from players that make up the team.
We might choose the output unit to be a linear unit layer `hat{y} = W^{T}vec{h} + vec{b}`.
Suppose, however, we were trying to predict the value of a binary variable `y`, a binary classification problem.
Using MLE, we define a Bernoulli distribution over `y` conditioned on `vec{x}`.
The net needs to predict `P(y=1|vec{x})`. For this number to be a valid probability it must lie in the interval `[0,1]`.
One way to force this to be true if we used a linear unit might be to define:
`P(y=1|vec{x}) = max(0, min(1,vec{w}^T vec{h} + b))`
that is we clip the value of the linear unit to the desired interval.
This has a drawback when using gradient approaches in that the derivative is 0 outside the unit interval.
Instead, a single perceptron unit as the output could be used with a sigmoid activation function which has a well defined derivative.
Motivating the Sigmoid Function, Softmax
Split the top preceptron of our network into two parts `z = vec{w}^T vec{h} + b` and `y=g(z)` for our activation function. The latter step converts `z` to a probability.
Suppose we want to define a probability distribution over `y` using the value for `z`
We can imagine we initially have an unnormalized probability distribution `tilde{P}(y)` that we divide by an appropriate constant to make a probability distribution.
If we assume that the unnormalized log probabilities are linear in `y` and `z`, we have:
\begin{eqnarray*}
\log \tilde{P}(y) &=& yz\\
\tilde{P}(y) &=& exp(yz),\\
P(y) &=& exp(yz)/ \sum_{y'=0}^{1}exp(y'z)\\
P(y) &=& \sigma((2y-1)z)
\end{eqnarray*}
where sigma is the sigmoid function.
This motivates using the sigmoid function in the last layer if one has two possibilities.
If we have a k-label classifier, we can generalize this argument to say we should a softmax layer:
`soft\m\ax(vec{z})_i = exp(z_i)/(sum_j(exp(z_j)))`
Training a Softmax Layer
Suppose we want to train a softmax layer with `k` labels.
If training input `vec{x^{(j}}` has label `i`, then the vector `vec{y}^{(j)}` we train with should
be a length `k` vector with all components `0` except the `i`th component which should have value 1.
This kind of training vector is called a 1-hot vector.
