Outline

• Neural Net Experiments
• In-class Exercise
• Feedforward Networks

Neural Net Experiments

• People use neural nets for tasks where it would be hard for a human to come up with an exact algorithm for that task. For example, handwriting recognition.
• Not everyone uses the same hardware or even neural network software.
• On the other hand, people do want to be able to share their neural net algorithms.
• To share in this context means to describe the architecture of ones network and the training algorithm used in sufficient detail that someone on different hardware and using different neural net software has some hope of still producing a similarly trained network.
• Since for neural networks it might be hard to figure out exactly what a trained model is doing, it is hard to give a formal proof of correctness for the trained model.
• Neural network experiments which show how well a model generalizes, how accurate it is, etc. are used therefore in lieu of such a proof.
• They serve as an argument as to why or why not use a particular architecture and training algortihm to solve some problem.

What should a Neural Network Experiment contain?

1. A statement of a hypothesis which could be proven true or false.
   • The hypothesis should be settling something interesting to know about your neural net. For example, how well it generalizes.
   • Your hypothesis should be robust under tweakings of hyperparameters. I.e., if you can change the value of the answer of your hypothesis too easily by tweaking hyperparameters, then it suggests your results don't generalize.
2. A description of your neural net and training set ups for the experiment conducted. This should be in sufficient details so that someone else could try to replicate your experiment.
3. The results of your experiments. I.e., what was the output of your experiments.
4. A conclusion from your results with regard to your initial hypothesis.

Things to Measure with Experiments 1 - Accuracy , Confusion Matrix

What kind of hypothesis we have is dependent on the things we can actually measure with an NN experiment. On the next few slides I give some examples of the kinds of things we can measure. After this, I give some example hypothesis which could be tested using these kind of experiments.

• Accuracy = Number of times model correct/Number of test examples.
• We can refine this a little by asking for the following four different rates which can be arranged into 2x2 table known as a confusion matrix (here I am using rates for the entries, people often use raw counts (the numerators of the below)):

  TPR	FPR
  FNR	TNR

  1. True Positive Rate (TPR, aka sensitivity)= number of times model output true when test example was true/number of times model output true.
  2. False Positive Rate(FPR) = number of times model output true when test example was false/number of times model output true.
  3. False Negative Rate (FNR) = number of times model output false when test example was true/number of times model output false.
  4. True Negative Rate (TNR, aka specificity) = number of times model output false when test example was false/number of times model output false.
• If we use raw counts rather than rates, we can generalize the confusion matrix to the multi-label setting. Below is an example of doing this for a situation with three labels A,B,C:

  		Observed Label
  		A	B	C
  Predicted Label	A	5	1	2
  	B	3	7	4
  	C	0	2	9

Things to Measure with Experiments 2 - ROC Curve

• Receiver Operator Curve (ROC) curve were developed during WW II for analysis of radar signals.
• An ROC curve plots the TPR versus the FPR for a binary classifier.
• To create it:
  1. Compute a scatter plot of classifier scores versus training items.
  2. For each possible score value `v` from largest to smallest:
     1. Choose it as a classifier threshold and figure out what the `TPR` and `FPR` would be for that threshold, `TPR_v`, `FPR_v`.
     2. Plot `(FPR_v, TPR_v)` as a point on the ROC curve.
  3. The curve is then made by joining the points plotted.
• In the figure above, for the threshold v, we get the point `(1/2, 1)`.
• The curve will always be above the line y= x.
• A larger area under this curve indicates a better classifier.

Things to Measure with Experiments 3 - Performance

• Runtime of training as a function of data set size.
• Memory usage during training as a function of data set size.
• Number of specific function calls as a function of data set size.
• If your training algorithm involved random numbers, one should repeat the experiments several times and calculate a standard deviation for the rates above.

Example Hypotheses

• Neural Net Architecture A is more accurate than Neural Net Architecture B.
• Training Algorithm A converges to a more accurate model than Training Algorithm B.
• Neural Net Architecture A requires less memory, less neurons, etc to train than Neural Net Architecture B
• Training Algorithm A converges faster than Training Algorithm B.
• Accuracy of model using Architecture A and Training Algorithm B is such and such function of training time/some other parameter.
• The best value for hyperparameter `lambda` is `c`.
• The model will generalize to never before seen data if hyperparameter `lambda` is set to value `c`.

Representing Our Results - Graphing and Charting

• Often plotting the results of our experiments on a graph or chart can make them easier to understand.
• When we use a graph, it is important to give the reader enough detail so that someone else can understand what it represents.
• At a minimum, this means:
  1. Give the graph/chart a title or caption
  2. Label for a graph each of the axes, each function plotted, and specifying units.
  3. Label for a chart each of the items being chartted.
• In python, we can use matplotlib (created by John Hunter, 2003) to do a variety of plotting and charting tasks.
• It can be installed via the command:

  pip install matplotlib
  

• The matplotlib Gallery has a wide variety of plotting examples.

matplotlib - Basic Graphing

• If we have two numpy arrays of points we can graph them together as follows:
  

  import matplotlib.pyplot as plt
  import numpy as np
  a = np.arange(0, 10,.5, dtype=float);
  b = a * a
  plt.title("The function y=x^2")
  plt.xlabel("x-axis")
  plt.ylabel("y-axis")
  plt.plot(a,b)
  plt.show()
  

  
• After the show() call, the plot is reset. A second call to plt.show() would output nothing: We need to make new calls to set up the another plot.
• We can specify as part of the plot() a variety of attributes of the points to be plot such as color, linetype, and linewidth.
• We can call plot() multiple times to plot multiple functions on the same graph.
• Below we do this and illustrate different linetypes and how to make a legend:
  

  import matplotlib.pyplot as plt
  import numpy as np
  a = np.arange(0, 10,.5, dtype=float);
  plt.title("Growth rates y=x, y=x^2, and y=x^3")
  plt.xlabel("x-axis")
  plt.ylabel("y-axis")
  id_line, = plt.plot(a,a, color="blue",
      label="y=x", linestyle='dashed', linewidth=2) 
  quad_line, = plt.plot(a, a**2,
      color="red", label="y=x^2", linestyle='dotted') 
  cube_line, = plt.plot(a, a**3,
      color="green", label="y=x^2",
      linestyle='dashdot')
  plt.legend(handles=[id_line, quad_line, cube_line],
      loc=2) 
      #loc can be a number 1-4, number represents
      #which corner
  plt.show()
  

  

matplotlib - Scatter Plots, Bar, Histograms, Pie Charts

• matplotlib can be used to make scatter plots:
  

  import matplotlib.pyplot as plt
  import numpy as np
  a = np.arange(0, 10,.5, dtype=float);
  plt.title("Various polynomials")
  plt.xlabel("x-axis")
  plt.ylabel("y-axis")
  plt.scatter(a, a**2)
  plt.scatter(a, a**3, 
     marker="+", color="green");
  plt.scatter(a, a**4, 200, marker="o");
     # 200 is size in pixels
  plt.show()
  

  
• matplotlib can be used to plot histograms:
  

  import matplotlib.pyplot as plt
  import numpy as np
  a = np.array([1,1,2,2,2,2,3,4,4,4,4,
      5,5,5,5,6,6,7,8,8,9,9,9,10,10,10])
  plt.hist(a, bins=5)
  plt.show();
  

  
• matplotlib can be used to plot bar charts:
  

  import matplotlib.pyplot as plt
  import numpy as np
  plt.title("Product Comparison")
  plt.bar(["Product A", "Product B"],
      [50, 100])
  plt.show();
  

  
• matplotlib can be used to plot pie charts:
  

  import matplotlib.pyplot as plt
  import numpy as np
  plt.title("Marketshare Comparison")
  companies = ["Company A", "Company B",
     "Company C"]
  shares = [20, 50, 30]
  colors = ['red', 'green', 'blue']
  plt.pie(shares, labels=companies, 
      colors=colors, startangle=100)
  plt.show();
  

  

matplotlib - Saving Images

• Rather than using show() to display images to a window, we can use draw() and savefig() to write an a graph or chart to a file:

  import matplotlib.pyplot as plt
  plt.title("Product Comparison")
  plt.bar(["Product A", "Product B"],
      [50, 100])
  plt.draw();
  plt.savefig("product_comparison.png");
  

In-Class Exercise

• For the experiments of HW2, come up with a hypotheses for its outcomes, give a brief description of how you intend to test this hypothesis.
• Post your answers to the Oct 11 In-Class Exercise Thread.

Maximizing Your Test Data - Cross Validation

• Often one starts one training with a limited sized data. For example, one has only 10,000 crazy cat versus normal cat images.
• If you divide the data set into test and training data, you might end up with a small test set, which in turn results in the error in the accuracy of your test measurements being larger.
• If the errors are large in your tests, it might make it hard to claim algorithm A work better than algorithm B.
• One way to solve this problem is to use Cross-Validation: You split the data set into training and test data several times and in several different ways, perform your measurements for each of these trials, and combine these measurements by computing an appropriate aggregation such as an average. (Larson 1931).
• Cross-validation can be either exhaustive or non-exhaustive.
• An exhaustive cross-validation is one in which all possible ways to divide the data into training and test data are considered. For example, cycle over all possible p-subsets of the data sets, train on the data excluding the p-subset, test on the p-subset. This is called Leave-p-out cross validation.
• If the cross-validation doesn't consider all possible ways to divide the data, it is called inexhaustive.
• Two common inexhaustive techniques are:
  1. Repeated Random Sub-sampling Validation: Randomly repeat `m`-times, choose a subset of size `p` as the test data set, train on the remaining data, test on this subset.
  2. k-fold Cross Validation: Split the data into `k` equal sized subsets. `S_1, ..., S_k`. For `i=1,..., k`, train on all but subset `S_i`, then test on `S_i`, aggregate the results. (Often choose `k=5`)

Estimators

• When conducting experiments we are often interested in trying to come up with a "best" prediction of some quantity of interest, say accuracy.
• A point estimator (aka statistic) is any function of the data `hat{theta}_m = g(vec{x}^{(1)}, ..., vec{x}^{(m)})`.
• A "good estimator" is an estimator whose output is close to the underlying `theta` that generated the training data.
• For example, assume some distribution `Pr` on real world data (the sample space, `S`). Let `X subseteq S` be some set we are trying to build a classifier for. Let `X(x)` be 1 if `x in X` and 0 otherwise, let `M(x)` denote the value our trained model gives on `x`.
• The function `Co\r\r\e\c\t: x -> |X(x) - M(x) |` is a random variable.
• If we took `m` samples from our data and computed the accuracy of `M(x)` on those samples, `A\c\c_m`, then `A\c\c_m` would be an estimator of `E(Co\r\r\e\c\t)`.
• Since `A\c\c_m` is a function of the data which is assumed to be randomly drawn according to some distribution, it too is a random variable.
• When we do an operation like `k`-fold validation, we can ask if the result is actually a better estimator of the underlying quantity or random variable than say not doing it?
• We call an estimator a function estimator, if it is a point estimator where we are trying to approximate an underlying random variable based on the data rather than a quantity.

Estimator Bias

• The bias of an estimator is defined as: `bias(hat{theta}_m) = E(hat{theta}_m) - theta`.
• As an example suppose we have a coin which is heads (value 1) with probability `theta` and tails (value 0) with probability `1-theta`. Flipping coins in this way is an example of a Bernoulli Trial and gives rise to the Bernoulli Distribution. We want to perform trials to estimate `theta`. Let `{x^{(1)},..., x^{(m)}}` be the result of `m` trials.
• A common estimator for `theta` is `hat{theta}_m = 1/m sum_{i=1}^m x^{(i)}`.
• We can calculate its bias as follows: \begin{eqnarray*} bias(\hat{\theta}_m) &=& E(\hat{\theta}_m) - \theta\\ &=& E(1/m \sum_{i=1}^m x^{(i)}) - \theta\\ &=& 1/m \cdot E(\sum_{i=1}^m x^{(i)}) - \theta\\ &=& 1/m \sum_{i=1}^m \sum_{x^{(i)}=0}^1 x^{(i)}(\theta^{x^{(i)}}(1 - \theta)^{1 - x^{(i)}}) - \theta\\ &=& 1/m \sum_{i=1}^m (\theta) - \theta\\ &=& \theta - \theta = 0 \end{eqnarray*}
• So in this case, our estimator is unbiased, that is, it's bias is 0.

The Gaussian Distribution and an Example Biased Estimator

• The normal distribution, aka Gaussian distribution, is the distribution with sample space `RR` and probabilities given by:
  `Pr(x; mu, sigma^2) = sqrt(1/(2\pi\sigma^2))exp(-1/(2\sigma^2)(x - mu)^2).`
• Here `mu` and `sigma` are fixed constants (hyperparameters) and so we list them after a ";" in the left hand side of `Pr` equation above.
• Let `X_1,..., X_n` be `n` identical independent random variables on the reals. Let `Pr` be a probability distribution on the reals. Since they are all identical, with respect to the `Pr` distribution, they all have the same `mu = E(X_i)` and `sigma = Var(X_i)`. The central limit theorem (DeMoivre 1733) of probability theory says `sqrt(n)((sum_{i=1}^n X_i/n) - mu)` converges to the Gaussian distribution as `n` gets large.
• One estimator for the sample mean `mu ` of the Gaussian distribution is `\hat{mu}_m = 1/m(sum_{i=1}^m x^{(i)})`. One can show by the same kind of argument as the last slide that this is an unbiased estimator.
• One estimator for the sample variance `sigma` of the Gaussian distribution is `\hat{sigma}_m^2 = 1/m sum_{i=1}^m (x^{(i)} - hat{mu}_m)^2`.
• Notice `E(\hat{sigma}_m^2) = E(1/msum_{i=1}^m (x^{(i)} - hat{mu}_m)^2) = E(1/m sum_{i=1}^m (x^{(i)} - 1/m(sum_{i=1}^m x^{(i)}))^2)`
  `=E(1/m sum_{i=1}^m ((m-1)/m x^{(i)} - (m-1)/m E(x))^2) = (m-1)/m sigma^2`, which differs from `sigma^2` by its bias `-sigma^2/m`.
• To correct for this we can use the unbiased sample variance estimator (aka standard deviation): `\tilde{sigma}_m^2 = 1/(m-1) sum_{i=1}^m (x^{(i)} - hat{mu}_m)^2`.

Cross-Validation and Bias

• The accuracy measure one gets using cross validation turns out to be a biased estimator of the underlying expected correctness because the training data set is a strict subset of all of the data.
• Unfortunately, it is hard to prove confidence intervals around the accuracy one gets via cross validation.
• In most situations, the effect of the cross valdidation bias will tend to be conservative, i.e., the reported accuracy will be less than the actual, but one has to be careful in that one can sometimes end up tuning ones hyperparameters to falsely improve the cross validation accuracy.

Recurrent, MLP, and Feedforward Networks

• We are now going to start looking at networks of more than one neuron in more detail.
• Such a network is typically described by a directed graph where each of the internal nodes computes some neuron-like operation.
• To start we will assume that each of the node computes a perceptron. This kind of network is called a multi-layer perceptron (MLP) network.
• If we allow cycles in the graph, the network is called recurrent.
• For now, we will assume our network is not recurrent, so the underlying graph is a DAG. Such a network is called a feedforward network.
• To as much of a degree as possible, we would like to use the mathematical notation of linear algebra to say how our network computes things.
• To do this, we will require our network to be split into layers.
• We imagine our network receives its inputs as layer 0. The nodes in this layer are all in-degree 0.
• For layer `i >0 `, the only edges into nodes at layer `i` come from layer `i-1` and the only edges out of layer `i` go to layer `i+1`.
• To further simplify the notation, we will assume each node at layer `i` is connected to all the nodes at layer `i+1`. If we want to break a connection, we will multiply by a 0 weight on that input into the perceptron.
• We call the number of nodes in a given layer, the width of the layer, and the number of layers in the network, the depth of the network.

Computing a function from a feedforward MLP

• Let `g` be an activation function for a neuron, let `vec{w}` be its weights, and let `b` the bias. Let `vec{x}` be a vector of inputs. Then a single neuron computes the function `g(vec{w} \cdot vec{x} + b)`.
• Given an activation function `g`, let `vec{g}_m` be the vector-valued function that takes `vec{c} in RR^m` and computes the vector `[[g(c_1)],[...],[g(c_m)]]`.
• We will be interested in `vec{ReLU}_m`, `vec{tanh}_m`, `vec{step}_m`.
• To compute the value of two neurons, let `vec{w}_1` and `b_1` and `vec{w}_2` and `b_2` be the weights and biases of each neuron.
• Rather than take the dot product, we arrange the `vec{w}_i` as the rows of a matrix `W` and we use the two values `b_i` as the entries of a column vector `vec{b}`.
• Then the two perceptron network computes `vec{g}_2(W vec{x} + vec{b})`.
• In general, a layer of `m` neurons can be computed as `vec{g}_m(W vec{x} + vec{b})` where `W` and `vec{b}` each have `m` rows.
• If a feedforward neural network has layers `L_1, ..., L_n`, then it computes the function that results from composing the functions computed by the layers. I.e., `L_n(L_{n-1}(...L_2(L_1(vec{x}))...)).`
• If we had to code this, we could imagine storing all the weights in a 3D array with elements` W_{ij}^{(k)}`, where `k` says the layer, and `ij` give the weight of the `j`th input to the `i` neuron. Similarly, the biases become a 2D array with elements `b_{i}^{(k)}` where `k` says the layer and `i` says the bias to the `i` neuron. To keep things simple for now, let's assume all the neurons use the same activation.
• Given an input `vec{x}` we could then have a for loop that cycles over layers from `1` to `n`. For each layer, we look up its weights and biases, then use our equation above to compute the output of this layer. We then use this output as the input when we perform this next pass through the for loop.