Outline

• Syllabus Recap
• Slidy
• Introduction to Deep Learning
• Quiz
• What is Learning?

Syllabus Recap

I'm back from Computer Analysis of Images and Patterns 2017! Here's a brief summary of the CS 256 Syllabus Vasudha covered last week:

• Instructor: Chris Pollett (Office: MH214, Hours: MW 5:45pm-7pm)
• Add Codes: Only given out in class as I see space become available.
• Class Homepage/Syllabus: http://www.cs.sjsu.edu/faculty/pollett/256.1.17f/
  Has Prereqs, Lectures Notes, Videos, Homeworks, Important Dates, HW Submission, Your Grades, Class Discussion Board etc.
• Topics: How to understand, build, train, neural networks with many layers.
• Programming Languages: Python, Caffe, Tensor Flow (will introduce each)
• Exams: 2 Midterms (15pts each), Final (20pts), dates on classpage syllabus.
• Homeworks: 5 projects (10pts each). Each will involve code and exercises. Can work in groups of up to 3.
• Quizzes: Each Monday. 1pt each, up to 10pts total. Quiz points can only be used to replace a low or missed HW score.
• In-Class Exercises: Each Wednesday. 1pt each. Need to write-up and post to class discussion board. Points can only be used to replace points lost on midterms or finals, up to half of what you lost on a given test.
• Other Details: Regrades, test accommodations, academic policies, check class homepage...

Introduction to Deep Learning

• Deep learning is a branch of machine learning that focuses on learning algorithms that allow machines to understand the world through a hierarchy of intermediate learned concepts.
• It has been given various trendy names in the past such as cybernetics (1940-1960s) and connectionism (1980-1990s) with the term deep learning arising in the mid-2000s.

AI without Deep Learning

• Many successes in artificial intelligence (AI) such as Deep Blue's beating world chess champ Gary Kasporov in 1997 did not involve learning intermediate representations. In the Deep Blue, a brute force algorithm was used together with a large table lookup of end games.
• Early attempts at giving machines an understanding of the world such as Cyc (Lenat and Guha 1989) involved hand-coding intermediate concepts in a knowledge base.
• Simple machine learning algorithms such as logistic regression (curve fitting where the dependent variable has values 0 or 1) or naive Bayes (estimating the conditional probability of something based on the Baye's formula from probability) allow one to automatically learn a value or make a classification based on hand-coded features.
• These techniques have shown success in problems such as whether to perform cesarean section or not (Mor-Yosef, 1990) and in spam versus no spam email classification.
• They do not involve learning what features are important.

Quiz

Features and Depth of Learning models

• Learning good features/a good representation is important for many problems. For example, if one doesn't having a counting board or abacus at hand, it is much easier to multiply if we use the Arabic numeral representation of numbers rather than Roman numerals representation.
• The sophistication of a network of representations or features can be measured in couple of ways:
  1. We can look at the number of sequential (non-parallelizable) operations that need to be executed in order to evaluate the architecture.
  2. We can look at the representations needed for a learning model as nodes in a directed graph where an edge indicates a representation depending on another, then the longest path (may involve cycle) that needs to be executed in the graph to evaluate the model represents the depth of the model.
• There is no hard and fast rule as to when an architecture/model becomes deep, but we'll usually assume the model has depth at least 3.
• Two common approaches to learn new features/representations are artificial neural networks and deep probabilistic networks.
• This semester we will focus on artificial neural networks.

Artificial Neurons

• Humans seem capable of learning intermediate representation, so it is little surprise that one approach to getting machines to learn such representations is to mimic what humans do.
• It has been known since Santiago Ramon y Cajal in the 1880s that the basic unit of computation in the nervous system and the brain is the neuron.
• In the cybernetics era, McCulloch and Pitts in 1943 proposed a computational model for a neuron and formulated it in terms of the mathematical logic of Russell and Whitehead 1910 (the book that takes a few hundred pages to prove `1+1 = 2`). They optimistically hoped that since neurons could be expressed in terms of mathematics that theorems about psychiatry would follow.
• Rosenblatt in two papers in 1958, 1962 gave the simpler perceptron model of a neuron, as well as the first learning algorithm for it. The image above shows the custom hardware he used to implement a perceptron to recognize simple letters images (like X).
• At about the same time, Widrow and Hoff 1960 proposed the ADALINE (adaptive linear element) neural learning algorithm.
• Minsky and Papert 1969 showed that one layer of perceptrons is insufficient to learn to the parity function based on training examples.

Artificial Neural Networks and Representation Learning

• This killed off a lot of research on neural networks until three advances:
  1. In the connectionism era, Rumelhart, et al (1986) developed and popularized the backprogation algorithm.
  2. Models of recurrent networks were developed by Hochreiter (1991), Bengio et al (1994), and Hochreiter and Schmidhuber (1997).
  3. LeCun et al (1998) inspired by the visual system in the brain showed how to reduce the number of weights needed to recognize features and developed the first multi-layer, deep neural network for handwriting recognition.
• Lack of fast enough hardware and lack of data sets of sufficient size prevented this deep neural network approach from advancing until the mid-2000s.

Deep Neural Network Era

• In the mean time other approaches to machine learning such as support vector machines (SVMs) became popular. These use the kernel trick (basically, a hand-coded "good representation") to avoid the limitations of single perceptrons.
• By the mid-2000s, the deep learning era began as both hardware and data set size caught up, so that deep neural networks could be trained in a reasonable time.
• Whereas LeCun net might have involved thousands of neurons, the largest trained nets as of 2014 (GoogleNet) have around 10 million neurons. The number of connections between neurons in GoogleNet is on the order of a few hundred.
• The MNIST data set used by LeCun et al had around 100,000 items, whereas, the 2011 street view house number dataset has around a million items.
• Trained deep neural networks have been used to halve the rate of errors in computer models of speech recognition (Dahl 2010), to improvement in image recognition (Krizhevsky, 2010), and pedestrian detection (Seramet et al 2013).
• Trained deep neural networks have led to networks that have super human performance for a variety of tasks: traffic sign classification (Ciresan et al 2012), Atari video game playing (Mnih et al 2015).

Let's learn about how to build deep neural networks!

• This semester we are going to learn how to design and train multi-layer neural networks to solve interesting problems.
• We will learn how to code neural networks, but also we will learn why neural networks and particular arrangements of such networks are a useful model for learning.
• To start I want to discuss what is learning?
• I also want to discuss why neural networks are a good fit for many learning tasks (Wednesday).
• As I do this, I will try to review some of the math we will need this semester.

What is Learning? Kinds of Learning algorithms

• A learning algorithm is an algorithm which makes use of observations of the world to either improve its own performance on future tasks or to improve the performance of some other program on future tasks.
• Learning algorithms can be grouped according to the kind of observations they get as inputs:
  • A supervised learning algorithm observes some examples input-output pairs and learns a function that maps from input to output. I.e., Given points `(\vec{x}_1, y_1), ...(\vec{x}_k, y_k)`, we want to learn a simple function `f` such that for all `i`, `f(\vec{x}_i) approx y_i`, and such that `f` makes reasonable predictions on new data. If there are only a fixed set `1,..., k` of values (classes) that the `y_i` can be in, we call this kind of learning classification.
  • An unsupervised learning algorithm learns patterns in the input even though no explicit feedback is supplied. A common task in this area is clustering: detecting potentially useful clusters of input examples.
  • A reinforcement learning algorithm learns from a series of reinforcements -- rewards or punishments. For example, the lack of a tip at the end of the journey gives a taxi agent an indication that it did something wrong. Winning a chess game gives a chess playing agent an indication is did something right.

Learning algorithms Tasks

• Some tasks we might use a learning algorithm to solve are:
  • Classification How to assign a label (for example, spam/no spam) based on input vector `vec{x}`
  • Classification with Missing Inputs Same task as above but some of the components of `vec{x}` might be missing. For example, `vec{x}` might be a vector of medical test results, one of the tests of which is missing.
  • Regression Find the best curve of some type (for example, a line) through a set of data points. Then use this curve to predict previous unseen values. For example, might train on predicted claim amounts an insured person might make for certain circumstances, then use this to predict expected new claim amounts.
  • Transcription Observe non-textual data and convert it to text. For example, images of street signs.
  • Machine Translation Convert a sequence of symbols from one language to a sequence of symbols in another. For example, English to Chinese.
  • Structured Output Take unstructured input data and convert to data with annotated relationships. For example, label parts of speech of an English sentence.
  • Anomaly Detection Find atypical events in an event sequence. For example, misuses of a credit card.
  • Synthesis and Sampling Generate new examples similar to training data. Might use for automatic scene generation in video games.
  • Imputation of Missing Values This is similar to classification with missing inputs. Task is given a `vec{x}` where some of the coordinates are missing predict those coordinates. For example, the predict movie rating of someone for a movie they didn't rate based on ones they did.
  • Denoising given a corrupted input `\tilde{\vec{x}}` compute a clean one `\vec{x}`.
  • Density Estimation Learn a `p_{mbox{model}}: RR^n->RR` a probability density function based on training examples. This can be used for things like the imputation of missing values problem.

Learning Algorithm Performance

• To evaluate the abilities of a machine learning algorithm, we must come up with a way to measure its performance.
• For tasks like classification, accuracy of the model is often measured. This is the proportion of the examples on which the model produced the correct output.
• Error Rate is an equivalent measure: It is the proportion of the inputs on which the model got the answer wrong.
• For tasks where it doesn't make sense to measure accuracy, for example, density estimation, a continuous valued score might be used.
• A common approach is to report the average log-probability the models assigns to some examples.
• Usually we are interested in how well the machine learning algorithm performs on data that it has not seen before.
• We call this data that is separate from the training data, the test set.

Compression versus Learning

• Typically, there are two main ingredients that one trades offs when designing a learning algorithm: compression and generality.
• A compression algorithm `c` takes as input a long string `y` (for example, the 1's and 0's of an image) and tries to come up with a short string `c(y) = x` such that when a decompression algorithm `d` is applied to `x`, we have `d(x) approx y`.
• For supervised learning, we can imagine `c` as training algorithm that tries to find a short description of all the data. I.e., `c` takes as input a string which is the concatenation of `(\vec{x}_1, y_1), ...(\vec{x}_k, y_k)` and outputs some `w` which is at most the size of this concatenation. The function `d` now takes two arguments `(w, x)`, and we want `d(w,x_i) approx y_i`. So `d` is an evaluator on the trained model `w` and on inputs `x`.
• If the length of `w` is always fixed size, regardless of the amount of training data, we call the learning algorithm, a parametric learning algorithm, otherwise, we will call it a non-parametric learning algorithm.
• As we will see, neural network training is an example of a parametric learning algorithm. On the other hand, `k`-nearest neighbors which gives a point `x` the value `y`, based on what a simple function such as majority or average of the values of the `k` nearest `x_i`'s to `x` is an example of a nonparametric learning algorithm.
• If `y` is `n` bits long, it is one of `2^n` many `n` bits strings. On the other hand, there are only
  `1+2+2^2 + \cdots +2^{n-1} = (2-1)(1+2+2^2 + \cdots + 2^{n-1}) = 2^n -1`
  binary strings of shorter length. So not all strings of length `n` can be compressed.
• So we cannot expect that all data sets can be parametrically learned. If we allow larger `w`'s, it is more likely we can learn, but then we run into our second trade-off...