Outline

• Learning
• In-Class Exercise
• Unsupervised Learning
• Supervised Learning

Learning

• An agent is learning if it improves its performance on future tasks after making observations about the world.
• Improvements to components of an agent, and techniques used to make them, depend on:
  • Which component is improved. For today's class we will focus on the mapping from conditions on the current state to actions.
  • What prior knowledge the agent already has. For today, we will look at learning using known input-output pairs and attempt to learn a general function. (Inductive Learning) rather than start form known rules and try to derive things from them (Deductive Learning).
  • What representation is used for the data and the component. Although learning techniques exists for all the representations considered this semester, today, we will look at a factored representation given by a vector of attribute values.
  • What feedback is available to learn from... Let's look at this in more detail.

Feedback to learn from

There are three main types of feedback that correspond to the three main types of learning:

• In unsupervised learning the agent learns patterns in the input even though no explicit feedback is supplied. A common task in this area is clustering: detecting potentially useful clusters of input examples.
• In reinforcement learning the agent learns from a series of reinforcements -- rewards or punishments. For example, the lack of a tip at the end of the journey gives the taxi agent an indication that it did something wrong. Winning a chess game give an agent an indication is did something right.
• In supervised learning the agent observes some examples input-output pairs and learns a function that maps from input to output.

In addition, to the above there is also things such as semi-supervised learning where we are given a few labeled examples and must make what we can of a large collection of unlabeled examples.

Unsupervised Learning

• For the rest of this class we will mainly be interested in supervised learning algorithms.
• Still, it is not hard to describe a simple mean-based hierarchal clustering algorithm, so let me do that one this slide.
• Suppose the data we want to learn clusters from example vectors `E_i = (e_(i1), ..., e_(i\n))` in `RR^n`.
• There are several possible distance functions we might choose between such vectors. For example, Euclidean distance square:
  `||E_i - E_j||^2 := sum_(k=1)^(n)( e_(ik) - e_(jk) )^2`.
• Suppose we have a data set of `N` examples. To cluster these, we use a data structure that maintains a forest of labelled trees. This forest is initialized to a list of single node trees labeled with `E_1, ... E_N`.
• Our algorithm has a while loop that cycles while our forest has more than one tree.
• In the body of this while we cycle over pairs of trees and compute the square euclidean distance of the root labels of these trees to find the two trees which are nearest each other.
• Once we have found these, we delete these two trees `T_i`, `T_j` from our forest. Let `E_i`, `E_j` denote the vectors labeling the roots of these trees. Let `E' = (frac(e_(i1) + e_(j1))(2), ..., frac(e_(i\n) + e_(jn))(2))`. We make a new tree `T'` with root label `E'` and subtree `T_i` and `T_j` and insert this tree into our forest.
• When the algorithm stops we have a tree. If we want the `k` most important clusters of our examples, we stop the algorithm when the forest has exactly `k` trees in it, and look at the examples in the leaves of these trees.

In-Class Exercise

• Suppose we have the three vectors (1,2,3), (4,5,6), (5,6,7).
• If we run the clustering algorithm of the last slide and we output two clusters, what would they be?
• Please post your solution to the In-Class Exercise Thread.

Supervised Learning

• The task of supervised learning is:

  Given a training set of `N` example input-output pairs
  `(x_1, y_1), ..., (x_n, y_n)`,
  where each `y_j` was generated by an unknown function `y=f(x)`, discover a function `h` that approximates the true function `f`.

• Here `x` and `y` can be any values; they need not be numbers.
• The function `h` is a hypothesis.
• Learning is a search through the space of possible hypotheses for one that performs well.
• What we want is that it continues to perform well on examples beyond the training set.
• To measure this we give it a test set of examples that are distinct from the training set and see how well it predicts their values.
• Sometimes the the function `f` is stochastic -- we want to learn a conditional probability distribution `\vec{P}(Y|x)`.
• When the output `y` is one of a finite collection of values, the learning problem is called classification.
• When `y` is always a number the learning problem is called regression.

Decision Trees

• We are now going to look at one of the simplest models in which we can do supervised inductive learning: decision trees.
• A decision tree represents a function that takes as input a vector of attribute values and returns a decision -- a single output value.
• The input and output can be discrete (for example, Patrons? in above images) or continuous (WaitEstimate? above).
• The above diagram shows an example decision tree for whether to wait for a table at a restaurant.
• We are going to focus on the case where the output is Boolean: true (a positive example) or false (a negative example). These will be the values of the leaves of our trees.
• A decision tree reaches its decision by performing a sequence of tests.
• Each internal node of our tree will correspond to querying the value of one of the attributes of the problem. For example, depending on the value of Patrons? we go down different branches of the above tree until we reach a leaf which tells us what to do.

Inducing decision trees from examples.

• An example for a Boolean decision tree consists of a pair `(vec(x), y)`. Here `\vec(x)` is a value for each attribute and `y` is either true or false.
• Such an example can be viewed as a path in a decision tree.
• For example, if the problem was whether to wait for a seat at a restaurant, the first coordinate of `vec(x)` might correspond to whether there are already patrons at the restaurant. There might be three possibilities: None, Some, and Full. But `vec(x)` says one specific choice. This corresponds to the decision at the top node of the tree.
• Similarly, the next attribute might be something like Wait Estimate? The particular value for this coordinate might be 30-60. It corresponds to a next level in the tree.
• Here we are envisioning that the top of the tree corresponds to the decision made by the first element of `\vec(x)`. Typically, we have a set E of training pairs. A tree might actually query the variable coordinates in `vec(x)` in whatever order it wants. Not all branches in the tree need to be the same length. A tree is consistent with the examples in E if for every pair `(vec(x), y)` in E if we follow the path down the tree according to the settings in `vec(x)` when we hit a leaf it has value `y`.

Decision Tree Learning Algorithm

Here is an algorithm for coming up with a decision for a set of examples.

function Decision-Tree-Learning(examples, attributes, parent_examples) returns a tree
    if examples is empty then return Plurality-Value(parent_examples)
    else if all examples have the same classification then return the classification
    else if attributes is empty then return Plurality-Value(examples)
    else
        A := argmax_(a in attributes)Importance(a, examples)
        tree :=  a new decision tree with root test A
        for each value v_k of A do
            exs := {e : e in examples and e.A = v_k}
            subtree := Decision-Tree-Learning(exs, attributes -  A, examples)
            add a branch to tree with label (A=v_k) and subtree subtree
        return tree

Choosing the Attribute to Test

• Our greedy search algorithm is designed to approximately minimize the depth of the final decision.
• When it comes to pick the next attribute to test, we would like to pick the attribute that goes as far as possible toward providing an exact classification of the examples.
• A really "good" attribute divides the examples into sets, each of which are all positive or all negative examples and so will be leaves of our decision tree.
• A "bad" attribute leaves the example sets with roughly the same proportion of positive or negative examples.
• For example, if we are trying to decide whether to wait for a seat at a restaurant. The number of Patrons might be a good attribute, but the type of the restaurant might be a bad attribute.
• To quantity "goodness" we need to define a notion of information gain.

Information Gain

• Information gain is defined in terms of entropy, the fundamental quantity in information theory.
• Entropy is a measure of the uncertainty of a domain variable; acquisition of information corresponds to a reduction in entropy.
• For example, a domain variable with only one value -- a coin that always comes up heads -- has no uncertainty. In this case, its entropy is defined as zero; thus, we gain no information by observing its value.
• On the other hand, a two-sided fair coin gives one bit of information if we know its value; a pair of coins gives two bits of info, etc.
• What about an unfair coin? Suppose one had a coin that was heads 99% of the time, tails 1%. Knowing that it is heads only gives us a little bit of information; but, knowing that it is tails, gives us a fair bit more.``

Entropy

• The entropy of a domain variable is defined as:
  Entropy: `H(V) = sum_(k)P(v_k)log_2frac(1)(P(v_k)) = - sum_(k)P(v_k)log_2P(v_k)`.
  We can view this as the expected number of bits of information we will learn.
• For a fair coin we have:
  `H(Fair) = -(0.5 log_2 0.5 + 0.5 log_2 0.5) =1`
• For the 99% loaded coin, we get
  `H(Loaded) = -(0.99 log_2 0.99 + 0.01 log_2 0.01) approx 0.08 bits`
• Since we are interested in Boolean domain variables, we will define this special case as:
  `B(q) = -(qlog_2 q + (1-q) log_2(1-q))`.
• For example, `H(Loaded) = B(0.99) approx 0.08`.
• One can plot this function, finds its maximum (is at 1/2), etc.

Choosing a Attribute Revisited

• If a training set has `p` positive examples and `n` negative examples, then the entropy of the goal attribute on the whole set is:
  `H(Goal) = B(frac(p)(p+n))`
• An attribute `A` with `d` distinct values divides the training set `E` into subsets `E_1, .. E_d`.
• Each subset `E_k` has `p_k` positive examples and `n_k` negative examples.
• So if we go along that branch we will have gained `B(frac(p_k)(p_k+n_k))` bits of information.
• A randomly chosen example from the training set has `A` set to the `k`th value with probability `frac(p_k + n_k)(p+n)`, the expected entropy remaining after testing attribute `A`:
  `Remai\nder(A) = sum_(k=1)^d frac(p_k + n_k)(p+n)B(frac(p_k)(p_k+n_k))`
• The information gain from the attribute `A` is the expected reduction in entropy:
  `Gai\n(A) = B(frac(p)(p+n)) - Remai\nder(A)`.
• So we to implement Importance function, we compute `Gai\n(A)` for each attribute `A` then pick as the next attribute the one with the largest gain.