Outline

• Local Search
• Hill Climbing
• In-Class Exercise

Local Search

• On Monday, after looking some more at `A^star` search, we began talking about local search algorithms.
• To refresh our memory I reinterate the last slide we discussed on Monday.
• So far as part of the solution to a problem, we have been interested in the sequence of actions which take us to the goal state.
• For some problems, this sequence of actions is not needed. For example, a solution to the 8-queens problem is eight non-attacking queens on a chess board -- we don't care so much how they were found.
• This property holds of many problems. For example, IC Design, factory-floor layout, job scheduling, network optimization, vehicle routing, portfolio management, etc.
• A local search algorithm operates using a single current node and generally moves to neighbors of that node. We don't keep track of the path history.
• In addition to just finding goals, local search algorithms are often used in optimization problems: problems in which the aim is to find the best state with respect to a objective function.
• The search process can be viewed in terms of the state space landscape. The landscape has both a "location" (which state we are currently at) and an "elevation" (what the objective function is for this state -- some real number).
• Plotting this scalar-valued function `c:S -> RR` we can find local maxima which would be states so that no neighboring state has as large/small a value of `c`, we might find global maxima/minima which have larger/smaller values than any other values corresponding to states in the state space. There might be other phenomena, like shoulders, saddles, "flat" local maxima, etc.
• We say a local search algorithm is complete if it always finds a goal if one exists. We say a local search algorithm is optimal, if it can always finds a global minimum/maximum.

Hill Climbing

function HILL_CLIMBING(problem) returns a state that is a local maximum
    current := MAKE_NODE(problem, INITIAL-STATE)
    loop do
        neighbor := a highest-valued successor of current
        if(neighbor.value ≤ current.value) return current.state
        current := neighbor

• The above is the so-called hill-climbing algorithm. It follows the strategy of looking at ones neighbors and always picking the neighbor which moves one in the most upward direction.
• If no such neighbors exist, we return a solution. This will be a local maxima, but not necessarily a global maxima.
• It has no memory of previous moves.
• For example, we could apply this algorithm to the eight queens problem: A state would consist of eight queens on the board, one per column.
• The successor state consist of all states which can be derived by moving one queen to a different square.
• The heuristic cost function `h` is the number of pairs of queen that are attacking each other.

Hill Climbing

• The left board above has `h=17`, the right `h=1`.
• Hill-climbing is sometimes called greedy-local search.
• Hill-climbing will usually choose randomly between two sucessors if they have the same cost.
• From a bad solution, hill-climbing often makes quite rapid progress to reach a close to optimal solution. For example, the second board on the right above is reached in five steps from the initial board on the left.
• Unfortunately, it can get stuck on local maxima (the right board above is one). You can also have ridges and plateaux.

Variants of Hill-Climbing

• One way to avoid local minima, and also to hopefully solve the plateaux problem, is to allow moves that move to the same value as the current state. i.e., a sideways move.
• This could lead to an infinite loop, depending on the plateaux, so typically one puts limit on the total number of sideways moves.
• With a limit of a 100 consecutive sideways moves, the chance of solving the 8-queens puzzle given a random starting configuration goes from 14% to 94%.
• The average solution though takes much longer now 21 moves on success, 64 on failure.
• Another variant of hill-climbing is stochastic hill climbing: Choose an uphill move at random from the uphill moves with probability proportional to the steepness of the move. This typically converges to a solution more slowly than deterministic hill-climbing, but tends to find better solutions.
• One can make hill-climbing complete by allowing restarts. In random-hill climbing with restarts one runs the hill climbing algorithm from a randomly chosen initial state, sees if one gets a solution, and if not, picks a new random initial state and try again.
• If each hill-climbing search has a probability `p` of find a solution than the expected number of restarts is `1/p`.
• For the 8-queen problem with no sideways moves `p approx 0.14` and roughly 7 restarts on average are needed to find the goal.
• This technique can be used to solve instances of the `n`-queens problems into the millions in under a minute.

In-Class Exercise

• Suppose we wanted to do stochastic hill climbing on the above grid where the utility's of each sqaure are as indicated.
• The agent starts in the square with the @ in it. The goal square would be a square of locally maximal utility.
• We want our choice of action to have probability proportional to its utilities, so that each possible action from a given square to an adjacent square has non-zero probability.
• Suggest an algorithm to do this. Then run your algorithm by hand and say what it does at each step until you get to the 9 in the upper right corner.
• Please post your solution to the Sep 21 In-Class Exercise Thread.

Simulated Annealing

• Hill-climbing doesn't allow for downwards moves.
• We next consider an algorithm that does, simulated annealing.
• In metallurgy, annealing is the process used to harden glass or metal by heating a substance to a high temperature and then gradually cooling it. This allows the material to reach a lower crystalline energy state.
• In this context we are looking at a minimization problem rather than a maximization problem: We imagine our cost is the energy of the system and we are trying to get the system into as low an energy state as possible.
• For example, one could imagine the problem of getting a ping pong ball into the deepest crevasse in a bumpy surface.
• The way annealing tries to do this is to shake the surface a whole bunch initially, hoping the ball will go generally downwards and not "bump up" too often. Then one gradually slows down the shaking. Keeping some shaking allows us to climb out of local minima, but we gradually reduce the shaking as we get closer and closer to what we think is a minima.
• Simulated annealing (Khachaturyan et al 1979 adapted from Metropolis et al 1953) is a computer simulation of this idea. It has been used in VLSI layout since the 1980s.

Simulated Annealing Algorithm

function SIMULATED-ANNEALING(problem, schedule) returns a solution state
    inputs: problem, a problem
            schedule, a mapping from time "temperature"
    current := MAKE-NODE(problem.INITIAL_STATE)
    for t = 1 to infty do
        T := schedule(t) //typically as t gets larger schedule(t) is smaller
        if(T == 0) return current //when Temperature is 0 return state
        next := a randomly selected successor of current
        DeltaE := next.value - current.value //estimated change in energy
        if(DeltaE < 0) current := next 
        else current := next with probability e^(-DeltaE/T)

Local Beam Search

• Rather than keep track of only one state at a time, one could imagine keeping track of `k` states.
• This is what local beam search does (Reddy 1977). It begins with `k` randomly generated states.
• At each step the successors of all these states are generated.
• If any is the goal, it stops.
• Otherwise, the `k` best successors are chosen for the next round.
• Notice this is different from doing `k` random restarts as the `k` best states in a given round might not come from the successors of all `k` different current states, but instead may come from only some of them.
• As with hill-climbing, beam search can be made stochastic.
• The `k` states can be viewed as a pool of candidates and their successors as offspring.

Local Search via Genetic Algorithms

• A genetic algorithm is a stochastic beam search in which successor states are generated by combining two parent states rather than by modifying a single state.
• Again, an analogy can be made with natural selection, but, whereas, in the stochastic local beam search case offspring are generated by asexual reproduction, in the genetic algorithm case they are generated by sexual reproduction.
• GAs begin with a set of `k` randomly generated states called the population.
• Each state (individual) is represented as a string over a finite alphabet. For example in the 8-queens, the strings might consist of the position in each column of a queen.
• The initial states are ranked according to some fitness function which should be higher for better states. For 8-queens, one might use the number of non-attacking queens.
• One selects individuals at random according to fitness to "breed".
• A cross-over point is chosen randomly from the positions in the string to generate new strings. A mutation might then be applied.
• Finally, this whole process is repeated on the new strings or until a goal or closeness to a goal is achieved.
• Genetic algorithms of various types have been explored as far back as Alan Turing in 1950.