Adversarial search

So far we have only been considering single agent environments.
We now switch to looking at multi-agent environments where each agent needs to consider the actions of other agents and how they affects its own welfare.
In particular, we are going to look at competitive environments, which are environments in which agents' goals are in conflict.
Such environments, give rise to adversarial search problems, which are often called games.
We will consider games that involve turn taking, have two players, and are zero sum with perfect information.
Here perfect information means the game is deterministic and the environment visible to all playing; zero sum means if Player 1 gets payoff (+a), then player 2 gets payoff (-a). It is possible for both players to get pay-off 0.

Games defined More Formally

We consider games with two players MAX and MIN. A game consists of:

`S_0` the initial state which specifies how the game is set up at the start.
PLAYER(s): Defines which player has a move in a state `s`.
ACTIONS(s): Returns the set of legal moves in a state
RESULT(s, a): The transition model which defines the results of a move.
TERMINAL-TEST(s): A terminal test that determines when the game is over. States where this evaluates to true are called terminal states.
UTILITY(s, p): A utility function (aka an objective function) defines the final numeric value for a game that ends in terminal `s` for a player `p`. (Use to find out who won.)

Games Trees

The initial state, ACTIONS function, and RESULT function define the game tree for the game -- a tree where nodes are game states and the edges are moves.
Below is an example game tree for Tic-Tac-Toe:
From the initial state MAX has nine possible moves. MAX and MIN take turns, MAX placing X's, MIN placing O's, until we reach leaf nodes corresponding to terminal states.
The number on each leaf indicates the utility value of that terminal. High values are assumed to be good for MAX and bad for MIN.
The tic-tac-toe game is relatively small in that it has fewer than 9! = 362,880 terminal nodes.
Chess on the other hand has `10^(40)` or so nodes, so the game tree cannot be easily created in toto.
A search tree is a sub-tree of the full game tree, that contains enough nodes to allow a player to determine what move to make.

Optimal Strategies

In a normal search problem, the optimal solution is a sequence of actions leading to a goal state -- a terminal state that is a win.
In adversarial search, MAX has to worry about MIN's next move and vice-versa.
MAX, the player who moves first, wants to come up with a strategy for what to do contingent upon MIN playing his best.
An optimal strategy is a sequence of contingent decisions that will lead to outcomes at least as good as any other strategy when one is playing an infallible opponent.
In order to try to figure out an optimal strategy is useful to look at the game tree.

Minimax

To find the optimal strategy for a player, one can compute the minimax value, MINIMAX(n), of each node `n` in the game tree.
This value is the utility of being in the given state, assuming that both players play optimally from there to the end of the game.
`MINIMAX(s) := { ( UTILITY(s, MAX), if TERMINAL-TEST(s)), (max_(a in ACTION(s))MINIMAX(RESULT(s, a)),if PLAYER(s) = MAX), (min_(a in ACTION(s))MINIMAX(RESULT(s, a)),if PLAYER(s) = MIN):}`

Minimax Example

Consider the above two move game. In this area, each move is often called a ply.
The utility of each terminal node is listed under it. This would be the minimax value for that node.
At node B, it is MIN's turn, and there are three possible moves: `b_1`, `b_2`, and `b_3`. Of these, `b_1` has the smallest minimax value, so if MIN is playing optimally, MIN will choose `b_1` and so the minimax value of node B is 3.
In a similar fashion, we can compute the minimax values of node C as 2 and node D as 2.
If we move up to node A. Here it is MAX's turn. As MAX wants to get the highest payout possible, MAX will choose a_1 which leads to node B, as node B has the highest minimax value amongst B, C, D.
Thus, the minimax value of node A is 3.
Since A is the root of the current tree, the branch from A, namely `a_1`, which has the minimax value for A, tells MAX what to do next. So it is called the minimax decision for the node.

Quiz

Which of the following is true?

Local Beam search is an evolutionary hill-climbing algorithm where we breed possible solutions with each other to make new solutions.
The sum of Manhattan distances of misplaced pieces is the cost of a relaxed solution to the 8-puzzle problem.
The Hill-climbing algorithm is always complete and optimal.

The Minimax Algorithm

The minimax algorithm below computes the minimax decision from the current state. So we could use it to actually make an agent that could play a game.

Here `argmax_(a in s) f(a)` returns the element `a` of set `S` that has the maximum value of `f(a)`.

function MINIMAX-DECISION(state) return an action //it is assume is MAX's turn
    return argmax_(a in ACTION(state)) MIN-VALUE(RESULT(state, a))

function MAX-VALUE(state) return a utility value
    if (TERMINAL_TEST(state) == true) then return UTILITY(state, MAX)
    v := -infty
    for each a in ACTION(state) do
      v := MAX(v, MIN-VALUE(RESULT(state, a))) 
    return v

function MIN-VALUE(state) return a utility value
    if (TERMINAL_TEST(state) == true) then return UTILITY(state,MAX)
    v := infty
    for each a in ACTION(state) do
      v := MIN(v, MAX-VALUE(RESULT(state, a))) 
    return v

Remarks on Minimax Algorithm

That always playing according to the minimax decision guarantees a payoff of at least the minimax value regardless of how one's opponent plays is due to John von Neumann (1928) -- his Minimax Theorem is actually a little more general.
If the maximum depth of the game tree is `m`, and expected branching factor is `b`, then time complexity of minimax is `O(b^m)`.
It is possible depending on implementation to have a linear space complexity, ergo space complexity is not an issue.
`O(b^m)` for time complexity is impractical.
Can we do better?
There are a variety of pruning strategies that allow a player to quickly determine that a branch is not worth following.
We look next at one such strategy that can improve the time complexity to `O(b^(m/2))`.

Alpha-beta Pruning

Consider two level tree:
MAX has a current backed-up value of `3`. On the first branch under node C, MIN has a 2, so the largest value MIN could pick is 2 and this is less than 3. So MAX doesn't need to expand the other two nodes under C. The backed-up value `3` is called the alpha value, and ignoring the two remaining branches under C is called doing an alpha pruning, or alpha cut of the tree.
The analogous thing for MIN is a beta value. And MIN can do beta pruning (make beta cuts) of tree.
For MIN, the beta value is the largest value as opposed to alpha's smallest value.
On average, alpha/beta pruning makes the minimax algorithm time complexity `O(b^(m/2))`.
So in the same amount of time one can view a tree twice as deep as straight minimax.
In order, to achieve this one needs to expand the nodes somehow in close to best first order. For chess, one can get within a factor of two of best first, by considering capture moves before, threat moves, before forward, before backward moves.
According to Wikipedia and our book, the alpha-beta pruning technique was discovered multiple times: It was described by John McCarthy to his students in 1956, was used by Newell and Simon in 1958 in NSS Chess, and was used by Kotok in McCarthy-Kotok Chess 1962. It was published by Brudno in 1963, but also by Hart and Edwards in 1961. Arthur Samuel, Richards, Levine also have a claim in the same period. Asymptotic optimality of alpha-beta pruning among all fixed-depth game-tree search algorithms was shown by Pearl in 1982.

Imperfect Real-Time Decisions

For a game like chess, we can't completely expand out the game tree.
So we can't determine all the leaf nodes, so how do we do minimax?
Typically, we have a a heuristic function Eval(s) which gives an estimate for how good a given state is.
We also have a CUTOFF-TEST(s,d) which returns whether or not to keep evaluating given we are at a depth `d` and in state `s`.
Given these we can define a heuristic minimax as:
`H-MINIMAX(s, d) :=`
`{ ( EVAL(s), if CUTOFF-TEST(s,d)), (max_(a in ACTION(s))H-MINIMAX(RESULT(s, a), d+1),if PLAYER(s) = MAX), (min_(a in ACTION(s))H-MINIMAX(RESULT(s, a), d+1),if PLAYER(s) = MIN):}`

Stochastic Games

A stochastic game is a game (adversarial search problem) which also has a random element to it.
Examples of such games would be things like Backgammon, Parcheesi, or Sorry
To simulate things like rolling dice, we can add chance nodes after player turns in our game tree (as in the above game tree).
Such chance layers might be before or after every player move as in backgammon.
We can modify the minimax function to handle such moves by considering the expected value of a board rather than its exact value. This gives the EXPECTIMINIMAX function (MICHIE 1966): `EXPECTIMINIMAX(s) := `
` { ( UTILITY(s), if TERMINAL-TEST(s)), (max_(a in ACTION(s))EXPECTIMINIMAX(RESULT(s, a)),if PLAYER(s) = MAX), (min_(a in ACTION(s))EXPECTIMINIMAX(RESULT(s, a)),if PLAYER(s) = MIN), (sum_rP(r)EXPECTIMINIMAX(RESULT(s, r)),if PLAYER(s) = CHANCE):}`
In the last line `r` is cycling over the elementary random events (like different dice values), `P(r)` denotes the probability of that event, so the sum is the expected value of the board given the random events.
Ballard (1983) showed a way to modify alpha-beta pruning to this situation. One way to modify, alpha-beta pruning is to compute upper and lower bounds on the values of boards from chance moves, and use those when determining cuts.

Partial Observable Games

Often in a card game, you can see your cards, but not your opponents, so the game is partially observable
You know though that the game is being played with a standard card deck of 52 cards.
One way to choose moves in such a situation is via some kind of Monte Carlo simulation you do the following steps some fixed number of times:
1. Pick a random permutation on the cards you can't see
2. Run minimax with alpha-beta pruning on that
3. See what move you do, and record it.
From the moves that were chosen by doing the above, you then pick the one that occurred the most often.

Minimax - Alpha-Beta Pruning

Outline