Introduction

So far we have only been considering single agent environments.
We now switch to looking at multi-agent environments where each agent needs to consider the actions of other agents and how they affects its own welfare.
In particular, we are going to look at competitive environments, which are environments in which agents' goals are in conflict.
Such environments, give rise to adversarial search problems, which are often called games.
We will consider games that involve turn taking, have two players, and are zero sum with perfect information.
Here perfect information means the game is deterministic and the environment visible to all playing; zero sum means if Player 1 gets payoff (+a), then player 2 gets payoff (-a). It is possible for both players to get pay-off 0.

Games defined More Formally

We consider games with two players MAX and MIN. A game consists of:

`S_0` the initial state which specifies how the game is set up at the start.
PLAYER(s): Defines which player has a move in a state `s`.
ACTIONS(s): Returns the set of legal moves in a state
RESULT(s, a): The transition model which defines the results of a move.
TERMINAL-TEST(s): A terminal test that determines when the game is over. States where this evaluates to true are called terminal states.
UTILITY(s, p): A utility function (aka an objective function) defines the final numeric value for a game that ends in terminal `s` for a player `p`. (Use to find out who won.)

Games Trees

The initial state, ACTIONS function, and RESULT function define the game tree for the game -- a tree where nodes are game states and the edges are moves.
Below is an example game tree for Tic-Tac-Toe:
From the initial state MAX has nine possible moves. MAX and MIN take turns, MAX placing X's, MIN placing O's, until we reach leaf nodes corresponding to terminal states.
The number on each leaf indicates the utility value of that terminal. High values are assumed to be good for MAX and bad for MIN.
The tic-tac-toe game is relatively small in that it has fewer than 9! = 362,880 terminal nodes.
Chess on the other hand has `10^(40)` or so nodes, so the game tree cannot be easily created in toto.
A search tree is a sub-tree of the full game tree, that contains enough nodes to allow a player to determine what move to make.

Optimal Strategies

In a normal search problem, the optimal solution is a sequence of actions leading to a goal state -- a terminal state that is a win.
In adversarial search, MAX has to worry about MIN's next move and vice-versa.
MAX, the player who moves first, wants to come up with a strategy for what to do contingent upon MIN playing his best.
An optimal strategy is a sequence of contingent decisions that will lead to outcomes at least as good as any other strategy when one is playing an infallible opponent.
In order to try to figure out an optimal strategy is useful to look at the game tree.

Minimax

To find the optimal strategy for a player, one can compute the minimax value, MINIMAX(n), of each node `n` in the game tree.
This value is the utility of being in the given state, assuming that both players play optimally from there to the end of the game.
`MINIMAX(s) := { ( UTILITY(s), if TERMINAL-TEST(s)), (max_(a in ACTION(s))MINIMAX(RESULT(s, a)),if PLAYER(s) = MAX), (min_(a in ACTION(s))MINIMAX(RESULT(s, a)),if PLAYER(s) = MIN):}`

Minimax Example

Consider the above two move game. In this area, each move is often called a ply.
The utility of each terminal node is listed under it. This would be the minimax value for that node.
At node B, it is MIN's turn, and there are three possible moves: `b_1`, `b_2`, and `b_3`. Of these, `b_1` has the smallest minimax value, so if MIN is playing optimally, MIN will choose `b_1` and so the minimax value of node B is 3.
In a similar fashion, we can compute the minimax values of node C as 2 and node D as 2.
If we move up to node A. Here it is MAX's turn. As MAX wants to get the highest payout possible, MAX will choose a_1 which leads to node B, as node B has the highest minimax value amongst B, C, D.
Thus, the minimax value of node A is 3.
Since A is the root of the current tree, the branch from A, namely `a_1`, which has the minimax value for A, tells MAX what to do next. So it is called the minimax decision for the node.

The Minimax Algorithm

The minimax algorithm below computes the minimax decision from the current state. So we could use it to actually make an agent that could play a game.

Here `argmax_(a in s) f(a)` returns the element `a` of set `S` that has the maximum value of `f(a)`.

function MINIMAX-DECISION(state) return an action //it is assume is MAX's turn
    return argmax_(a in ACTION(s)) MIN-VALUE(RESULT(state, a))

function MAX-VALUE(state) return a utility value
    if (TERMINAL_TEST(state) == true) then return UTILITY(state)
    v := -infty
    for each a in ACTION(state) do
      v := MAX(v, MIN-VALUE(RESULT(s, a))) 
    return v

function MIN-VALUE(state) return a utility value
    if (TERMINAL_TEST(state) == true) then return UTILITY(state)
    v := infty
    for each a in ACTION(state) do
      v := MIN(v, MAX-VALUE(RESULT(s, a))) 
    return v

Remarks on Minimax Algorithm

If the maximum depth of the game tree is `m`, and expected branching factor is `b`, then time complexity of minimax is `O(b^m)`.
It is possible depending on implementation to have a linear space complexity, ergo space complexity is not an issue.
`O(b^m)` for time complexity is impractical.
Can we do better?
There are a variety of pruning strategies that allow a player to quickly determine that a branch is not worth following.
We conclude today by looking at one such strategy that can improve the time complexity to `O(b^(m/2))`.

Alpha-beta Pruning

Consider two level tree:
MAX has a current backed-up value of `3`. On the first branch under node C, MIN has a 2, so the largest value MIN could pick is 2 and this is less than 3. So MAX doesn't need to expand the other two nodes under C. The backed-up value `3` is called the alpha value, and ignoring the two remaining branches under C is called doing an alpha pruning, or alpha cut of the tree.
The analogous thing for MIN is a beta value. And MIN can do beta pruning (make beta cuts) of tree.
For MIN, the beta value is the largest value as opposed to alpha's smallest value.
On average, alpha/beta pruning makes the minimax algorithm time complex `O(b^(m/2))`.
So in the same amount of time one can view a tree twice as deep as straight minimax.
In order, to achieve this one needs to expand the nodes somehow in close to best first order. For chess, one can get within a factor of two of best first, by considering capture moves before, threat moves, before forward, before backward moves.

Imperfect Real-Time Decisions

For a game like chess, we can't completely expand out the game tree.
So we can't determine all the leaf nodes, so how do we do minimax?
Typically, we have a a heuristic function Eval(s) which gives an estimate for how good a given state is.
We also have a CUTOFF-TEST(s,d) which returns whether or not to keep evaluating given we are at a depth `d` and in state `s`.
Given these we can define a heuristic minimax as:
`H-MINIMAX(s, d) :=`
`{ ( EVAL(s), if CUTOFF-TEST(s,d)), (max_(a in ACTION(s))H-MINIMAX(RESULT(s, a), d+1),if PLAYER(s) = MAX), (min_(a in ACTION(s))H-MINIMAX(RESULT(s, a), d+1),if PLAYER(s) = MIN):}`

Adversarial Games

Outline