Introduction

On Monday, we started talking about minimum spanning trees.
The motivation was to try to come up with a way to connect a set of pins on a circuit board so as to minimize the amount of wire needed. We can view pins as vertices and connecting two pins as having an edge between them
We modeled this problem as:
Input: `G=(V,E)` be an undirected graph and let `w:E -> RR` be a weight function (think of as the amount of wire in an edge).
Output: An acyclic subset `T subseteq E` that connects all of the vertices and whose total weight
`w(T) = sum_((u,v) in T)w(u,v)`
is minimized.
We said a solution to the above problem will be tree that touches with at least one edge every vertex (i.e., a spanning tree) of minimal weight. i.e., a minimum spanning tree.
We said to solve this problem we were going to greedily grow our tree from a subset of our vertices `A` while maintaining the loop invariant:
```
Prior to each iteration, A is a subset of
some minimum spanning tree.
```

Our Generic Algorithm Revisited

A edge not in `A` that can be added to `A` and preserve the loop invariant above is called safe.

On Monday we gave this generic pseudo-code for computing a minimum spanning tree:

GENERIC-MST(G,w)
1 A = 0
2 while A does not form a spanning tree
3    find an edge (u,v) that is safe for A
4    A = A union {(u,v)}
5 return A

This of course relies on being able to choose a "safe" edge. So let's look at two ways to do this.

Krushkal's Algorithm

At any stage in the construction so far `A` will be a forest of trees.
We assume for this algorithm we have a way to represent dynamic sets, and that we have an algorithm that when given two dynamic sets `C_1` and `C_2` to compute their union.
A forest can be viewed as a set of sets. It might support operations: MAKE-SET(v) where `v` is a vertex which adds a new set to the forest which contains just `v`, FIND-SET(v) which takes a vertex and searches through the sets in the forest for the set containing `v`, returning a reference to it, and UNION(u,v) which finds the set containing `u` and the one containing `v` and combines them into a single set.
If we were using linked-lists, union could be done using list concatenation. Linked-lists are slow for some operations such as search, so a better choice might be a data structure from Chapter 21.
Let `(u,v)` denote the least weight edge connecting two distinct trees `C_1` and `C_2` in our forest.
The book shows this edge will be safe.

Assuming this and using our generic basic algorithm of the previous slide, we get Krushkal's algorithm for MST:

MST-KRUSHKAL(G, w)
1 A = 0
2 for each vertex v in G.V
3    MAKE-SET(v)
4 sort the edges of G.E into nondecreasing order by weight w
5 for each edge (u,v) in G.E with respect to this order
6    if FIND-SET(u) != FIND-SET(v) 
7       A = A union {(u,v)} 
8       UNION(u,v)
9 return A

Example of Krushkal's Algorithm

At each step above the darkened edges indicate the current content of our set `A` so far.
In addition to `A` at each step we are maintaining a forest `F`.
Initially, this `F` consists of trees of only a single vertex.
In (a), we find the minimal edge that connects two trees in `F`. In this case, `1`. weconnect those trees by calling the UNION method of `F` and add the edge to `A`.
So we now have an updated forest `F`, and an updated set of edges `A`. We then pick as edge of smallest weight that connects two trees in `F`. In this case, `2` and repeat the process ...

Runtime of Krushkal's Algorithm

The running time of Krushkal's algorithm depends on how we implement our forest (disjoint set) data structure.
Line 1 of the algorithm takes O(1) time.
Line 4 (sorting the edges) takes `O(|E|log|E|) = O(|E|log(|V|^2)) = O(2|E|log(|V|)) = O(|E|log(|V|))`.
The for loop of line 2-3 makes |V| MAKE-SET and the for of lines 5-8 performs O(|E|) FIND-SET and UNION operations on the forest.
Using the data structure of Chapter 21 these operations could be implemented so that the total cost would be `O((|V| +|E|)alpha(|V|))` where `alpha(n)` is the inverse Ackermann function (grows slower that `log^star n`)
Since our graph is connected `|E| ge |V| -1`, so this reduces to `O(|E|alpha(|V|))`, and as `alpha(|V|) in O(log |V|)` when we add this with the line 4 time gives a total time for the algorithm of `O(|E| log |V|)`.

Prim's Algorithm

Unlike Krushkal's algorithm, Prim's algorithm builds `A` so that at each step `A` will be a tree rather than a forest.
To do this at each step we pick the least weight edge which connects some vertex not-spanned by `A` to `A`.
The book shows such an edge will be safe.
To find the minimum weight edge, we keep a priority queue of all the vertices not in `A`.
The priority, `v.key`, for a vertex `v` in the queue is least weight of any edge connecting `v` with `A`. If no such edge exists than it is `infty`.
Let `v.pi` be the parent of `v` in the tree. Rather than explicitly have an `A` we use this parent structure to get the tree when the algorithm terminates.

Here is the pseudo-code:

MST-PRIM(G, w, r) // r is a starting node to grow the tree from
01 for each u in G.V
02    u.key = infty
03    u.pi = NIL
04 r.key = 0
05 Q = MAKE-QUEUE(G.V)
06 while Q != 0
07     u = EXTRACT-MIN(Q)
08     for each v in G.adj[u]
09        if v in Q  and w(u, v) < v.key
10            v.pi = u
11            v.key = w(u,v) //call appropriate DECREASE-KEY

Prim Example

An example of Prim's algorithm in action

Black nodes indicate vertices in the tree so far.
Darkened edges indicate least edges that were selected for the tree.
So we go from (a) to (b) as the least weight edge going out of the current tree (consisting of just node `a` ) was the edge of weight 4 connected `a` to `b`.
If a heap is used for the priority queue, the algorithm takes `O(|V|)` time to build the queue, does `|V|` many EXTRACT-MIN's and makes `|E| = sum_(v in V) |Adj[v]|` calls to DECREASE-key.
This gives a total runtime of `O(|V|log|V| + |E| log |V|)`.

When is a problem feasibly solved?

As we saw on the first homework, if an algorithm takes exponential time in the input size then we won't be able to solve very large instances of it.
A common definition for a problem (a clearly stated input, output pair) having a feasible solution is that there is an algorithm which solves it in `O(n^k)`-time for some fixed `k`. I.e., polynomial time.
A problem represents an input output pair. i.e, a function from some domain to some range `F:X->Y`.
We have from time to time this semester used the solution of one problem together with some additional work to solve another problem. For example, we used depth-first search and some additional work on the finish times to implement topologically sort.
We can view this as a composition of algorithms. One nice property of polynomial time is that if we compose two polynomial time algorithms the result is polynomial time.

Optimization Problems versus Decision Problems

When talking about dynamic programming and greedy algorithms we were often interested in solving optimization questions.
For example, what is the maximum amount of money I can get by cutting a rod into pieces?
A decision problem is a problem with a YES/NO, TRUE/FALSE, 1/0 answer. For example, can I cut the rod and make more than $7?
The language `L` of a decision problem is the set of all input instances (encoded as strings) for which the answer to the problem is YES or TRUE or 1.
In general, a language is just a set of strings.
We say a language `L` is in P (polynomial time) if there is a polynomial time algorithm which answers `1` when an input is in the language and `0` if it is not.
When people study feasibility they typically look at decision problems rather than optimization ones.
If an optimization problem has a feasible solution then the corresponding decision problem will have a feasible solution too: For example, I can compute the optimal solution to cut rod and check if it pays more than $7.
Similarly, if one can feasibly solve the decision problem one can typically feasibly solve the optimization problem. For example, I can take a black box for decision cut rod and ask:Is there a solution of more than $1?, If yes, is there a solution of more than $2?, If yes, is there one of more than $4... I keep going till the first no answer. At which point I know the optimal value is between `2^i` and `2^(i+1)`. I then binary search between these values. The total time will be logarithm time in the optimal answer which will polynomial time in the number of characters needed to write down the optimal answer.

Not so Feasible Problems

Starting in the late 1960s, it was realized that subtle changes in a problem's definition can affect whether there is a feasible solution.
For example, finding the shortest path between two vertices in a graph can be done polynomial time. Finding the longest path between two vertices that never repeats a vertex seems to be hard. The best known algorithms are exponential time.
The corresponding decision problem is the question: Does there exists a simple path between `s` and `t` in `G` with edge weight greater than `k`?
Notice that if someone gives you a path in a graph `G`, then you can in polynomial time determine if its edge weight is less than or greater than `k` -- summing the edges in the path can be done polynomial time.
Given a string `y` let `|y|` denote its length. We say a decision problem `L` is in NP (nondeterministic polynomial time) if there is a two input polynomial time algorithm `A` and a constant `c` such that:
L = {x | there exists a certificate `y`, `|y| < |x|^c` such that `A(x,y) = 1`}
So we have just argues that
`L = {(G,s,t,k) |` there exists a simple path in `G` from `s` to `t` of edge weight greater than `k`}
is in NP.
From the definition it is not hard to see P is a sub-class of NP. A wide-variety of problems are known to be in NP, but not known to be in P.

Hardest Problems in NP

Recall using a polynomial time additional computation we could perform a topological sort given that we had already performed a depth first search.
In complexity jargon, we would say that we have reduced topological sort to depth first search.
For languages `L_1` and `L_2` we say `L_1` is polynomial time reducible to a `L_2` if there exists a polynomial time computable function from inputs to `L_1` to inputs of `L_2` such that:
`x in L_1` if and only if `f(x) in L_2`.
We say a language `L` is NP-complete if it is in NP and any other `L'` in NP is polynomial time reducible to it.

Towards showing there is an NP-Complete Problem

We next want to look at the first problem that was actually shown to be `NP`-complete: SAT.
This example comes from propositional logic. To introduce it we need to introduce the notion of a Boolean formula.
Given a set of variables `u_1`, ..., `u_n` whose values can be `0` (false) or `1` (true), a boolean formula is either just one of these variables or is built from these variables using AND (`^^`), OR (`vv`), or NOT (`neg`).
A truth assignment `nu:[1, .. n] -> {0,1}` gives a value to `0` or `1` to the set of variables. So `nu(1) = 1`, also written `nu(u_1) = 1`, says that variable `u_1` has the value `1`. Truth assignments for variables can be extended in the natural way to given a `0` or `1` value to any boolean formula.
A formula is said to be satisfiable if some assignment to its input variables makes the formula output `1`.
Otherwise, the formula is said to be unsatisfiable.
For example, `(u_1 ^^ u_2) vv u_3` is satisfiable. `(u_1 ^^ neg u_1)` is unsatisfiable.

Conjunctive Normal Form

A literal, `l_i`, is used to mean either a variable `u_i` or its negation `neg u_i`. We often write `neg u_i` as `bar u_i`.
A formula is said to be in conjunctive normal form, (CNF), if it is AND of ORs of variables or their negation.
For example, `(u_1 vv bar u_2 vv u_3) ^^ (u_2 vv bar u_3 vv u_4) ^^ (bar u_1 vv u_2 vv bar u_4)`
We often write CNF formulas like `^^_i(vv_j nu_(i_j))`
`vv_j nu_(i_j)` are called clauses.
A kCNF formula is a CNF formula in which all clauses have at most `k` literals.
We denote by SAT the language of all satisfiable CNF formulas, and by 3SAT the language of all satisfiable 3CNF formulas.

SAT is NP-complete

First notice, given a truth assignment, we can check in polynomial time if each clause in a CNF is satisfied or not. So both SAT and 3SAT are in NP. So to show SAT is NP-complete, we need to show every problem in NP can be reduced to it.
Suppose `L` is in NP. There there is some algorithm A(x,y) and a constant `c` such that
L = {x | there exists a certificate `y`, `|y| < |x|^c` such that `A(x,y) = 1`}.
We will not rigorously give a reduction from `L` to SAT but we will try to give some intuition on how the argument works.
At the start of the semester we said that we envisioned our algorithms as being written in some kind of machine code, running on a single core CPU, and having access to some amount of memory.
On an input `(x,y)`, `A` will only be able to access `|(x,y)|^k` memory addresses in our machine where `k` is some fixed constant depending on `c` and the polynomial bounding `A`'s run time.
Let `C_i` denote the content of our machines memory after our algorithm runs on (x,y) for `i` steps as well as the start of each of its registers and program counter. So `C_1` is the state of the machine when we start our computation on `(x,y)`.
The tableaux of our algorithm running on `(x,y)`, is the sequence: `C_1`, `C_2`, ... `C_(|(x,y)|^k)`.
Memory is a sequence of bits, a bit can in turn be represented as a boolean variable. So we can view `x` as a vector of boolean variables `vec x`, and `y` similarly as a vector `vec y`.
On inputs `x` of length `n`, we can make in polynomial time a boolean formula Init(`vec x`, `vec y`, `vec C_1`) which will be true iff the variables `C_1` correspond an initial configuration of our machine and algorithm on inputs `x`, `y`. We can make in polynomial time a formula Next(`vec C`, `vec C'`) which be true iff `vec C` and `vec C'` correspond to configurations of our machine which follow from each other by executing the instruction given by the program counter in `C`. Finally, we can make formula Last(`C`) which is true iff `C` is an ending configuration of `A` with output `1`.
On input `x` an instance of `L`, our reduction to SAT computes in polynomial time the formula:
Init(`vec x`, `vec y`, `vec C_1`) `^^` Next(`C_1`, `C_2`) `^^ cdots ^^` Next(`C_(|(x,y)|^k -1)`, `C_(|(x,y)|^k)`) `^^` Last(`C_(|(x,y)|^k)`).

3SAT is NP-complete

Once we have one NP-complete problem to show other problem's are NP-complete typically involves an easier reduction (such as one from SAT).
We have already show 3SAT is in NP.
Given an instance `Psi` of SAT, we first scan it to determine the largest index `i` or a variable `u_i` used in the formula. We then scan each clause in `Psi` looking at clauses. If the clause is less than or equal to length 3 we copy it to the output. Our reduction algortihm uses a counter `j` that starts at the value `i+1` where `i` is the largest index we just found. If a clause is `l_(i_1) vv .. vv l_(i_m)` where `m > 3`, we output the sequence of clauses `l_(i_1) vv l_(i_2) vv u_j`, `bar u_j vv l_(i_3) vv l_(i_4) vv u_(j+1)` ...`bar u_(j+w) vvl_(i_(m-1)) vv l_(i_m)` where `w` will be around `m/3`. The resulting clauses on the output is the value of our reduction on `Psi`. It is easy to check the original clauses will be satisfiable iff only the output clauses are satisfiable. Further the output clauses all have length less than or equal to 3.

Minimum Spanning Tree, NP-completeness

Outline