Designing Algorithms

Last week, when we designed the algorithm for InsertionSort, we followed an incremental approach: Having sorted `A[1..j-1]` we gave an algorithm to place `A[j]` in the correct position to get a sorted `A[1..j]`.
Today, we start by looking at another strategy for coming up with algorithms called divide-and-conquer.
We will use this to get a `Theta(nlog n)` algorithm for sorting called MERGE-SORT.
Our technique for building this algorithm will also be recursive: To solve the problem, our algorithm will call itself recursively one or more times on smaller sub-problems of the original problem.

Divide-and-Conquer

A divide-and-conquer algorithm splits a problem into several subproblems of smaller size, solves these subproblems recursively, and then combines these solutions to create a solution to the original problem.
This approach involves the following three steps at each level of recursion:

Divide
the problem into a number of subproblems that are smaller instances of the same problem

Conquer
the subproblems by solving them recursively. If the subproblem sizes are small enough, just solve the problem in a straightforward manner without additional recursion.

Combine
the solutions to the subproblems into the solution of the original problem

Divide-and-Conquer -- MERGE-SORT

The merge sort algorithm operates

Divide
Divide the `n`-element sequence to be sorted into two subsequences of `n/2` elements.

Conquer
Sort the two subsequences recursively using merge sort.

Combine
Merge the two sorted subsequences to produce the sorted answer.
The recursion "bottoms out" when the sequence to be sorted has length 1.
Merging the two subsequences is done by calling an auxiliary procedure `text{MERGE}(A, p, q, r)`, where `A` is an array and `p,q,r` are indices into the array such that `p le q lt r`. The procedure assume that `A[p..q]` and `A[q+1..r]` are sorted. It merges them to form a single sorted subarray that replaces the current subarray `A[p..r]`.
MERGE will be `Theta(n)` where `n=r-p+1` is the total number of elements to merge.

Pseudo-code for MERGE

The basic idea is that we image having two piles of items. We look at the top item in each pile and move the smaller to an output pile, continuing until all the items have been placed in the output pile.

MERGE(A, p, q, r)
 1 n1 = q - p + 1
 2 n2 = r - q
 3 let L[1..n1+1] and R[1..n2+1] be new arrays
 4 for i = 1 to n1
 5     L[i] = A[p + i - 1] //copy to first subarray to L
 6 for j = 1 to n2
 7     R[j] = A[q +j] //copy second subarray to R
 8 L[n1 + 1] = infinity // a value bigger than any value in the array 
 9 R[n2 + 1] = infinity // acts as a "sentinel" value so will never
                        // go off end of array
10 i = 1
11 j = 1
12 for k = p to r 
13     if L[i] <= R[j]
14         A[k] = L[i]
15         i = i + 1
16     else 
           A[k] = R[j]
17         j= j + 1

MERGE Example

The above shows MERGE(A, 9, 12, 16).
Step (a) represents after we have copied the two subarrays to `L` and `R` respectively
(b) shows after merging the first element back to `A`.
(c) and (d) shows after merging the second and thrid elements back respectively.

Quiz (Sec 5)

Which of the following statements is true?

To show a loop in an algorithm has a loop invariant one needs to establish initialization, maintenance, and termination conditions for the invariant.
The random access machine model we discussed last week was based on a quad-core Intel i7 chip.
The best-case and worst-case run-times of insertion sort on an input of size `n` are the same.

Quiz (Sec 6)

Which of the following statements is true?

The input size to a computational problem is always the number of bits it takes to write down an algorithm for the problem.
We argued last week that in the average case setting, the run time of insertion sort was a linear function of the input size.
The loop invariant we showed for insertion sort was:

At the start of each iteration of the for loop of lines 1-8, `A[1, .. j-1]` consists of the elements originally in `A[1 .. j-1]`, but in sorted order.

A Loop Invariant for MERGE

Claim. MERGE satisfies the following loop invariant:

At the start of each iteration of the for loop of lines 12-17, the subarray `A[p..k-1]` contains the k - p smallest elements of `L[1..n1 + 1]` and `R[1..n2 + 1]` in sorted order. Moreover, `L[i]` and `R[j]` are the smallest elements of their arrays that have not been copied back to `A`.

Proof. We need to show the three properties of loop invariants:

Initialization: Prior to the first iteration of the loop, we have `k=p`, so `A[p..k-1]` is empty. The empty array contains the `k-p = 0` smallest elements of `L` and `R`. Since `i=j=1`, both `L[i]` and `R[j]` are the smallest elements of their arrays that have not been copied back into `A`.
Maintenance: Suppose `L[i] le R[j]`. Then `L[i]` is the smallest element not yet copied back to `A`. Because `A[p..k-1]` contains the `k-p` smallest elements, after line 14 copies `L[i]` into `A[k]`, the subarray `A[p..k]` will contain the `k-p +1` smallest elements. Incrementing `k` and `i` reestablishes the loop invariant for the next iteration. If instead `L[i] > R[j]`, then lines 16-17 perform the appropriate action to maintain the loop invariant.
Termination: When MERGE is done, `k=r+1`. By the loop invariant, the subarray `A[p..k-1] = A[p..r]` contains the `k-p = r - p + 1` smallest elements of `L[1..n1+1]` and `R[1..n2+1]`, in sorted order. The arrays `L` and `R` contain `n1 + n2 +2 = r -p + 3` elements. All but the two largest have not been copied back into A because these two largest elements are the sentinels (infinity).

Runtime of MERGE

Let `n= r - p + 1` be the number of elements to merge.
Observe lines 1-3 and 8-11 of MERGE take constant time, the for loops of lines 4-7 take `Theta(n1 +n2) = Theta(n)` time
The for loop of lines 12-17 involves executing `n` times lines which each take constant time, so is `Theta(n)`.
Adding these all together gives an overall `Theta(n)` running time.

MERGE-SORT

Given our MERGE subroutine we are now in a position to present the MERGE-SORT algorithm:

MERGE-SORT(A,p,r)
1 if p < r
2    q = floor((p+r)/2)
3    MERGE-SORT(A, p, q)
4    MERGE-SORT(A,q+1,r)
5    MERGE(A, p, q, r)

floor is the function which round down to the next lower integer.

Analyzing Divide-and-conquer Algorithms

When an algorithm makes a recursive call to itself, we can often describe its running time by a recurrence equation or recurrence, which describes the overall run time of a problem of size `n` in terms of the running time on smaller inputs.
We can then use mathematical tools to solve the recurrence and provide bounds on the performance of the algorithm.
Let `T(n)` be the running time on a problem of size `n`.
If the problem size `n leq c` for some constant `c`, we assume the problem falls into the straightforward solution case and so take time `Theta(1)`.
Suppose that the division of the problem yields `a` subproblems each of which is `1/b` the size of the original.
It takes `T(n/b)` time to solve one subproblem of size `n/b`, and so it takes `aT(n/b)` time to solve `a` of them.
Suppose it takes `D(n)` time to divide the problem into subproblems and `C(n)` time to combine the solutions to the subproblems. Then the total time to solve the problem is:
`T(n)={(Theta(1),if n le c),(a T(n/b) + D(n) + C(n),text{otherwise}):}`

Analysis of Merge Sort

Divide: The divide step just computes the middle of the subarray which takes constant time, so `D(n) = Theta(1)`.
Conquer: We recursively solve two subproblems, each of size `n/2` which contributes `2T(n/2)` to the running time.
Combine: This involves MERGE. So `C(n) = Theta(n)`

Together we have:
`T(n)={(Theta(1),if n =1 ),(2 T(n/2) + Theta(n),text{n } gt 1):}`
using what `Theta(n)` means we have:
`T(n)={(Theta(1),if n =1 ),(2 T(n/2) + cn,text{n } gt 1):}`
We can use this to build a recursion tree and derive a total run time of `cn log n`.

Recursion Tree for MERGE-SORT

We build tree (d) above by expanding the recurrence of the last slide successively in (a), (b), (c), etc. until we get to the `n=1` case of MERGE-SORT.
This gives a tree with `log n` levels. If we sum the work for the MERGE calls across a whole level above we get `cn` for each level.
So the total run time is `c n log n + cn`.

Mergesort, Recurrences

Outline