Introduction

On Monday we introduced the quicksort algorithm.
Recall QUICKSORT(A, p, r) for `r - p + 1 > 0` sets `x=A[r]`, then partitions the array into three parts `A[p .. q-1]` which after partitioning will have values `le x`, `A[q] = x`, and `A[q+1 .. r]` which all have values bigger than `x`. We then quicksort `A[p .. q-1]` and `A[q+1 .. r]` to get a completely sorted array.
We argued that the worst case was the case of the already sorted list where the run-time would be `Theta(n^2)`.
We also argued that in the best case where the partition always splits the array in halves the run-time would be `O(n log n)`.
We start today by trying to give some intuitions why the best case is what tends to happen more often.

Balanced Splits

Suppose when running quicksort, the split were always the highly unbalanced, but at a fixed ratio, say nine elements on one side of `q` for every one on the other side.
If the splits were always 9-to-1, the recurrence would be
`T(n) = T((9n)/10) + T(n/10) +cn`.
Every level of the recursion tree has cost `c n` until the recursion reaches the boundary condition `log_10 n = Theta(log n)`. After that the levels are bounded by `c n` until on gets to the maximum recursion depth of `log_(10/9) n = Theta(log n)`.
So the total cost is `O(n log n)`.
In fact, splits of any fixed ratio such as 99-1 also give a `O(n log n)` run time by a similar analysis.

Intuitions on Quicksort

When run on a random input array the partitioning is highly unlikely to happen in the same way at every level.
We expect some of the splits to be reasonably balanced and some to be fairly unbalanced.
One of the book exercises is to show that about 80% of the time PARTITION produces a split that is more than 9 to 1 and that about 20 percent of the time it produces a split that is less balanced than 9 to 1.
In the average case, PARTITION produces a mix of "good" and "bad" splits and these splits would be distributed randomly in the recursion tree.
If the good and bad cuts happened on alternate levels one would get a situation like the (a) figure above.
The subarray of the children above are `0`, `(n-1)/2 -1` and `(n-1)/2` and so the combined cost of the two levels is `Theta(n) + Theta(n-1) = Theta(n)`.
The situation after the two levels is a balanced tree still to compute. So it behaves like a good split but with a slightly worse constant in the `Theta(n)`.
So this gives some intuition that the average case might be like the best case, but now let's try to make this more rigorous. To do this we first look at a quicksort variant.

Randomized Quicksort

In our analysis so far, we have assumed that all permutations of the input are equally likely.
This might not actually be the case, for instance, the we might quite often send the already sorted array to our sorter.
To make sure that the pivot value `x = A[r]` is actually random we can modify our partition function as follows:
```
RANDOMIZED-PARTITION(A, p, r)
1 i = RANDOM(p,r)
2 exchange A[r] with A[i]
3 return PARTITION(A,p,r)
```

Here PARTITION is the following procedure from Monday:

PARTITION(A, p, r)
1 x = A[r]
2 i = p - 1
3 for j = p to r - 1
4     if A[j] <= x
5         i = i + 1
6         exchange A[i] with A[j]
7 exchange A[i+1] with A[r]
8 return i + 1

Call the version of quicksort that uses this partition procedure RANDOMIZED-QUICKSORT(A, p, r).

Probability

We are going to show that RANDOMIZED-QUICKSORT has an expected running time of `O(n log n)`.
In order to appreciate our argument it helps to know a little probability.
We first by imagining we have a set `S` of elementary events called a sample space.
For example, for a coin we might have {HEADS, TAILS} as our sample space. For a six-sided die we might have `{1, 2, 3, 4, 5, 6}`. For sorting, `S` might be the possible inputs to our sorting procedure.
A probability distribution is a function from `A subseteq S` (`A` is called a sample or an event) into the unit interval `[0,1]` such that:
1. If `A_1`, ..., `A_i` ..., are disjoint events (i.e., disjoint sets) then `Pr(cup_i A_i) = sum_i Pr(A_i)`.
2. `Pr(S)=1`.
For example, if Pr is a probability distribution on HEADS and TAILS, then `Pr({HEADS, TAILS}) = Pr({HEADS}) + Pr({TAILS}) = 1`. One way to come with a Pr is to set it value on the elementary events say `Pr({HEADS}) = 0.2`, `Pr({TAILS}) = 0.8` so that the sum is 1 and use the properties above to determine its value `Pr(emptyset) = 0` and `Pr({HEADS, TAILS}) = 1`.
A random variable `X` is a function `X:S -> RR`. For example, we might bet $3 on whether a coin flip is heads or tails. So `X(HEADS) =3` (I win), `X(TAILS) = -3` (I lose).
Given a probability distribution `Pr`, the expected value of random variable `X`, `E(X)`, with respect to this distribution is defined as:
`E[X] = sum_(s in S) X(s)Pr(s)`.
So for the distribution `Pr` and random variable `X` above, `E[X] = 3 times 0.2 + (-3) times 0.8 = -1.8`.
One can verify from the definition that expectations are linear. So if `Y` and `Z` are random variables `E[X+Y] = E[Y] + E[Z]`.

Expected Run-time of RANDOMIZED-QUICKSORT

QUICKSORT and RANDOMIZED-QUICKSORT differ only in how the pivot is selected
So we will now be lazy (save typing) and say things in terms of QUICKSORT but assume that the pivots elements are selected randomly.
The running time of QUICKSORT is dominated by the time spent in the PARTITION procedure
Each time it is called we select a pivot and this element is never in any future recursive calls.
So PARTITION can be called at most `n` times on a subarray of size `n`.
A call to PARTITION take time `O(1)` plus the time to execute its for-loop in lines 3-6. Thus, we have argued for the following lemma.
Lemma. Let `X` be the number of comparisons performed in line 4 of PARTITION over the entire execution of QUICKSORT on an `n`-element array. Then the running time of QUICKSORT is `O(n + X)`.
In case you are wondering what lemma means, a lemma is a little math result that we prove along the way to a more major result that we would call a theorem.
So we want to compute an expected value for `X` to get an average run-time.

More Analysis

Let `z_1, z_2, ..., z_n` be the elements of `A` arranged so that `z_i` is the `i`th smallest element.
Let `Z_(ij) = {z_i, z_(i+1), ..., z_j}`.
When does the algorithm compare `z_i` and `z_j`?
Notice elements are compared at most once as an element is only compared against a pivot and a pivot is once used by one call of PARTITION.
Define `X_(ij)` to be the random variable which is 1 if `z_i` is compared to `z_j` for a particular input to quicksort and `0` otherwise.
Since each pair `i,j` is compared at most once, the total number of comparisons `X` performed by the algorithm is
`X = sum_(i=1)^(n-1)sum_(j=i+1)^nX_(ij)`
So the expected value of `X` is given by (using linearity of expectations)
`E[X] = E[sum_(i=1)^(n-1)sum_(j=i+1)^nX_(ij)] = sum_(i=1)^(n-1)sum_(j=i+1)^nE[X_(ij)]`
`= sum_(i=1)^(n-1)sum_(j=i+1)^n Pr{z_i \text( is compared to )z_j}`
So to compute this expectation we just need to compute `Pr{z_i \text( is compared to )z_j}`.

When are two items not compared?

Consider quick-sorting the numbers 1 to 10 and suppose the pivot was 7. Then the first call to PARTITION separates the numbers into the two sets {1,2,3,4,5,6} and {8,9,10}. In the process, 7 is compard to all other elements, but no number in the first set will ever be compared to any number in the second set.
I.e., once a pivot `x` is chosen so that `z_i < x < z_j`, we know `z_i` and `z_j` cannot be compared at any subsequent time.
If, on the other hand, `z_i` is chosen as a pivot before any other item in `Z_(ij)`, then `z_i` will be compared to each item in `Z_(ij)` except for itself. Similarly, if `z_j` is chosen as pivot before any element in `Z_(ij)`, then `z_j` will be compared to each item in `Z_(ij)` except for itself.
In the example above, `7 and 9` are compared because `7` is the first item from `Z_(7,9)` to be chosen as a pivot. In contrast, `2` and `9` will never be compared because the first pivot element chosen from `Z_(2,9)` was `7`.
Prior to the point at which an element from `Z_(ij)` is chosen as a pivot, the whole set `Z_(ij)` is together in the same partition. Therefore, any element in `Z_(ij)` is equally likely to be chosen as the first pivot. As `Z_(ij)` has `j-i+1` elements, the probability that any given element is the first one chosen is `1/(j - i + 1)`.

`Pr{z_i \text( is compared to )z_j}`

From the discussion of the last slide we have
`Pr{z_i \text( is compared to )z_j} = Pr{z_i \text( or )z_j \text( is first pivot chosen from )Z_(ij)}`
`= Pr{z_i \text( is first pivot chosen from )Z_(ij)} + Pr{z_j \text( is first pivot chosen from )Z_(ij)}`
`= 1/(j - i + 1) + 1/(j - i + 1)`
`= 2/(j - i + 1)`.

`E[X]` and Expected Run-time of QUICKSORT

Substituting the equation for `Pr{z_i \text( is compared to )z_j}` of the last slide into our earlier formula for `E[X]` gives:
`E[X] = sum_(i=1)^(n-1)sum_(j=i+1)^n 2/(j - i + 1)`
To evaluate the sum we can do a change of variable `k= j -i`
`E[X] = sum_(i=1)^(n-1)sum_(j=i+1)^n 2/(j - i + 1)`
`=sum_(i=1)^(n-1)sum_(k=1)^(n-i) 2/(k + 1)` ` < sum_(i=1)^(n-1)sum_(k=1)^(n) 2/k` ( now bound using `sum_(k=1)^(n) 2/k = 2 + sum_(k=2)^(n) 2/k < 2 + int_(k=1)^(n) 2/k dk`)
` = sum_(i=1)^(n-1) O(\log n)` ` = O(n log n)`.
Since by our lemma earlier the runtime of quicksort is bounded by `O(n + X)`, we get the expected run-time of quicksort is `O(n log n)`.

Sorting in Linear Time

We have now introduced several algorithms that can sort `n` numbers in `O(n log n)` time
Merge sort and heap sort achieve this in the worst case; quicksort achieves it on average.
Moreover, for each of these algorithms, we can produce a sequence of `n` input numbers that cause the algorithm to run in `Omega(n log n)`.
Each of these algorithms determines the sorted order of the output based only on comparisons between the input elements.
Such sorting algorithms are called comparison sorts.
It turns out comparison sort algorithms need `Omega(n log n)` to sort `n` items. Next week we will look at linear time, non-comparison sort algorithms.

Quicksort Analysis, Start Linear Time Search

Outline

Introduction

Balanced Splits

Intuitions on Quicksort

Randomized Quicksort

Probability

Expected Run-time of RANDOMIZED-QUICKSORT

More Analysis

When are two items not compared?

`Pr{z_i \text( is compared to )z_j}`

`E[X]` and Expected Run-time of QUICKSORT

Sorting in Linear Time