Outline

• Bucket Sort
• Selection Algorithms

Introduction

• On Monday, we started looking at sorting techniques which do not rely on comparisons between the elements in an array to determine the location of an element in the sorted array.
• We looked at counting sort and then used this to build a second sorting algorithm called radix sort, both of which were linear time.
• We start today by looking at a third linear time sorting algorithm known as bucket sort...

Bucket Sort

• Like counting sort, bucket sort will assume something about the inputs to be sorted.
• In the case of counting sort, we assumed the inputs were integers between 0, and `k`.
• In the case of bucket sort, we will assume the inputs are drawn uniformly chosen real numbers from the interval `[0, 1)`.
• For example, we might want to sort 0.1, .8, .33, .21, .7
• To sort these numbers, bucket sort divides `[0,1)` into `n`-equal-sized sub-intervals called buckets.
• It scans each element of the input and puts the element into the appropriate bucket.
• Then it insertion sorts each bucket to get the final results.

BUCKET-SORT Pseudo-code

• The pseudo code assumes the input is an array `A` with values `0 le A[i] < 1`.
• It makes use of an auxiliary array `B[0 .. n-1]` of linked list buckets and it assumes that there is a mechanism for maintaining such lists.

BUCKET-SORT(A)
1 n = A.length
2 let B[0..n-1] be a new array
3 for i = 0 to n - 1
4    make B[i] an empty list
5 for i = 1 to n
6    insert A[i] into list B[floor(nA[i])]
7 for i = 0 to n - 1
8    sort list B[i] with insertion sort
9 concatenate the lists B[0], B[1], ... , B[n - 1] together in order

Example

• In the above the input was (a).
• The number of buckets is the size of the array, in this case, 10. So we label the buckets 0 to 9.
• To determine which bucket `A[1] = 0.78` should go to, we multiply 0.78 by the number of buckets 10 to give 7.8 and then round down to give 7. So 0.78 should go into the 7th bucket.
• We continue through the whole array in this fashion.
• Once done we sort each of the resulting linked lists to get (b) above. We then output the buckets in order from least to greatest to get .12, .17, .21, .23, 26, .39, .68, .72, .78, .94.
• Notice rather than insertion sort, when we are adding an element to the linked-list, we could scan along the list till we find the place that the elment should go so that the list is in order and insert the element there. This might be slightly faster but would make our analysis harder.

Remarks on Correctness

• Consider two elements `A[i]` and `A[j]`.
• Assume without loss of generality that `A[i] le A[j]`.
• So `lfloor n A[i] rfloor le lfloor n A[j] rfloor` and so `A[i]` will either go into the same bucket as `A[j]` or one with lower index.
• If is goes into one with lower index it will be output earlier as we want. If it goes into one with the same index, then either the insertion operation (if using the last slide optimization) or lines 7-8 ensure that it will be output in the correct order.
• The optimization from the last slide is harder to analyze so we will analyze the procedure as it was presented in the book.

Expected Run-time

• Observe all lines except 7-8 take O(n) time. line 7-8 might involve `O(n)` calls to something that it quadratic in the longest list size.
• To analyze the cost of the calls to insertion sort, let `n_i` be the random variable denotes the number of elements placed in bucket `B[i]`.
• Recall we said that expectations of random variables have linearity properties. Given two random variable `X` and `Y` and any real `c`, from the definition of expectation one can show: (a) `E[X+Y] = E[X] + E[Y]` and (b) `E[c cdot X] = c cdot E[X]`.
• Since insertion sort is quadratic time, the running time of bucket sort is:
  `T(n) = Theta(n) + sum_(i=0)^(n-1)O(n_i^2)`.
• The expected run-time is therefore
  `E[T(n)] = E[Theta(n) + sum_(i=0)^(n-1)O(n_i^2)]`
  `= Theta(n) + sum_(i=0)^(n-1)E[O(n_i^2)]` (by linearity of expectation (a) above )
  `= Theta(n) + sum_(i=0)^(n-1)O(E[n_i^2])` (by linearity of expectation (b) above )
• We claim that `E[n_i^2] = 2 - 1/n` for `i=0,1,..., n-1`.

Towards showing `E[n_i^2] = 2 - 1/n`

• Since each value in `A` is equally likely to fall in any bucket, `E[n_i^2]` will be the same for each `i`.
• Let `X_(ij)` be the random variable which is `1` if `A[j]` fall in bucket `i` and `0` otherwise.
• Thus, `n_i = sum_(j=1)^n X_(ij)`.
• So we have
  `E[n_i^2] = E[(sum_(j=1)^n X_(ij))^2]`
  `= E[sum_(j=1)^n sum_(k=1)^n X_(ij) X_(ik)]`
  `= sum_(j=1)^n E[X_(ij)^2] + sum_(1 le j le n)^n sum_((1 le k le n,k ne j))^n E[X_(ij)X_(ik)]` (using linearity)
• We evaluate each sum separately. `X_(ij)` is 1 with probability `1/n` and `0` otherwise, so
  `E[X_(ij)^2] = 1^2cdot 1/n + 0^2(1 - 1/n) = 1/n`.
• When `k ne j` the variables `X_(ij)` and `X_(ik)` are independent, and hence
  `E[X_(ij)X_(ik)] = E[X_(ij)]E[X_(ik)] = 1/n cdot 1/n = 1/n^2`

`E[n_i^2] = 2 - 1/n`

• Substituting the values we just calculated we have
  `E[n_i^2] = sum_(j=1)^n E[X_(ij)^2] + sum_(1 le j le n)^n sum_((1 le k le n,k ne j))^n E[X_(ij)X_(ik)]`
  `= sum_(j=1)^n 1/n + sum_(1 le j le n)^n sum_((1 le k le n,k ne j))^n 1/n^2`
  `= n cdot 1/n + n(n-1) cdot 1/n^2`
  `= 1 + (n-1)/n`
  `= 2 - 1/n` as we claimed.

Expected Run-time of BUCKET-SORT

• Before we got the expected run-time of BUCKET-SORT was `E[T(n)] = Theta(n) + sum_(i=0)^(n-1)O(E[n_i^2])`
• And we just calculated that `E[n_i^2] = 2 - 1/n`
• So
  `E[T(n)] = Theta(n) + sum_(i=0)^(n-1)O(2 - 1/n) = Theta(n) + n cdot O(2- 1/n) = Theta(n)`.

Median and Order Statistics

• The `i`th order statistics of a set of `n` elements is the `i`th smallest element.
• For example, the minimum of a set is the first order statistic (`i=1`) and the maximum is the `n`th order statistic (`i=n`).
• A median of a set is `i=(n+1)/2`th order statistic when `n` is odd and `i=n/2` or `i=n/2 +1`th order statistic when `n` is even.
• In the even case, the former is called the lower median and the latter is the upper median.
• From now on we will use median to mean lower median.
• We next consider the selection problem:

  Input:

  A set A of `n` (distinct) numbers and an integer `i`, with `1 le i le n`.

  Output:

  The element `x in A` that is larger than exactly `i-1` other elements of `A`.

Naive Selection Algorithm

• One way to solve the selection problem is to sort the list using `O(n log n)` time and return the `i`th element in the output array.
• If we were looking for the minimum (maximum would be similar) we could use pseudo-code like:

  MINIMUM(A)
  1 min = A[1]
  2 for i = 2 to A.length
  3    if min > A[i]
  4        min = A[i]
  5 return min
  

• This is an `O(n)` time algorithm that makes `n-1` comparisons.
• Suppose we wanted to find both the minimum and the maximum simultaneously. We can obviously do it using `2n - 2` comparisons.
• We can actually do it using just `3lfloor n/2 rfloor` comparisons: We maintain the min and max seen so far. Rather than compare each element of the input against the current minimum and maximum, we process elements in pairs.
• We compare pairs of elements from the input with each other, then we compare the smaller with the min and the larger with the max. This costs 3 comparisons for every 2 elements.

Selection in Expected Linear Time

• It turns out that there is a variant of quicksort which can be used to solve the selection problem in expected linear time.
• The pseudo-code below returns the `i`th smallest element of the array `A[p..r]`

  RANDOMIZED-SELECT(A, p, r, i)
  1 if p == r
  2     return A[p]
  3 q = RANDOMIZED-PARTITION(A, p, r)
  4 k = q - p + 1
  5 if i == k // the pivot was the answer
  6      return A[q]
  7 elseif i < k
  8      return RANDOMIZED-SELECT(A, p, q - 1, i)
  9 else return RANDOMIZED-SELECT(A, q + 1, r, i - k)
  

• The analysis of this algorithm is similar to that of quicksort. The idea is we do a `Theta(n)` partition followed by a call to a problem which we expect is half the size. So the run-time will be `Theta(n) + Theta(n/2) + Theta(n/4) +... + Theta(1) = Theta(n)`.