Outline

• Sorting on PRAMs
• In-Class Exercise

Introduction

• On Wednesday, we started talking about parallel random access machines (PRAMs).
• In this parallel model, we have a collection of processors each using the same instruction set and clock but able to run in parallel and able to read and write a global memory.
• We considered different kind of conflict resolution for accessing the global memory: Exclusive Read/Exclusive Write (EREW), Concurrent Read/Exlusive Write(CREW), and Concurrent Read/Concurrent Write (CRCW).
• We also considered a few different complexity classes related to PRAMs:
  • NC - (Nick's Class) languages recognized on a PRAM using polylog time on polynomially many processors.
  • RNC - languages recognized by PRAMs with access to random number generators. I.e., such that if `x in L` the PRAM algorithm accepts at least 50% of the time, if it's not, the algorithm always rejects.
  • ZNC = RNC `cap` co-RNC.
• We said we can also create analogous function classes for each of these and that we'd abuse notation and call these NC, RNC, and ZNC as well.
• We begin today by looking at a ZNC algorithm for sorting.

Sorting on a PRAM

We now work towards a ZNC algorithm for sorting which run in `O(log n)` time doing a total of `O(n log n)` operations on all processors (Reischuk 1985). Let `P_i` denote the `i`th processor.

We begin by considering a PRAM variant of Quicksort:

1. If `n=1` stop.
2. Otherwise, pick a splitter uniformly at random from the `n` input elements
3. Each processor determines whether its element is bigger or smaller than the splitter.
4. Let `j` denote the rank of the splitter. If `j` is not in `[n/4, (3n)/4]` the step is declared a failure and we go back to step 1. Otherwise, we move the splitter to processor `P_j`. Each element that is smaller than the splitter is moved to a distinct processor is `P_i` such that `i lt j`. Each element that is larger than the splitter is moved to a distinct processor `P_k` where `k gt j`.
5. We sort recursively the data in the processors `P_1` through `P_(j-1)` and the data in `P_(j+1)` through `P_(n)`.

Notice this algorithm uses randomness, but it has zero error -- we can detect if we are in a bad case in which case we rerun -- it is in this sense it is a ZNC algorithm for sorting.

Analysis

• The third step above is O(1) time.
• To do step 4, we assume we have a global input array A and a global output arrays `B`, `C` and `S`. Each processor `P_i` sets a bit `b_i` in `B` in one of the global registers to `0` if its element is greater than the splitter and to `1` otherwise.
• For all `i`, let `S_i=Sigma_(t le i)b_t`. Using a `log`-depth tree of processors, all the `S_i`'s can be computed in parallel in `O(log n)` steps into `S`. This allows us to map into the output array those elements which are smaller than `j`. Namely, each processor in parallel checks, if `b_i` is 1 (where `i` is its global id), if so, it maps the element in `A_i` to `C_(S_i)`. Using a similar setting of bits and computing sums, we can also handle all those element which should be mapped to locations larger than `j`. From this we can do stage 4 in `O(log n)` steps.
• The expected number of time before step 1-4 succeeds is `1/(1/2) = 2` since half the `j`'s are in the interval `[n/4, (3n)/4]`. So the total runtime will be `O(log^2 n)`.
• Remark: To output the bits `b_i` above takes only constant time. We could imagine rather than use `n` processors to output the bits we use `n/(log n)` processors, each outputing `log n` many bits in `O(log n)` time. Similarly, we can adjust the computation of the `S_i` to only need `n/(log n)` processors. We thus save using some processors at the expense of making the constants in the `O` larger.

In-Class Exercise

Suppose we want to sort the items {3, -1, 5, 6, 2} and have 5 processors. Show the computation steps involved for each processor for each step in the QuickSort algorithm.

Post your solutions to the Feb 21 In-Class Exercise Thread.

Using More Splitters

Suppose we have `n` processors and `n` elements. Suppose the first `r` processors have values in sorted order.

• We can use these `r` elements as splitters to sort the remaining `n-r` elements.
• The goal is to insert the `n-r` unsorted elements among the splitters in the following sense:
  • Each processor should end up with a distinct input element.
  • Let `s_j` denote the `j`th largest splitter and `i_(s_j)` denote the index of the processor containing `s_j` after the insertion. Then for all `k lt i_(s_j)`, processor `P_k` contains an element that is smaller than `s_j` and for all `k gt i_(s_j)`, `P_k` contains an element that is larger than `s_j`.
• We hope we can do this in `O(log n)` time.

A Good Choice for `r`, a Complete Algorithm (called BoxSort)

1. If so, we could pick `n^(1/2)` elements at random and then using all `n` processors sort them in `O(log n)` steps.
   • We can imagine having an array `R` giving the indices of the randomly selected elements.
   • After the sorting the array `R` has its indices rearranged so they give the selected elements in ascending order.
   • To do the sorting we imagine for each `i` of the `sqrt(n)` many element of `R`, using `sqrt(n)-1` processors to compare it with the other `sqrt(n)-1` elements, then using `O(log n)` time to sum the values of these comparisons to determine the number of elements smaller than `i`.
2. Then using these sorted elements insert the remaining elements among them in `O(log n)` steps:
   • Notice, using binary search, a processor `i` can determine between which two splitters pointed to by `R`, `A_i` should go in `O(log n)` time.
   • So it can output a bit `b_i` saying whether location `i` is less than its splitter's location as pointed to by the index in `R` or not.
   • We can then compute sums `S_i` of these bits in parallel as with the QuickSort case and move in the same fashion.
3. Treat the remaining elements that are inserted between splitter as subproblems, recur on each subproblem whose size exceed `log n`, otherwise, use LogSort:
   • Compare each element in parallel with its neighbor first to its left, swap if necessary; then to its right, swap if necessary, do this O(log n) times.

Intuition on Splitting

• Doing splitting should take `O(log mbox((the size of box we need to split)))`.
• In a perfect world, at each level the expected size of the box goes down by a square root, so we get the sum
  `O(log n + log n^(1/2) + log n^(1/4) +...) = O(log n +1/2log n +1/4logn +...) = `
  `O((log n) cdot (1+ 1/2+1/4+ ...)) = O(log n)`
• We will argue even in a non perfect world that with high probability the sum of the log of the sizes of the boxes along any path is `O(log n)`, so the runtime will be `O(log n)`.

Analysis of Splitting

• To see the intuition of the last slide is true, partition the interval `[1,n]` into sub-intervals `I_0, I_1, ...` We will then bound the probability that a box whose size is in `I_k` has a child whose size is also in `I_k`.
• Fix `gamma` and `d` such that `1/2 lt gamma lt 1` and `1 lt d lt 1/gamma`. For positive integers `k`, let `tau_k=d^k`, `rho_k= n^(gamma^k)`. Define `I_k=[rho_(k+1), rho_k]`.
• `n = (log n)^{(log_(log n)2)log n }`. So if `gamma^k lt 1/ (log n log_(log n)2)`, then `rho_k= n^( gamma^k) lt log n`. This will happen for some `k lt c log log n`.
• So we will only be interested in `O(log log n)` many intervals `I_k`.
• For a box `B` in the tree, we let `alpha(B)=k` if `|B|` is in `I_k`.
• In terms of our notation, the time to split Box `B` is `O(log rho_(alpha(B)))` .
• For a root-leaf path, `P = (B_1, ..., B_t)`, the runtime is given by `sum_(j=1)^t log rho_(alpha(B_j))`.
• The total runtime of the algorithm will be O of this plus log n (to sort the leaves).
• Define the event `E_P` to be that the sequence `alpha(B_1), ..., alpha(B_t)` does not contain the value `k` more than `tau_k` times for `1 le k le c log log n`.
• If `E_P` holds then the number of PRAM steps on path `P` will be:
  `O(log n + sum_(k=1)^infty log tau_k gamma^k log n))`

The End of Sorting

• Since `tau_k=d^k` and `d cdot gamma lt 1`, this sums to `O(log n)`.
• So it suffices to show `E_P` happens with high probability.
  Lemma. There is a constant `b gt 1` such that `E_P` holds with probability `1- exp(-log^b n)`.
  Proof The proof of this is given as a sequence of exercises in the book which we omit. It makes use of Exercise 12.6 that follows from Chernoff bounds. (Chernoff Bounds: If `X` is the sum of independent random variables which outputs either 0 and 1, the latter with probability `p`, then for a `0 le theta le 1`, `Pr{X ge (1+theta)pn} lt e^(-(theta^2 p n)/3)`).
• From this we can conclude:
  Theorem. There is a constant `b gt 1` such that probability at least `1-exp(-log^b n)`, the algorithm BoxSort terminates in `O(log n)` steps.
• So although in a bad case it might take longer, with high probability this is a `O(log n)` time algorithm.