Introduction

So far this semester we have been looking at various techniques to do probabilistic analysis of algorithms.
For example, under the assumption that candidates arrived in a uniformly at random fashion, we used indicator random variables and linearity of expectation to show that the expected number of candidates we would hire and fire in the Hiring problem was `ln n +O(1)`.
To ensure the random ordering of candidates we gave two algorithms for generating random permutations: sort-by-priorities and random-swaps-to-the-right.
To analyze the first, we argued sorting by distinct priorities produces a random permutation and then that the odds of getting distinct priorities if they were chosen in the range `1` to `n^3` was high. Both arguments made use of conditional probabilities. The main trick in the latter was using the inequality `(1 - a)(1 -b) > 1 -a -b` for `a,b` positive to get a lower bound for an expression of the form `(1 - 1/n^k)^(n-1)`.
The analysis of the second random permutation algorithm was a proof by induction that the odds that it produced a particular `(i-1)`-permutation after `i` swaps had a particular form.
On Monday, we started considering two common arguments for probabilistic analysis: the birthday problem, and ball and bins arguments. Both of these often come up in analysing things such as hashing.
To compute the odds of two people sharing a birthday we actually bounded the probability of the event that no one shares a birthday. We used the inequality `1 +x le e^x` to bound a product that would be hard to evaluate in terms of e to a sum we could.
For ball and bin arguments, we answered two of three questions described in the book: How many balls fall in a given bin? (The expected value of the binomial distribution); How many balls must one toss on average, until a given bin contains a ball? (The expected value of the geometric distribution).
Today, we begin by analyzing a third ball in bins question: How many balls must one toss on average, until all bins contains a ball?
Next, we consider how likely a streak of a given length occurs when tossing coins.
Finally, we consider an online version of our hiring problem.

Third Ball and Bin Question

How many balls must one toss on average, until every bin contains at least one ball?

Call a toss into an empty bin a hit.
We want to know the expected number `n` of tosses to get `b` hits.
Can partition the `n` tosses into stages where the `i`th stage is the number of tosses after the `(i-1)`st hit until the `i`th hit. The first stage is thus just the first toss.
The probability of there being a hit for a given toss in stage `i` is `(b - i+1)/b`
Let `n_i` denote the number of tosses in stage `i`. So the number of tosses to get `b` hits is
`n = sum_(i=1)^b n_i`.
Each random variable `n_i` follows a geometric distribution with probability of success `(b-i+1)/b`. Using the same kind of calculation as the last day `E[n_i]=b/(b-i+1)`.
Using linearity of expectation:
`E[n] = E[sum_(i=1)^b n_i] = sum_(i=1)^b E[n_i] = sum_(i=1)^b b/(b-i+1) = b sum_(i=1)^b 1/i` using the integral bound for 1/i
`= b (ln b + O(1))`.
This problem is also called the coupon collector's problem.

Streaks

Suppose you flip a fair coin n times. What is the longest streak of consecutive heads you expect to see?

The answer is `Theta(log n)`. We will show it is `O(log n)` and give some intuitions on the lower bound -- the full argument is in the book.
What is interesting in the upper bound is we break the expected number into two sums and show how to bound each independently.

Upper Bound on Streaks

Since we are assuming the coin is fair, a coin flip is heads with probability `1/2`.
Let `A_(ik)` be the event that a streak of heads of length at least `k` begins with the `i`th coin flip.
Since coin flips are mutually independent, `Pr{A_(ik)} = 1/(2^k)`
For `k= 2 |~ log n ~|`,
`Pr{A_(i,2 |~ log n ~|)} = 1/2^(2 |~ log n ~|) le 1/(2^(2 log n )) = 1/n^2`.
There are at most `n- 2 |~ log n ~| + 1` positions such a streak can begin. So
`Pr{cup_(i=1)^(n - 2 |~ log n ~| +1) A_(i,2 |~ log n ~|)} le sum_(i=1)^(n - 2 |~ log n ~| +1) 1/(n^2) lt sum_(i=1)^n 1/(n^2) = 1/n` (*).
For example, plugging `n=1000` in the above and noting `2 log n le 20`, the odds of getting a streak of length `20` is less than `1/1000`.
Let `L_j` be the event that the longest streak has length exactly `j`. and let `L` be the length of the longest streak. Thus,
`E[L] = sum_(j = 0)^n j Pr{L_j}`.

Bounding `E[L]`

The probability that a streak of heads of length at least `2 |~ log n ~|` is ` sum_(j = 2 |~ log n ~|)^n Pr{L_j}`.
By (*) on the previous slide, `sum_(j = 2 |~ log n ~|)^n Pr{L_j} lt 1/n`.
Notice also that `sum_(j = 0)^(2 |~ log n ~|) Pr{L_j} le sum_(j = 0)^n Pr{L_j} = 1`.
Hence, we have
`E[L] = sum_(j = 0)^n j Pr{L_j}`
`= sum_(j = 0)^(2 |~ log n ~|) j Pr{L_j} + sum_(j = 2 |~ log n ~|)^n j Pr{L_j}`
`lt sum_(j = 0)^(2 |~ log n ~|) (2 |~ log n ~|) Pr{L_j} + sum_(j = 2 |~ log n ~|)^n n Pr{L_j}`
`= (2 |~ log n ~|) sum_(j = 0)^(2 |~ log n ~|)Pr{L_j} + n sum_(j = 2 |~ log n ~|)^n Pr{L_j}`
`lt (2 |~ log n ~|) cdot 1 + n cdot (1/n)`
`= O(log n)`. So we have proven the upper bound.

In-Class Exercise

`|~ log n ~|` heads is one particular result of performing `|~ log n ~|` many coin tosses.
How many distinct strings of head and tails of length `|~ log n ~|` are there?
Upper bound the number of coin tosses you would need to make to expect to see them all.
Post your answers to the Feb 6 In-Class Exercise Thread.

Using Indicator Variables for Lower Bound Intuition

Let `X_(ik) = I(A_(ik))`. i.e., `X_(ik)` indicates a streak of length at least `k` beginning with the `i`th location.
We can count the total number of such streaks with the random variable
`X = sum_(i=1)^(n - k + 1) X_(ik)`.
The expectation of this is:
`E[X] = E[sum_(i=1)^(n - k + 1) X_(ik)]`
`= sum_(i=1)^(n - k + 1) E[X_(ik)]`
`= sum_(i=1)^(n - k + 1) Pr{A_(ik)}`
`= sum_(i=1)^(n - k + 1) 1/(2^k)`
`= (n - k + 1)/(2^k)`.
If we plug in `1/2 log n` for `k`, we get `E[X] = (n - (1/2 log n) + 1)/(2^(1/2 log n)) = n^(1/2) - O(1)`.
So the expected number of streaks of length at least `1/2 log n` is an increasing function of `n` greater than 1, thus, giving a lower bound on `E[L]` as defined on the earlier slides.

The On-Line Hiring Problem

Suppose we don't want to interview everyone in the hiring problem to find the best candidate.
Suppose further we only want to hire once.

So we follow the following algorithm:

On-Line-Maximum(k,n)
1 bestscore = -infinity 
2 for i := 1 to k
3    if score(i) > bestscore
4        bestscore := score(i)
5 for i := k + 1 to n
6   if score(i) > bestscore then return i
7 return n

I.e., we interview the first `k` applicants and determine the best score, which we will denote bestscore. Then we start interviewing the remaining canididates and hire the first candidate that has a score higher than bestscore.
We want to determine the odds of getting the best candidate as a function of `k`.

Still More on the On-Line hiring Problem

`O_i` only depends on the relative order of the values in the positions `1` through `i - 1`.
`B_i` depends only on whether the value at position `i` is greater than all other positions.
This turns out to imply `B_i` and `O_i` are independent.
So `Pr{S_i} = Pr{B_i cap O_i} = Pr{B_i}Pr{O_i} = (1/n)cdot(k/(i-1)) = k/(n(i-1))`.
- `Pr{B_i} =1/n` since the best value is equally likely to be in any position
- `Pr{O_i} =k/(i-1)` as the maximum value in position `1.. i-1` is equally likely to be in any position. So the odds its in the first `k` of these `i-1` positions, is `k/(i-1)`.
Thus,
`Pr{S} = sum_(i = k+1)^n Pr{S_i} = sum_(i = k+1)^n k/(n(i-1)) = k/n sum_(i = k+1)^n 1/(i-1) = k/n sum_(i = k)^(n-1) 1/i`
Bounding the sum in terms of the integral for `1/i`. We get
`k/n(ln n - ln k) le Pr{S} le k/n (ln (n-1) - ln (k-1) )`
Differentiating with respect to ``k and setting to `0` one can show this is maximized when `k= n/e`.

Multithreaded Algorithms

In most undergrad algorithms classes all of the algorithms are serial algorithms. That is, they are suitable for running on a uniprocessor computer in which only one instruction executes at a time.
For our next topic, we are going to extend our algorithmic model in different ways to encompass parallel algorithms. These algorithms will be designed to run on a multiprocessor computer that permits multiple instructions to execute concurrently.
Most laptops today contain chip multiprocessors. Each such chip contains several cores each of which contains a full-fledged processor that can access a common memory.
This is a shared memory.
It is also possible to have models of parallel processing in which each processor has it own memory. This is called a distributed memory set-up.
To program a shared memory multiprocessor, it is common to use a software abstraction, static threading, that provides "virtual processors", or threads, sharing a common memory.
Since creating and destroying threads is often slow compared to a typical computation, we will assume that our threads "live" for the durations of our computations. Hence, the number of threads is "static".
As directly programming threads can be a pain to do correctly, various concurrency platforms have been proposed that provide a layer of software that coordinates, schedules, and manages the parallel-computing resources.

Dynamic Multithreaded Programming

Dynamic multithreading is an important particular type of concurrency platform.
It contains a scheduler which load balances the computation automatically, simplifying life for the programmer.
It supports nested parallelism and parallel loops.
Nested Parallelism means that a subroutine can by "spawned", allowing the caller to continue while the spawned subroutine is computing results.
Parallel loops are like for loops, except that the iterations of the loop can execute concurrently.

Streaks, Online Hiring Problem, Start Threads

Outline