Introduction

On Monday, we were looking at the hiring problem which was to work out the total cost of interviewing `n` candidates, where if we find candidate `i+1` is better than candidate `i` we fire `i` and hire `i+1`.
We assumed there was a cost `c_i` to interview a candidate and a code `c_h\>\>c_i` to hire and fire and candidates.
We worked out that the expected total cost would be `O(c_i n + c_h ln n)` provided that the order of arriving candidates was chosen uniformly at random.
In order to ensure this situation, we said we would generate a random permutation and apply it to the candidate order, then start the interview process.
We gave one algorithm for generating random permutations by first randomly picking priorities from 1 to `n^3` and then sorting by these priorities.
We start today by looking at a second technique for generating a random permutation.

Randomly Permuting Arrays (Method 2)

Randomize-In-Place(A) 
1. n := length[A]
2. for i :=1 to n
3.     swap(A[i], A[Random(i,n)])

Analysis of Method 2

Lemma. Just prior to the ith iteration of the for loop, for each possible `(i-1)`-permutation, the subarray `A[1,i-1]` contains this permutation with probability `((n-i+1)!)/(n!)`

Proof. By induction on `i`.

Base case: When `i=1`, `A[1..0]` is the empty array. It is supposed to contain a given 0-permutation with probability `((n-1+1)!)/(n!) = (n!)/(n!)=1`. As a `0`-permutation has no elements and there is only one of them this is true.

Induction step: Assume just before the `i`th iteration, each `(i-1)`-permutation occurs in the `A[1..i-1]` with probability `((n-i+1)!)/(n!)`. A particular, `i`-permutation `langle x_1,...,x_(i-1), x_i rangle` consists of an `(i-1)`-permutation followed by `x_i`. By the induction hypothesis, the probability of the `i`-permutation is thus
`[((n-i+1)!)/(n!)] cdot Pr{A[i]= x_i| A[1..i-1] = langle x_1,...,x_(i-1) rangle}`.
The second factor is `1/(n-i+1)` since by line 3 of Randomize-in-Place, `x_i` is choosen at random from `A[i..n]`. So the probability of the `i`-permutation is
`((n-i+1)!)/(n!) cdot 1/(n-i+1) = ((n-i)!)/(n!)` as desired.

More Analysis of Method 2

Theorem. Randomize-In-Place produces a uniformly chosen random permutation.

Proof. The program could generate any `n`-permutation. Further it terminates just before its `(n+1)`st iteration and thus by the lemma generates a given random `n`-permutation with probability:
`((n - (n+1) +1)!)/(n!) = (0!)/(n!) = 1/(n!)`
as desired.

In-Class Exercise

What are the runtimes as a function of `n` (use O-notation) for Randomize-in-Place and Permute-By-Sorting?

Work out your answer. If you haven't already signed up for an account, sign up for an account on the discussion board (you need to create an account on https://www.yioop.com/, them add the discussion group).

Post your solutions to the Jan 31 In-Class Exercise Thread.

The Birthday Problem

How many people must there be in a room before there is a `50%` chance that two were born on the same day of the year?

Let `b_1, b_2,.., b_k` be the birthday's for the people in the room. These are independent random events.
Let `n` be the number of days in a year. (i.e., `n=365`). Let `r` be the `r`th day of year.
Assume `Pr{birthday(b_i) = r}= 1/n.`
`Pr{mbox(birthday)(b_i) = r ^^ mbox(birthday)(b_j)=r} = Pr{mbox(birthday)(b_i) = r} cdot Pr{mbox(birthday)(b_j) = r} = 1/(n^2)`.
So the probability any `b_i` and `b_j` have any day `r` as their birthday is:
`Pr{mbox(birthday)(b_i) = mbox(birthday)(b_j)} =`
`= sum_(r=1)^n Pr{mbox(birthday)(b_i) = r} cdot Pr{mbox(birthday)(b_j) = r}`
`= sum_(r=1)^n 1/(n^2)`
`= 1/n`.

Number of Expected Shared Birthdays

Rather than look at the odds that two people share a birthday, we might ask the related question: For a given class size how many shared birthdays would we expect to see?
Let \begin{eqnarray*} X_{ij} & = & I\{\mbox{person } i \mbox{ and person } j \mbox{ have the same birthday}\}\\ & = & \left\{ \begin{array}{ll} 1 & \text{if person $i$ and person $j$ have the same birthday}\\ 0 & \text{otherwise.} \end{array} \right. \end{eqnarray*}
`E[X_(ij)] = Pr\{\text{person i and person j have the same birthday} \} = 1/n`.
Let `X = sum_(i=1)^(k)sum_(j=i+1)^k X_(ij)`.
So
`E[X] = E[\sum_{i=1}^k\sum_{j=i+1}^k X_{ij}]`
`= \sum_{i=1}^k\sum_{j=i+1}^kE[X_{ij}]`
`= ((k),(2))1/n = \frac{k(k-1)}{2n}`
using linearity of expectations.
When `n=365`, this is greater than 1 if `k=28`

Balls and Bins

Consider the process of tossing identical balls into `b` bins.
Tosses will be assumed to be independent and any ball is equally likely to end up in any bin.
The odds of ending up in a particular bin are thus `1/b`.
Successfully, landing in a given bin can be viewed as a so-called Bernoulli trial. A Bernoulli trial is an experiment with two possible outcomes.
Balls and bins arguments are useful for modeling a variety of processes in computer science such as hashing.

Ball and Bin Questions

We can ask a variety of questions about the ball tossing process. As a first question, we ask:

How many balls fall in a given bin?

Since tossing a ball into a given bin is a Bernoulli trial, the odds of `k` successes given `n` tosses follows the binomial distribution `b(k;n, x_(i\n\-\b\i\n))`, where `x_(i\n\-\b\i\n)= Pr{\text{ball in bin}} = 1/b`; `x_(n\ot\-i\n\-b\i\n) = Pr{\text(ball not in bin)}= 1-1/b`.
The sample space for the binomial distribution is `{1,2,...}` the possible values for `k`.
The probability of a given event can be determined by considering
`(x_(i\n\-\b\i\n) + x_(n\ot\-i\n\-b\i\n))^n=(1/b +(1-1/b))^n = 1`.
Expanding the left hand side and right hand side gives terms
`((n),(k)) cdot x_(i\n\-\b\i\n)^k cdot x_(n\ot\-i\n\-b\i\n)^(n-k) = ((n),(k)) 1/(b^k) cdot (1 - 1/b)^(n-k)`
which represent the probability of getting `k` balls after `n` tosses.
Let `X` be a random variable whose value is the number of tosses to fall in the bin. Then
`E[X]= sum_(k=1)^n k ((n),(k)) 1/(b^k) cdot (1 - 1/b)^(n-k)`
`= n(1/b) sum_(k=1)^n ((n - 1),(k -1)) 1/(b^(k-1)) cdot (1 - 1/b)^(n-k)`
`= (n/b) sum_(k=0)^(n -1) ((n - 1),(k))1/(b^k) cdot (1 - 1/b)^((n - 1) -k)` notice the sum is `(x_(i\n\-\b\i\n) + x_(n\ot\-i\n\-b\i\n))^(n -1) = 1` so
`=n/b`

Second Ball and Bin Question

How many balls must one toss on average, until a given bin contains a ball?

Let `b` still be the number of bins.
Our sample space `{1,2, ...}` where an event `k` is supposed to indicate that the first time one got into the bin was the `k`th toss.
If `X` is the number of trials to succeed `Pr{X=k} = (1- 1/b)^(k-1) cdot 1/b`. This is a geometric distribution.
Before we work out `E[X]` let's recall the following fact about series:
`sum_(k=0)^(\infty) q^k = lim_(n->infty) sum_(k=0)^(n-1) q^k`
`= lim_(n->infty) (1 - q^n)/(1-q)` as `(1 - q)sum_(k=0)^(n-1) q^k = 1 - q^n`
`= 1/(1 - q)`
Given this, `E[X] = `
`= sum_(k=1)^(infty) k (1 - 1/b)^(k-1) cdot (1/b)`
`= 1/b sum_(k=1)^(infty) k (1 - 1/b)^(k-1)` let `q = 1- 1/b`.
`= (1/b) d/(dq) (sum_(k=0)^(infty) q^k) = (1/b) d/(dq) (1/(1 - q)) = (1/b) 1/((1 - q)^2)`
`= (1/b) 1/((1 - (1 - 1/b))^2) = b^2/b = b`.

Third Ball and Bin Question

How many balls must one toss on average, until every bin contains at least one ball?

Call a toss into an empty bin a hit.
We want to know the expected number `n` of tosses to get `b` hits.
Can partition the `n` tosses into stages where the `i`th stage is the number of tosses after the `(i-1)`st hit until the `i`th hit. The first stage is thus just the first toss.
The probability of there being a hit for a given toss in stage `i` is `(b - i+1)/b`
Let `n_i` denote the number of tosses in stage `i`. So the number of tosses to get `b` hits is
`n = sum_(i=1)^b n_i`.
Each random variable `n_i` follows a geometric distribution with probability of success `(b-i+1)/b`. Using the same kind of calculation as the last slide `E[n_i]=b/(b-i+1)`.
Using linearity of expectation:
`E[n] = E[sum_(i=1)^b n_i] = sum_(i=1)^b E[n_i] = sum_(i=1)^b b/(b-i+1) = b sum_(i=1)^b 1/i` using the integral bound for 1/i
`= b (ln b + O(1))`.
This problem is also called the coupon collector's problem.

Random Permutations, the Birthday Problem, Ball and Bins Arguments

Outline