Introduction

Last Wednesday, we finished our first pass of talking about Probabilistic Analysis and Randomized Algorithms.
We then introduced the topic of multithreaded algorithms.
These were algorithms suited to be run on chips with more than one independent processing cores (multiprocessors) but where these cores have a shared memory.
We assumed a software abstraction called a thread which corresponds to one "virtual processor" sharing the common memory.
We assumed a static threading situation where the number of these threads does not change during the computation.
We defined a concurrency platform to be a layer of software that coordinates, schedules, and manages the parallel-computing resources.
Lastly, we considered the Dynamic Multithreading concurrency platform which had a load balancing scheduler and supported nested parallelism (spawning) and parallel loops (parallel-for).
We start today by considering a simple Dynamic Multithreading algorithm.

Going from Serial Algorithm to Parallel Algorithm

Recall the Fibonacci numbers (Fibonnaci, Liber Abaci, 1202) are defined by the recurrence:
`F_0 = 0`,
`F_1 = 1`,
`F_i = F_(i-1) + F_(i-2)` for `i ge 2`.
A closed form solution to find the `F_n` can be found using generating functions (DeMoivre, 1730), but let's first consider a simple serial algorithm to compute `F_n`:
```
Fib(n)
1 if n ≤ 1
2     return n
3 else 
    x = Fib(n - 1)
4   y = Fib(n - 2)   
5   return x + y
```
In a moment we will show how to parallelize this, but first let's analyze it.
The runtime of this algorithm, because of the calls to subproblems, satisfies the recurrence: `T(n) = T(n-1) + T(n-2) + Theta(1)`.
We can bound this as `T(n) = Theta(F_n)`. To see the upper bound, let's use the substitution method.
Assume that `T(n') le a F_(n') - b` for all `n' lt n` where `a > 1` and `b >0` are constants. Substituting into our recurrence we obtain:
`T(n) le (a F_(n-1) - b) + (a F_(n-2) - b) + Theta(1)`
`= a(F_(n-1) + F_(n-2)) - b - (b - Theta(1))`
`le a F_n - b`
for suitable large `a` and `b`.

Method of Generating Functions

Let `g(x) = sum_(i=0)^(infty) F_i x^i`. We call this power series the generating function for `F_i`.
Consider the expression `g(x) - xg(x) - x^2g(x)`. For `i ge 2`, the coefficients will be `F_i - F_(i-1) - F_(i-2)` = 0.
So we have:
`(1 - x -x^2)g(x) = g(x) - xg(x) - x^2g(x) = F_0 + (F_1 - F_0)x + 0x^2 + 0x^3 + cdots = x`
This implies `g(x) = x/(1-x-x^2)`.
Taylor expanding `x/(1-x-x^2)` and matching the coefficients with the coefficients of `g(x)` (i.e., `F_i`) gives:
`F_i = 1/sqrt(5)[((1+sqrt(5))/2)^i - ((1 - sqrt(5))/2)^i]`
So `T(n)` of the last slide is `Theta(F_n) = Theta(phi^n)` where `phi = (1+sqrt(5))/2` and this is exponential in `n`.
The number `phi` is known as the the Golden Ratio and it comes up all over the place in nature and art.

Dynamic Multithreaded Fibonacci

We augment our pseudocode to indicate parallelism by adding the concurrency keywords spawn and sync.

P-Fib(n)
1 if n ≤ 1
2     return n
3 else 
    x = spawn P-Fib(n - 1)
4   y = P-Fib(n - 2)
5   sync 
6   return x + y

Other than these new keywords the code is identical to before. We define serialization of a multithreaded algorithm to be the serial algorithm that results from deleting all occurrences of spawn, sync, and parallel (this keyword doesn't appear above).
When spawn precedes a procedure call the procedure issuing the call, the parent, may continue running in parallel to a spawned subroutine, called a child.
In this case, the spawned child computes P-Fib(n - 1) and the parent goes on computing P-Fib(n - 2).
Spawn does not force parallel execution, it just indicates that it may occur.
The concurrency keywords say where the logical parallelism of the computation occurs. At runtime, it is up to the scheduler to determine which subroutines actually run in parallel.
The return value of a spawned child cannot be used until after the procedure executes a sync statement as in line 5.
The sync keyword indicates that the procedure must wait for all its spawned children to complete before proceeding to the statement after the sync.
If a procedure with spawned children ends without any sync statement, we assume there was an implicit sync statement at the last line of the procedure.

Quiz

Which of the following statements is true?

Assuming people are equally like to be born any day of the year, if there were more than 50 people in a room, there would be more than a 50% chance two shared a birthday.
In `n` fair coin tosses you would expect to see streaks of length `Omega((log n)^2)`.
Our analysis of the online hiring problem relied heavily on facts about the geometric distribution.

Model of Multithreaded Execution

A multithreaded DAG representing the computation of P-Fib(n)

We can think of a multithreaded computation as a directed acyclic graph (dag) `G= (V,E)`.
The vertices represent instructions and the edges represent dependencies between instructions.
An edge `(u,v) in E` means that instruction `u` must execute before instruction `v`.
We group chains of instructions that don't contain spawn, sync, return from spawn, or parallel, into strands representing one or more instructions.
A strand with two successors must be followed by a spawn statement.
A strand with multiple predecessors must be preceded by a sync statement.
If `G` has a directed path from distinct strand `u` to distinct strand `v`, we say that the two strands are (logically) in series. Otherwise, the distinct strands `u` and `v` are (logically) in parallel.

More on Multithreaded Execution

The image of the last slide shows the computation of P-Fib(4).

Each circle represents one strand. Blacks circles are either base cases or the part of procedure up to the spawn of P-Fib(n-1).
Shaded circles represent the part of the procedure that calls P-Fib(n-2) up till the sync of line 5. Each group of strands belonging to the same procedure is surrounded by a rounded rectangle.
Horizontal edges represent continuation edges, that is, an edge for execution within the same procedure
The rectangle is lightly shaded for spawned procedures and darkly shaded for called one. A downward edge to a spawned procedure is called a spawn edge. A downward edge to a called procedure is called a call edge. The upward edges representing the return from a spawn or call are called return edges.
A computation starts from an initial strand, in this case, the black circle in P-Fib(4) and ends with a single final strand, in this case the white vertex in P-Fib(4).

Performance Measures

We use the two metrics work and span to measure the theoretical efficiency of a multithreaded algorithm.
The work of a multithreaded computation is the total time to execute the entire computation on one processor. I.e., the sum of the times taken by each of the strands.
If we assume each strand executes in unit time, the work is just the numbers of vertices in the dag.
The span is the longest time to execute the strands along any path in the dag.
If we assume each strand executes in unit time, the span equals the number of vertices on a longest or critical path in the dag.
For our graph of two slides back, if each node takes the same amount of time, then the work is 17 and the span is 8.

Runtime of a Multithreaded Computation

The run time of a multithreaded computation depends not only on the work and the span, but also on how many processors are available and how the scheduler allocates strands to processors.
We denote the running time of an algorithm on P processors by `T_P`.
So work is `T_1`, the run time on one processor.
The span is the running time if we could assign each strand its own processor. We denote it by `T_(infty)`.
Using work and span we can get two lower bounds on `T_P`:
- Since an ideal parallel processor with `P` processors can at most do `P` units of work in one time step we have the work law:
  `T_P ge T_1/P`
- A P-processor ideal parallel computer cannot run any faster than a ideal machine with an unlimited number of processors, so we have the span law:
  `T_P ge T_(infty)`.
The speedup of a computation on `P` processors is defined to be the ratio `T_1/T_P`. By the work law the speedup is always less than `P`.
If `T_1/T_P = Theta(P)` we say the computation exhibits linear speedup, if `T_1/T_P = P`, we have perfect linear speedup.

Parallelism

The ratio `T_1/T_(infty)` of the work to the span gives the parallelism of the multithreaded computation.
It can be viewed as the average amount of work that can be performed in parallel along each step of the critical path.
It upper bounds, the maximum possible speedup that can be achieved on any number of processors.
It also provides a limit on the possibility of attaining perfect linear speedup.
To see the last statement suppose `P ge T_1/T_(infty)`. The span law implies `T_1/T_P le T_1/T_(infty) le P`. If `P >``> T_1/T_(infty)` then ` T_1/T_P lt lt P`, so the speedup is much less than the number of processors.
As an example, consider P-Fib(4) where we assume each strand takes unit time. Then `T_1 = 17` is the time, and the span is `T_(infty) = 8`. The parallelism is `T_1/T_(infty) = 17/8 =2.125`. So achieving much more than double speedup is impossible, no matter how many processors we use.
We define the (parallel) slackness of a multithreaded computation on an ideal P processor computer to be the ration `(T_1/T_(infty))/P = T_1/(P T_(infty))`.
It represents the factor by which the parallelism of the computation exceeds the number of processors in the machine.
If the slackness is less than 1 we cannot hope to achieve perfect linear speedup as `T_1/(P T_(infty)) lt 1` and the span law imply `T_1/T_P le T_1/T_(infty) lt P`.

Scheduling

A multithreaded scheduler must schedule the computation with no advance knowledge of when strands will be spawned or when they will complete. That is, the scheduler must operate online.
A centralized scheduler (as opposed to a distributed one) knows the global state of the computation at any time.
A greedy scheduler is a centralized scheduler that assign as many strands to processors as possible in each step.
A complete step is a step in which at least P strands are ready to execute during a given step; otherwise, the step is incomplete.
In a complete step a greedy scheduler assigns any `P` of the ready strands to processors.

Multithreaded Algorithms

Outline