Outline

• Basics of Dynamic Multithreading
• In-class Exercise
• Model of Multithreaded Execution
• Performance Measures

Introduction

• On Monday, we finished our first pass of talking abut Probabilistic Analysis and Randomized Algorithms.
• We then introduced the topic of multithreaded algorithms.
• These were algorithms suited to be run on chips with more than one independent processing cores (multiprocessors) but where these cores have a shared memory.
• We assumed a software abstraction called a thread which corresponds to one "virtual processor" sharing the common memory.
• We assumed a static threading situation where the number of these threads does not change during the computation.
• We defined a concurrency platform to be a layer of software that coordinates, schedules, and manages the parallel-computing resources.
• Lastly, we considered the Dynamic Multithreading concurrency platform which had a load balancing scheduler and supported nested parallelism (spawning) and parallel loops (parallel-for).
• We start today by considering a simple Dynamic Multithreading algorithm.

Going from Serial Algorithm to Parallel Algorithm

• Recall the Fibonacci numbers (Fibonnaci, Liber Abaci, 1202) are defined by the recurrence:
  `F_0 = 0`,
  `F_1 = 1`,
  `F_i = F_(i-1) + F_(i-2)` for `i ge 2`.
• A closed form solution to find the `F_n` can be found using generating functions (DeMoivre, 1730), but let's first consider a simple serial algorithm to compute `F_n`:

  Fib(n)
  1 if n ≤ 1
  2     return n
  3 else 
      x = Fib(n - 1)
  4   y = Fib(n - 2)   
  5   return x + y
  

• In a moment we will show how to parallelize this, but first let's analyze it.
• The runtime of this algorithm because of the call to subproblems satisfies the recurrence: `T(n) = T(n-1) + T(n-2) + Theta(1)`.
• We can bound this as `T(n) = Theta(F_n)`. To see the upper bound, let's use the substitution method.
• Assume that `T(n') le a F_(n') - b` for all `n' lt n` where `a > 1` and `b >0` are constants. Substituting into our recurrence we obtain:
  `T(n) le (a F_(n-1) - b) + (a F_(n-2) - b) + Theta(1)`
  `= a(F_(n-1) + F_(n-2)) - b - (b - Theta(1))`
  `le a F_n - b`
  for suitable large `a` and `b`.

Method of Generating Functions

• Let `g(x) = sum_(i=0)^(infty) F_i x^i`. We call this power series the generating function for `F_i`.
• Consider the expression `g(x) - xg(x) - x^2g(x)`. For `i ge 2`, the coefficients will be `F_i - F_(i-1) - F_(i-2)` = 0.
• So we have:
  `(1 - x -x^2)g(x) = g(x) - xg(x) - x^2g(x) = F_0 + (F_1 - F_0)x + 0x^2 + 0x^3 + cdots = x`
• This implies `g(x) = x/(1-x-x^2)`.
• Taylor expanding `x/(1-x-x^2)` and matching the coefficients with the coefficients of `g(x)` (i.e., `F_i`) gives:
  `F_i = 1/sqrt(5)[((1+sqrt(5))/2)^i - ((1 - sqrt(5))/2)^i]`
• So `T(n)` of the last slide is `Theta(F_n) = Theta(phi^n)` where `phi = (1+sqrt(5))/2` and this is exponential in `n`.
• The number `phi` is known as the the Golden Ratio and it comes up all over the place in nature and art.

Dynamic Multithreaded Fibonacci

• We augment our pseudocode to indicate parallelism by adding the concurrency keywords spawn and sync.

  P-Fib(n)
  1 if n ≤ 1
  2     return n
  3 else 
      x = spawn P-Fib(n - 1)
  4   y = P-Fib(n - 2)
  5   sync 
  6   return x + y
  

• Other than these new keywords the code is identical to before. We define serialization of a multithreaded algorithm to be the serial algorithm that results from deleting all occurrences of spawn, sync, and parallel (this keyword doesn't appear above).
• When spawn precedes a procedure call the procedure issuing the call, the parent, may continue running in parallel to a spawned subroutine called a child.
• In this case, the spawned child computes P-Fib(n - 1) and the parent goes on computing P-Fib(n - 2).
• Spawn does not force parallel execution, it just indicates that it may occur.
• The concurrency keywords say where the logical parallelism of the computation occurs. At runtime, it is up to the scheduler to determine which subroutines actually run in parallel.
• The return value of a spawned child cannot be used until after the procedure executes a sync statement as in line 5.
• The sync keyword indicates that the procedure must wait for all its spawned children to complete before proceeding to the statement after the sync.
• If a procedure with spawned children ends without any sync statement, we assume there was an implicit sync statement at the last line of the procedure.

In-Class Exercise

• Write pseudo-code for the mergesort (Merge sort was invented by von Neumann in 1945).
• Mark it up with spawn and sync instructions at any place you think you could execute code independently on different processors that share memory.
• Post your solutions to the Feb 7 In-Class Exercise Thread.

Model of Multithreaded Execution

• We can think of a multithreaded computation as a directed acyclic graph (dag) `G= (V,E)`.
• The vertices represent instructions and the edges represent dependencies between instructions.
• An edge `(u,v) in E` means that instruction `u` must execute before instruction `v`.
• We group chains of instructions that don't contain spawn, sync, return from spawn, or parallel, into strands representing one or more instructions.
• A strand with two successors must be followed by a spawn statement.
• A strand with multiple predecessors must be preceded by a sync statement.
• If `G` has a directed path from distinct strand `u` to distinct strand `v`, we say that the two strands are (logically) in series. Otherwise, the distinct strands `u` and `v` are (logically) in parallel.

More on Multithreaded Execution

The image of the last slide shows the computation of P-Fib(4).

• Each circle represents one strand. Blacks circles are either base cases or the part of procedure up to the spawn of P-Fib(n-1).
• Shaded circles represent the part of the procedure that calls P-Fib(n-2) up till the sync of line 5. Each group of strands belonging to the same procedure is surrounded by a rounded rectangle.
• Horizontal edges represent continuation edges, that is, an edge for execution within the same procedure
• The rectangle is lightly shaded for spawned procedures and darkly shaded for called one. A downward edge to a spawned procedure is called a spawn edge. A downward edge to a called procedure is called a call edge. The upward edges representing the return from a spawn or call are called return edges.
• A computation starts from an initial strand, in this case, the black circle in P-Fib(4) and ends with a single final strand, in this case the white vertex in P-Fib(4).

Performance Measures

• We use the two metrics work and span to measure the theoretical efficiency of a multithreaded algorithm.
• The work of a multithreaded computation is the total time to execute the entire computation on one processor. I.e., the sum of the times taken by each of the strands.
• If we assume each strand executes in unit time, the work is just the numbers of vertices in the dag.
• The span is the longest time to execute the strands along any path in the dag.
• If we assume each strand executes in unit time, the span equals the number of vertices on a longest or critical path in the dag.
• For our graph of two slides back, if each node takes the same amount of time, then the work is 17 and the span is 8.

Runtime of a Multithreaded Computation

• The run time of a multithreaded computation depends not only on the work and the span, but also on how many processors are available and how the scheduler allocates strands to processors.
• We denote the running time of an algorithm on P processors by `T_P`.
• So work is `T_1`, the run time on one processor.
• The span is the running time if we could assign each strand its own processor. We denote it by `T_(infty)`.
• Using work and span we can get two lower bounds on `T_P`:
  • Since an ideal parallel processor with `P` processors can at most do `P` units of work in one time step we have the work law:
    `T_P ge T_1/P`
  • A P-processor ideal parallel computer cannot run any faster than a ideal machine with an unlimited number of processors, so we have the span law:
    `T_P ge T_(infty)`.
• The speedup of a computation on `P` processors is defined to be the ratio `T_1/T_P`. By the work law the speedup is always less than `P`.
• If `T_1/T_P = Theta(P)` we say the computation exhibits linear speedup, if `T_1/T_P = P`, we have perfect linear speedup.

Parallelism

• The ratio `T_1/T_(infty)` of the work to the span gives the parallelism of the multithreaded computation.
• It can be viewed as the average amount of work that can be performed in parallel along each step of the critical path.
• It upper bounds, the maximum possible speedup that can be achieved on any number of processors.
• It also provides a limit on the possibility of attaining perfect linear speedup.
• To see the last statement suppose `P ge T_1/T_(infty)`. The span law implies `T_1/T_P le T_1/T_(infty) le P`. If `P >``> T_1/T_(infty)` then ` T_1/T_P lt lt P`, so the speedup is much less than the number of processors.
• As an example, consider P-Fib(4) where we assume each strand takes unit time. Then `T_1 = 17` is the time, and the span is `T_(infty) = 8`. The parallelism is `T_1/T_(infty) = 17/8 =2.125`. So achieving much more than double speedup is impossible, no matter how many processors we use.
• We define the (parallel) slackness of a multithreaded computation on an ideal P processor computer to be the ration `(T_1/T_(infty))/P = T_1/(P T_(infty))`.
• It represents the factor by which the parallelism of the computation exceeds the number of processors in the machine.
• If the slackness is less than 1 we cannot hope to achieve perfect linear speedup as `T_1/(P T_(infty)) lt 1` and the span law imply `T_1/T_P le T_1/T_(infty) lt P`.