Outline

• Race Conditions
• Quiz
• Chess Example
• Matrix Multiplication

Introduction

• We have been talking about multithreaded algorithms.
• These were algorithms which run on a Dynamic Multithreading concurrency platform, a concurrency platform with a load balancing scheduler and supported nested parallelism (spawning) and parallel loops (parallel-for).
• We discussed the performance measures for our algorithms of work, `T_1`, total amount of computation needed to be done or the amount of time on one processor; and span, `T_(infty)`, the amount of time if we had as many processors as desired.
• We gave the work and span laws as lower bounds on how long a parallel algorithm takes to run on p processors, `T_P`.
• We also proved Brent's Theorem which gave an upper bound.
• We analyzed a parallel version of our Fibonacci algorithm using span-and-sync, and computed that its parallelism, `T_1/T_(infty)`, was exponential.
• Finally, we looked at matrix-vector multiplication, gave a parallel algorithm for it using parallel for loops, and showed in its parallelism was `Theta(n)`.
• Today, we continue to talk about multithreaded algorithms, starting by looking at race conditions.

Race Conditions

• A multithreaded algorithm is deterministic if it always does the same thing on the same input, no matter how the instructions are scheduled on the multicore computer; otherwise, it is said to be nondeterministic.
• A determinancy race occurs when two logically parallel instructions access the same memory location and at least one of the instructions performs a write.

  RACE-EXAMPLE
  1 x = 0
  2 parallel for i = 1 to 2
  3     x = x + 1
  4 print x
  

  After line 1 two parallel strands try to increment x in line 3, and print x in line 4.
• This might print 1 or it might print 2.
• If both strands read x before either increments it, then 1 will be output.
• If one strand increments x before the other reads it then the output will be 2.

Preventing Race Conditions

• There are many different ways used to try to prevent race conditions: exclusion locks, synchronization blocks, etc.
• To keep things simple, we shall insist that strands that operate in parallel are independent: they have no determinancy races among them.
• So in a parallel for construct, all the iterations should be independent.
• Between a spawn and the corresponding sync, the code of the spawned child should be independent of the code of the parent, including code executed by additional spawned or called children.

Another Race Example

• Here is another example of code which has a race condition:

  MAT-VEC-WRONG(A, x)
  1 n = A.rows
  2 let y be a new vector of length n
  3 parallel for i = 1 to n
  4    y[i] = 0
  5 parallel for i = 1 to n
  6    parallel for j = 1 to n
  7        y[i] = y[i] + a[i][j] * x[j]
  8 return y
  

• This code is like our code from last week, except the second inner for loop has been parallelized, giving it a span of `Theta(log n)` rather than `Theta(n)`.
• Unfortunately, the updates in line 7 are not all independent and so we have a determinancy race.
• For the homework you will modify this code to still get a span of `Theta(log n)` without a race condition.

Quiz

Which of the following statements is true?

1. For parallel algorithms, work is the longest time to execute strands along any path in a computation dag.
2. Brent's Theorem is the upper bound: `T_P le T_1/P`.
3. The running time `T_P` of any multithreaded computation scheduled by a greedy scheduler on an ideal parallel computer with P processors is within a factor of 2 of optimal.

Chess Program Socrates

• The chess program Socrates was programmed using the Cilk programming language and competed in matches in late 1990s, early 2000s.
• It was developed on a 32 processor system but ran in matches on a 512 processor system.
• During development the programmers came up with an optimization that reduced `T_(32) = 65` to `T_(32)' = 40` seconds on one of the testing benchmarks.
• The original code was eventually favored over the optimization because it was determined the original would be faster on 512 processors.
• Here is the analysis:
  • The original code had `T_1 =2048 sec` and `T_(infty) = 1 sec`.
  • Suppose Brent's Theorem was exactly true rather than an inequality. Then `T_P = T_1/P + T_(infty)` and the run time on 32 processors would be
    `T_(32) = 2048/32 + 1 = 65`.
  • The optimization had `T_1' =1024 sec` and `T_(infty)' = 8 sec`. Using the same approximation we get
    `T_(32)' = 1024/32 + 8 = 40`.
  • For 512 processors, though, `T_(512) = 2048/512 + 1 = 5` and `T_(512)' = 1024/512 + 8 = 10`
• From this, we see that work and span together provide a better means of extrapolating performance than just measuring running times.

Multithreaded Matrix Multiplication

• If we parallelize the standard matrix multiply algorithm, we get the following:

  P-SQUARE-MATRIX-MULTIPLY(A, B)
  1 n = A.rows
  2 let C be a new n x n matrix
  3 parallel for i = 1 to n
  4     parallel for j = 1 to n
  5         c[i][j] = 0
  6         for k = 1 to n
  7             c[i][j] = c[i][j] + a[i][k] * b[k][j]
  8 return C
  

• The serialization of this has three nested for loops up to n and so `T_1(n) = Theta(n^3)`.
• The span is `T_(infty)(n) = Theta(n)` because we follow a `Theta(log n)` path through the tree of recursion for the outer parallel for, followed by another path of `Theta(log n)` path through the tree of recursion for the inner parallel for and then we serial execute the innermost for `n` times.
• So we have `Theta(log n) + Theta(log n) + Theta(n) = Theta(n)`.
• This gives a parallelism of `(Theta(n^3))/(Theta(n)) = Theta(n^2)`.

Multithreaded Matrix Multiplication - Take 2

• There are several ways we can improve the parallel runtime of matrix multiplication.
• One could start with a faster single processor algorithm such as Strassen's algorithm -- the book shows the analysis of such an algorithm.
• Or we can try to improve the use of parallelism in the existing algorithm
• We take this approach and give a first pass at reducing `T_(infty)(n)` to `Theta(log^2 n)`. This can actually be improved to `Theta(log n)`.
• To start, let's assume `n` is a power of 2. We can split the matrices `A` and `B` into four quadrants `A_(11)`, `A_(12)`, `A_(21)`, `A_(22)` and `B_(11)`, `B_(12)`, `B_(21)`, `B_(22)` each with `n/2 times n/2` entries. We can then multiply these quadrants to get `C` using matrix multiply for `2 times 2` matrices:
  `A = [[A_(11),A_(12)],[A_(21),A_(22)]]`, `B = [[B_(11),B_(12)],[B_(21),B_(22)]]`, `C = [[C_(11),C_(12)],[C_(21),C_(22)]]`
  where `C = A cdot B` is given by:
  `[[C_(11),C_(12)],[C_(21),C_(22)]] = [[A_(11),A_(12)],[A_(21),A_(22)]][[B_(11),B_(12)],[B_(21),B_(22)]]`
  `= [[A_(11)B_(11) + A_(12)B_(21),A_(11)B_(12) + A_(12)B_(22)],[A_(21)B_(11) + A_(22)B_(21),A_(21)B_(12) + A_(22)B_(22)]]`
• We use the above to give a parallel divide-and-conquer algorithm for matrix multiply.

Divide-and-Conquer Matrix Multiply with Parallelism

P-MATRIX-MULTIPLY-RECURSIVE(C, A, B)
1 n = A.rows
2 if n == 1
3     c[1][1] = a[1][1] * b[1][1]
4 else let T be a new n x n matrix
5     partition A, B , C , and T into n/2 x n/2 submatrices 
          A_(11), A_(12), A_(21), A_(22); B_(11), B_(12), B_(21), B_(22);
          C_(11), C_(12), C_(21), C_(22); and T_(11), T_(12), T_(21), T_(22) respectively
6     spawn P-MATRIX-MULTIPLY-RECURSIVE(C_(11), A_(11), B_(11))
7     spawn P-MATRIX-MULTIPLY-RECURSIVE(C_(12), A_(11), B_(12))
8     spawn P-MATRIX-MULTIPLY-RECURSIVE(C_(21), A_(21), B_(11))
9     spawn P-MATRIX-MULTIPLY-RECURSIVE(C_(22), A_(21), B_(12))
10    spawn P-MATRIX-MULTIPLY-RECURSIVE(T_(11), A_(12), B_(21))
11    spawn P-MATRIX-MULTIPLY-RECURSIVE(T_(12), A_(12), B_(22))
12    spawn P-MATRIX-MULTIPLY-RECURSIVE(T_(21), A_(22), B_(21))
13    P-MATRIX-MULTIPLY-RECURSIVE(T_(22), A_(22), B_(22))
14    sync
15    parallel for i = 1 to n
16        parallel for j = 1 to n
17            c[i][j] = c[i][j] + t[i][j]

• The idea of the above code is that the `T_(ij)`'s represent the second product in the sum used to compute the `C_(ij)`'s
• We do each of the eight subproducts in parallel then add pairs of products to get the final result.

Analysis of P-MATRIX-MULTIPLY-RECURSIVE(C, A, B)

• We perform eight recursive multiplications of `n/2 times n/2` matrices and finish up with `Theta(n^2)` work from adding two `n times n` matrices.
• Hence, we have the recurrence `T_1(n) = 8T_1(n/2) + Theta(n^2)`. By case 1 of the Master Theorem, this gives, `T_1(n) = Theta(n^3)`.
• For `T_(infty)(n)` the span for partitioning is `Theta(1)`. Since all the subcalls are to matrices of the same size, the max span over the subcalls is just the span of any one of them, so `T_(infty)(n/2)`. Finally, the nested parallel for loops at the `n` have span `Theta(log n)`.
• So `T_(infty)(n) = T_(infty)(n/2) + Theta(log n)`. This gives `T_(infty)(n) = Theta(log^2 n)`.
• Hence, the parallelism is `(T_1(n))/(T_(infty)(n)) = Theta((n^3)/(log^2 n))` as claimed.