Outline

• The Byzantine Agreement Problem
• Map Reduce

Byzantine Agreement Problem

• The problem was introduced by Pease, Shostak, and Lamport in 1980.
• It is similar to the Choice Coordination Problem.
• The original name of the problem was the Byzantine Generals Problem and it was supposed we have `n` Byzantine Generals that have to agree on a battle plan.
• In our set-up, we want to agree on one of two possible values (say, heads or tails `[0,1]`).
• We have `n` processors `t` of which may be faulty.
• We require that the decision reached by a protocol should have:
  1. All good processors finish with the same decision.
  2. If all the good processors begin with the same value `v`, then they should all finish with same value `v`.

More on the Set-up of the Byzantine Problem

• The set of faulty processors is fixed before the computation begins.
• The good processors do not know which processors are faulty.
• During a round, each processor may send one message to each other processor.
• Each processor receives a vote from each of the remaining processors before the following round begins.
• A processor is allowed to send different messages to different processors.
• Good processors will be assumed to follow our algorithm exactly.
• It is known any deterministic algorithm for this problem needs at least `t+1` rounds.
• We will present a randomized algorithm with `O(1)` expected runtime.

Some More Remarks before we Begin

• Our solution to the choice coordination problem was only for two processors choosing between two values.
• Neither of these processors was faulty.
• The algorithm we present today works for `n` processors choosing between two values, so is already more general, ignoring the allowance for faulty processors.
• The original choice coordination problem was for `n` processors to choose among `m` choices.
• Notice by repeating the Byzantine procedure `log_2 m` times we can have our processors agree on a first bit of the number between `1` to `m`, then a second bit of the number between `1` to `m`, etc.
• If each such single bit agreement can be done in constant time as a function of `n`, then agreeing on a number between 1 and `m` can be done in `O(log m)` time. Notice this does not depend on the number of processors.

Quiz

Which of the following statements is true?

1. Our parallel MIS algorithm always outputs the lexicographically first maximal independent set.
2. The odds that the Synch-CCP protocol doesn't stop in 3 rounds is at least 1/2.
3. Parallel MIS had an expected running time on EREW PRAM of `O(log^2 n )`.

Randomized Algorithm for Byzantine Agreement

• We will assume that at the start of each round a trusted third party flips a fair coin.
• Any of the processors have access to this coin.
• We will assume that the number of faulty processors is a number `t < n/8`.
• Each round a good processor sends the same vote to all the other processors.
• A faulty processor may send arbitrary or even inconsistent votes to each other processor.
• Let `L=((5n)/8)+1`, `H=((3n)/4)+1`, and `G=(7n)/8`.

What the ith Processor does (if it is good).

Input: A value for b[i], our current decision choice. 
Output: A decision d[i].
1. vote = b[i].
2  For each round, do
3.    Broadcast vote;
4.    Receive votes from all the other processors.
5.    Set maj = majority (0 or 1) value among the votes cast
6.    Set tally = the number of votes that maj received.
7.    If coin = heads then set threshold = L; else set threshold = H
8.    If tally >= threshold then set vote = maj; else vote = 0
9.    If tally >= G then set d[i] = maj permanently.

Analysis

• First, if all processors begin the round with the same vote, then 9 will apply and so this value will be the value eventually settled upon.
• Suppose the processors begin the round with different values for the vote.
• If two processors compute different values for maj in Step 5, since at most 1/8 of the processors are faulty, the tally does not exceed threshold regardless of whether L or H was chosen as threshold. So all good processors would set their votes to 0 and an agreement would be reached.
• We say a faulty processor foils a threshold `x in {L,H}` in a round if, by sending different messages to the good processors, they cause tally to exceed `x` for at least one good processor, and to be no more than `x` for at least one good processor.
• Since the difference between the two possible thresholds is at least `t`, the faulty processor can foil at most one threshold in a round.
• Since the threshold is chosen with equal probability from `{L,H}`, it is foiled with probability at most `1/2`.
• Thus, the expected number of rounds before we have an unfoiled threshold is at most `2`. If the threshold is not foiled then all good processors compute the same vote in Step 8.
• Since 7/8th of the processors are good, this ensure that by at most the next round 9 applies.

Parallel and Distributed Algorithms so Far

• So far we have looked two models for parallel processing: Dynamic Multithreading concurrency platforms and PRAMs.
• We have just considered two distributed algorithms one for Async-CCP and one for Byzantine Agreement.
• These can all be viewed as "software-oriented" in some sense: PRAM algorithms being machine code to be run on machines with many processors.
• There are also "hardware" oriented models such as circuits or sorting networks. These might consist of networks which are built-out of gates.
• In the circuit setting the gates might compute boolean functions of their inputs. I.e., they might compute AND, OR, or NOT gates. In the sorting setting, the gates might compare two inputs and output the input values in sorted order.
• The network or circuit is then a directed acyclic graph. Inputs are variables attaches to nodes of in-degree 0, outputs are nodes of out-degree 0.
• The "time/level" at which a gate can operate is when all of its inputs have acquired a value. This would be some constant times the length of the longest path to a gate.
• Gates of the same level can operate in parallel.
• Although both circuits and sorting networks are important models of parallel computation we won't talk about them more this semester.
• Instead, we are going to introduce a more recent famous parallel distributed model of computation.

Map Reduce

• Map Reduce was first described in a published paper by Dean and Ghemawat in 2004, but had been used at Google for some time before this.
• At a high-level, the basic set up of map reduce is a multi-round computation.
• In a given round, jobs are mapped to various machines.
• These machines then perform some action on the mapped data.
• The machines then perform a "shuffle" phase in which data might be grouped by a key value and then sent to a reducer machine.
• The reducer does some operation on the grouped values to produce a result for the next round, or maybe a final result.
• Karloff, Suri, and Vassilvitskii in 2010 have formalized this model more precisely and shown that a class of PRAM algorithms which can be simulated by Map Reduce and have also shown to what extent MapReduce jobs can be simulated by PRAMs.
• Next day, I would like to go over their proof. For the rest of today, I would like to go over some examples.

Example of How Map Reduce Might be Useful

• Suppose we run a online video streaming service, where we allow users to upload videos of various lengths.
• Videos might not get uploaded in a format which is convenient for streaming, so your site recodes the videos.
• Recoding is expensive and reasonably slow, so you don't want to do it on a single machine.
• Even if you map each uploaded video to a collection of machines for recoding, if someone uploads a long video, you don't want to hog the resources of a single machine to only recode that video while other machines go idle.
• To solve these problems, when videos are uploaded, you first segment them into equal size chunks. So you have a mapper that maps input files to machines for segmentation. Each segment produced has a key consisting of what video and where in that video it belonged to.
• Then these segments are then mapped to machines in a load balanced for recoding.
• In a shuffle phase (key, (recoded video segments)) are sent and grouped together on a reducer machine.
• The reducer then joins these segments to produce a total recoded video.

The Basic Framework

• MapReduce was inspired by the map and reduce functions found in functional programming languages such as Lisp.
• The map function takes as its argument a function `f` and a list of elements `l = langle l_1, ..., l_n rangle`. It return a new list
  `map(f,l) = langle f(l_1), ..., f(l_n) rangle`.
• The reduce function takes a function `g` and a list of elements `l = langle l_1, ..., l_n rangle`. It returns a new element `l'` such that
  `l' = \r\e\d\u\c\e(g,l) = g(l_1,g(l_2, g(l_3, ... )))`.
• From a high-level view, a MapReduce programs creates a sequence of key/value pairs, performs some computations on them, and outputs another sequence of key/value pairs.
• Keys and values are often strings, but may be of any data type.

Distinct Phases of a MapReduce Job

• MapPhase, key/value pairs are read from the input and the map function is applied to each of them individually. The function is of the general form:
  `map: langle k, v rangle |-> langle langle k_1, v_1 rangle, langle k_2, v_2 rangle, ...,rangle`
• Shuffle Phase, the pairs produced during the map phase are sorted by their key, and all values for the same key are grouped together.
• Reduce Phase, the reduce function is applied to each key and its values.
  `\r\e\d\u\c\e: langle k, langle v_1, v_2, ... rangle rangle |-> langle k, langle v_1', v_2', ... rangle rangle`
  That is, for each key the reduce function processes the list of associated values and outputs another list of values. The output values and their number may not be the same as the input values.

Example MapReduce Job for Counting

• In the job below, we want to produce the number of each type of token in some collection of documents. I.e., for each word in these documents the number of occurrences of that word.
• Our map sends the different documents to different machines.
• `k` is a key for the document, `v` is its text. On a machine, we then split the document into tokens (words) and output pairs `(t,1)`.
• During a shuffle step, all pairs with the same value of `t` would get sent to the same machine and grouped together in a list.
• Then the reduce procedure below would then count the `1`'s in this list.

Parallelizing Map Reduce

• Both Map and Reduce jobs can be parallelized, that is, executed on many machines.
• Map jobs are typically broken into smaller pieces called map shards each of 16 or 64 MB of data.
• These are treated independently.
• The output of these then is broken into reduce shards and sent to reduce workers.
• Assignment of a key-value to a given reduce shard can be done by applying a hash value to its key.

Combiners

• In many MapReduce jobs, a single map shard may produce a large number of key/value pairs for the same key. For example, the word "the" might account for 6-7% of the output in our word counting example.
• Forwarding all these tuples to the reduce worker responsible for "the" wastes network and storage resources and also causes load imbalances.
• To overcome this problem each map worker can also do the shuffle/reduce phase on its portion then send the result to the corresponding reduce worker.
• This kind of reduce that is applied to a map shard is called a combiner.

Fault Tolerance

• If a machine computing a map shard fails the key-values can be sent to a new machine and that shard can be recomputed without having to recompute any of the other map shard jobs.
• At the reduce level, a given reduce shard might depend on keys from several map jobs.
• To prevent having to recompute each of these dependencies on failure, typically, the inputs to a reduce shard are stored in distributed file system, so in the event of a failure they can be re-read from there.