Outline

• Huffman Coding
• In-Class Exercise

More on Prefix Codes

• On Monday, we started talking about compression algorithms.
• We said one approach to compression is come up with some kind of model for the probability of a symbol by itself, or a prior sequence of symbols, then come up with a code for that symbol.
• We were looking at coding our symbols using prefix codes, codes where no code word is a prefix of another.
• Such codes are natural uniquely decodable into the original symbols.
• A prefix code C can be thought as a binary tree in which each leaf node corresponds to a symbol `sigma`.
• The labels (0 for left, 1 for right) encountered along the path from the root to a leaf correspond to the symbol's codeword `C(sigma)`.
• Without prefix freeness, some symbol would be associated with an internal node in this kind of tree.
• Suppose for all of our symbols `M(sigma_i) = 2^(-lambda_i)` for some `lambda_i in NN`.
• Because `M` is a probability distribution, we have
  `sum_(i=1)^(n)M(sigma_i) = sum_(i=1)^(n)2^(-lambda_i) = 1`.
• Let's try to find an optimal code tree for this distribution.

Making an optimal code tree

• Every node in our tree must be either a leaf node (and have a codeword associated with it) or an internal node with exactly two children. i.e., A proper binary tree.
• If the tree had an internal node with only one child, we could remove the internal node and shorten its descendant codewords by 1.
• Let `L = {L_1, ..., L_n}` be the set of leaf nodes in a proper binary tree. Then: `sum_(i=1)^n2^(-d(L_i)) = 1`, where `d(L_i)` is the depth of node `L_i`.
• Thus, it makes sense to try to choose our codewords to symbols in such a way that: `|C(sigma_i)| = d(L_i) = lambda_i = - log(M(sigma_i))`.
• The resulting tree would represent an optimal code for the probability distribution `M` because the average number of bits per symbol used by `C` if we encode a sequence of symbols generated according to `M` would be: `sum_(i=1)^(n)Pr[sigma_i]cdot |C(sigma_i)| = -sum_(i=1)^(n) M(sigma_i) cdot log(M(sigma_i))`
  because `|C(sigma_i)| = - log(M(S_i))`.

Making an optimal code tree

• This would be optimal because of the following theorem from Shannon (1949):
  Source Coding Theorem. Given a symbol source S, emitting symbols from an alphabet A according to a probability distribution `P_A`, a sequence of symbols cannot be compressed to consume less than
  `H(A) = -sum P_A(sigma) cdot log(P_A(sigma))`
  bits per symbol on average. Here H(A) is called the entropy of the symbol source S.

Huffman Coding

• One of the most popular bitwise coding techniques is due to Huffman (1952).
• For a given probability distribution `M` on a finite set of symbols `{sigma_1, ... sigma_n}`, this scheme produces a prefix code `C` that minimizes
  `sum_(i=1)^n M(sigma_i) |C(sigma_i)|`.
• A Huffman code is optimal amongst codes that use an integral number of bits per symbol.

Huffman Tree Construction

• Suppose we have a compression model `M` with `M(sigma_i) = Pr(sigma_i)`.
• To construct a Huffman code, we:
  1. Start with a set of trees `T_i` one for each `sigma_i`. The probability of a tree will be the sum of the probabilities of the symbols in it.
  2. We next repeat until there is only one tree left and do:
     Take the two trees of least probability, say `T_j` and `T_k`, and merge these into a single tree `T_i` consisting of a new top node labeled with the sum of `T_j` and `T_k`'s probabilities and having the more probable tree as the left child and the less probable as the right child. The edge to left child is labeled 0; to the right is labeled 1.
• Once this tree is constructed, codes for symbols correspond to paths down this tree to the symbol in question.

Example

In-Class Exercise

Suppose the `i`th letter of the English alphabet occurs with probability `i/600` for the letters `a` through `x` and that the letters `y` and `z` occurs with probability `1/4` what would the Huffmann tree look like?

Post your solution to the Nov 1 In-Class Exercise Thread.

Facts About Huffman Codes

• An optimal prefix code `C_(opt)` must satisfy `Pr[x] < Pr[y] => |C_(opt)(x)| ge |C_(opt)(y)|` for every pair of symbols `x`, `y`.
• The book uses this together with a proof by induction to show the Huffman code is in fact optimal.
• The Huffman tree has `2n-1` nodes. So after the tree is constructed we can traverse the tree to assign codewords to symbols in `Theta(n)` time.
• To actually build the tree, we need to keep track of the two trees of least probability.
• If one implements this using a priority queue, the tree construction phase can be done in `Theta(n log n)` time.
• Thus, the complete algorithm is `Theta(n log n)` time.

Canonical Huffman Codes

• In order to use a Huffman code, the decoder needs to know how each symbol is encoded.
• This description is prepended to the actual message being compressed in what's known as the preamble.
• It might consist of a list of symbols together with their codes. For example, `((sigma_1, 10)`, `(sigma_2, 110)`, `(sigma_3, 01)`, `(sigma_4, 00)`, `(sigma_5, 111))`
• Consider the following Huffman tree:
  
• The length of each code word is the same as for the first tree we had built a couple slides back. So there is no reason to prefer that code to the one above.
• Moreover, reading the symbols `sigma_i` from left to right in the above, corresponds to ordering them by length of their respective codewords; ties broken according to lex order (i.e., `sigma_1` before `sigma_3`).
• A code with such a property is called a canonical Huffman code.
• For any Huffman code a canonical Huffman code exists because for two nodes of the same depth in the Huffman tree we can reorder them at that level without affecting the length on any symbol's code word.
• For such a code it suffices to write down the length of the codes rather than the actual codewords: `((sigma_1, 2)`, `(sigma_2, 3)`, `(sigma_3, 2)`, `(sigma_4, 2)`, `(sigma_5, 3))`.
• Since we known the symbols in advance, we can further simplify this to: `(2, 3, 2, 2, 3)`