Outline

• Huffman Coding
• Arithmetic Coding
• Quiz
• Gap Compression

Huffman Coding

• One of the most popular bitwise coding techniques is due to Huffman (1952).
• For a given probability distribution `M` on a finite set of symbols `{sigma_1, ... sigma_n}`, this scheme produces a prefix code `C` that minimizes
  `sum_(i=1)^n M(sigma_i) |C(sigma_i)|`.
• A Huffman code is optimal amongst codes that use an integral number of bits per symbol.

Huffman Tree Construction

• Suppose we have a compression model `M` with `M(sigma_i) = Pr(sigma_i)`.
• To construct a Huffman code, we:
  1. Start with a set of trees `T_i` one for each `sigma_i`. The probability of a tree will be the sum of the probabilities of the symbols in it.
  2. We next repeat until there is only one tree left and do:
     Take the two trees of least probability, say `T_j` and `T_k`, and merge these into a single tree `T_i` consisting of a new top node labeled with the sum of `T_j` and `T_k`'s probabilities and having the more probable tree as the left child and the less probable as the right child. The edge to left child is labeled 0; to the right is labeled 1.
• Once this tree is constructed, codes for symbols correspond to paths down this tree to the symbol in question.

Example

Facts About Huffman Codes

• An optimal prefix code `C_(opt)` must satisfy `Pr[x] < Pr[y] => |C_(opt)(x)| ge |C_(opt)(y)|` for every pair of symbols `x`, `y`.
• The book uses this together with a proof by induction to show the Huffman code is in fact optimal.
• The Huffman tree has `2n-1` nodes. So after the tree is constructed we can traverse the tree to assign codewords to symbols in `Theta(n)` time.
• To actually build the tree, we need to keep track of the two trees of least probability.
• If one implements this using a priority queue, the tree construction phase can be done in `Theta(n log n)` time.
• Thus, the complete algorithm is `Theta(n log n)` time.

Canonical Huffman Codes

• In order to use a Huffman code, the decoder needs to know how each symbol is encoded.
• This description is prepended to the actual message being compressed in what's known as the preamble.
• It might consist of a list of symbols together with their codes. For example, `((sigma_1, 10)`, `(sigma_2, 110)`, `(sigma_3, 01)`, `(sigma_4, 00)`, `(sigma_5, 111))`
• Consider the following Huffman tree:
  
• The length of each code word is the same as for the first tree we had built a couple slides back. So there is no reason to prefer that code to the one above.
• Moreover, reading the symbols `sigma_i` from left to right in the above, corresponds to ordering them by length of their respective codewords; ties broken according to lex order (i.e., `sigma_1` before `sigma_3`).
• A code with such a property is called a canonical Huffman code.
• For any Huffman code a canonical Huffman code exists because for two nodes of the same depth in the Huffman tree we can reorder them at that level without affecting the length on any symbol's code word.
• For such a code it suffices to write down the length of the codes rather than the actual codewords: `((sigma_1, 2)`, `(sigma_2, 3)`, `(sigma_3, 2)`, `(sigma_4, 2)`, `(sigma_5, 3))`.
• Since we known the symbols in advance, we can further simplify this to: `(2, 3, 2, 2, 3)`

Motivating Arithmetic Coding

• Consider the following probability distribution for a two symbol alphabet `S = {a, b}`: `Pr(a) = 0.8`, `Pr(b) = 0.2`.
• Shannon's Theorem says symbol sequences generated according to this distribution cannot be encoded using less than `0.7219` bits/symbol on average.
• However, a Huffman code will need to use `1` bit per symbol, `39%` more than the lower bound.
• Arithmetic codes will allow as to get closer to the lower bound by dispensing with the idea that each symbol gets a separate codeword.
• As an example, we could improve upon the Huffman code for single symbols by taking pairs of symbols and making the Huffman code for them. In our example, we code "aa", "ab", "ba", and "bb" using a Huffman code. This actually reduces the the bits per symbol to `0.78`.

Arithmetic Coding

• Consider a sequence of `k` symbols from the set `S={sigma_1, ... sigma_n}`:
  `langle s_1, s_2 ..., s_k rangle in S^k`.
• Each such sequence has a certain probability associated with it: `prod_(i=1)^kPr(s_i)`.
• The sum over all such sequence is 1.
• We can think of a sequence `x` as an interval `[x_1, x_2)`, with `0 le x_1 le x_2 < 1` and `x_2 - x_1 = Pr(x)`.
• These sequences are arranged as subintervals of the interval `[0, 1)`, in lex order of the respective symbol sequence and form a partitioning of `[0, 1)`.
  

More Arithmetic Coding

• We can code a message by encoding the associated interval `I` instead of the message itself.
• It actually suffice to encode a smaller interval `I'` such that `I' subseteq I` and such that `I'` have the form `I' = [x, x+2^(-q))` with `x = sum_(i=1)^q a_i cdot 2^(-i)` where (`a_i in {0, 1}`).
• Such an interval is called a binary interval.
• We encode a binary interval as just the bit sequence `langle a_1, a_2, ... a_q rangle`.
• For example, `0`, encodes the interval `[0, 0.5)`, the bit sequence `01` encodes `[0.25, 0.5)`.
• These two steps: (1) Transforming a message into an equivalent interval; (2) Encoding a binary interval within `I` as a simple bit sequence -- is called arithmetic coding.
• If `I=[y, y +p)`, the interval `I' = [x, x+2^(-q))` we want to use is the one of smallest `q` such that
  `y le x < x + 2^(-q) le y +p.`
• The smallest `q` for which this works is `q= |~ - log (p) ~| + 1`.

Redux

• If probabilities of each k sequence are of finite precision, one can imagine using an integer to store some maximum denominator and then avoid using floating points and their associated error when doing arithmetic coding.
• Witten et al. (1987) have shown how to do this in a systematic way.
• Huffman tends to be faster to encode and decode.
• Both of these algorithms are used in more sophisticated algorithms such as gzip (Ziv Lemmpel, 1977) and bzip (Burrows and Wheeler, 1994).
• Straight text compression as we have seen might be used to compress the downloaded pages of a search engine or snippet-ready summaries generated from them.

Quiz

Which of the following is true?

1. Given a list of intervals `S`, the function `G(S)` return a sublist of `S` which is a GC-list.
2. The trec_eval software is used to replace human relevance judgments.
3. For any `n` bit number, it takes `3n` bits to write it as a `gamma` code.

Finishing Up General Text Compression

• Up till now we have been looking at general purpose text-compression algorithms.
• Such algorithms might be suitable for compressing the raw text downloaded from a collection of web sites.
• For example, the file formats that the internet archive uses for web downloads, arc and warc, consist of a file which contains a sequence of objects each of which is compressed using gzip.
• We have looked at Huffman and Arithmetic coding, gzip and bzip use more sophisticated generalization of these techniques.
• `k`th order versions of Huffman coding use a separate dictionary for each `k-1` previous character contexts and then code the `k`th character.
• Length-limited Huffman codes start with a canonical Huffman tree and do manipulations to it to produce a prefix-free code in which the no codeword for an symbol is longer than some value.
• The book has a slide comparing 0th, 1st, 2nd order variant of Huffman, Arithmetic coding versus each other and versus gzip and bzip2 for Shakespeare, TREC45, and Gov2.
• Because Huffman has to store the code table, it performed the worst, followed by arithmetic coding followed by gzip, followed by bzip2 (2.5 times better than Huffman, and between 4.5 and 5 times compression over the original text).
• There is a Huffman encoding scheme called LLRUN, which is sometimes used to do posting list compression.

Compressing Posting Lists: `Delta`-values

• A posting list consists of a sequence of integers giving the doc id's of the document that contained a particular word.
• So it looks like a sequence of increasing integers such as: `L = langle 3, 7, 11, 23, 29, 37, 41 ... rangle`
• In the worst case you might imagine having to use a whole int to store a particular posting.
• On the other hand if we look at the difference between adjacent pairs, the numbers involved are usually quite small (but all greater than 0): `Delta(L) = \langle 3, 4, 4, 12, 6, 8, 4... rangle `.
• These are called `Delta`-values. One of the games of index compression is to understand the distribution of `Delta`-values so that one can find a good compression scheme for them.
• It should be noted that if one has a positional index rather than a schema independent index it also makes sense to compute the delta values for the positions within a document.
• Two common classes of codes used to do this are nonparametric and parametric codes: The former don't take into account the distribution when encoding, the latter do. Usually, the former are faster, the latter compress better.

Nonparametric Gap Compression

• The simplest nonparametric code on the positive integers is the unary code: code `k` as k-1 0's followed by a 1.
• This code is optimal if the `Delta`-values follow a geometric distribution of the form: `Pr[Delta = k] = 2^(-k)`. This is usually not the case.
• Elias (1975) proposed the `gamma` code which we have already mentioned.
• To code `k` we use `lfloor log_2(k) rfloor` 0's followed k written in binary.
• So `|\gamma(k)| = 2 lfloor log_2(k) rfloor + 1` bits.
• The `gamma`-code is good for compressing gaps of size less than 32. It is optimal when the gaps are distributed according to `1/(2k^2)`.
• If we encode `k` by first encoding its length with a `gamma` code followed by its binary representation we get a `delta`-code. It performs better for larger gaps and is optimal for distributions of the form `1/(2k(\log k)^2)`.
• One could imagine encoding `log^star k` using a unary code, followed by binary codes for each of the `m`-fold iterates of `log` between `log^star k` and `0` to get a code for `k`. This is essentially the `omega` code for `k`.

Parametric Gap Compression

• Unlike nonparametric codes, parametric ones take into account the specific characteristics of the list to be compressed.
• Parametric codes fall into two categories: global, which choose a single parameter value that is used for all lists in the index; and local, which use a different value for each list.
• We will mainly consider local methods on the next few slides...
• One example of a global techniques is to to group gaps into buckets, `B_j = [2^j, 2^(j+1))`.
• We use Huffman coding to encode the gaps seen and store the encoding table once. Then an individual gap is encoded as its Huffman code for its bucket, followed by its binary representation.
• This scheme is called LLRUN.
• It uses a 0th order language model. The book also talks about variants that use higher-order models.

Geometric Distributions and Posting Lists

• Suppose the `Delta`-values follow a geometric distribution: `Pr[Delta = k] = (1 - p)^(k-1)p`.
• This is not unreasonable: If our collection has N documents and term T appears in `N_T` of them. One would expect that the probability `p` that `T` appears in a given document chosen at random is something like `N_T/N`.
• So if all documents are independent of each other, the probability of seeing a gap of size `k` between two subsequent occurrences of `T` is:
  `Pr[Delta =k] = (1 - frac(N_T)(N))^(k-1)cdot frac(N_t)(N)`
• The diagram below show this distribution:
  

Golomb/Rice Codes

• Notice that most of the gaps fall around a common length. In the picture in the last slide this was between length 6 and 8.
• Call this common length the modulus.
• One way to encode the gaps is to: (1) determine an appropriate modulus, (2) split each `Delta`-value `k` into two components: a quotient q(k) and a remainder r(k) where:
  `q(k) = lfloor (k-1)/M rfloor, r(k) = (k -1)mod M`
  Encode `k` by writing `q(k)+1` in unary followed by `r(k)` as a `lfloor log(M) rfloor` bit or `|~ log(M) ~|` bit number.
• For general `M` the above code is called a Golomb code after S. Golomb (1966). If `M` is a power of `2` then is is called a Rice code after the paper that popularized this kind of coding R. Rice (1971).
• Rice codes tends to be only slightly less efficient in terms of compression, but run faster.

Finding the Modulus

• Recall a code `C` is optimal with respect to a given probability distribution `M` if the relationship
  `|C(sigma_1)| = |C(sigma_2)|+ 1`
  for two symbols `sigma_1` and `sigma_2` implies `M(sigma_1) = 1/2 cdotM(sigma_2)`.
• We know that the Golomb codeword for integer `k+M` is `1` bit longer than the codeword for the integer `k`.
• Therefore, the optimal parameter value for the modulus (call this modulus `M^star`) is: `Pr[Delta = k + M^star] = 1/2Pr[Delta = k] iff (1 - N_t/N)^(k+M^star - 1) = 1/2 (1 - N_T/N)^(k-1)`
  (divide the common `N_t/N` factor on of both sides)
  `iff M^star = frac(- log(2))(log(1 - N_T/N))`.
• This is not usually an integer.
• For Rice coding we need to choose between `M = 2^(lfloor log_2 M^star rfloor)` and `M = 2^(|~ log_2 M^star ~|)`.
• For Golomb coding we choose between `\lfloor M^star rfloor` and `|~ M^star ~|`.
• Gallagher and van Voorhis (1975) have shown that the optimal choice is:
  `M_(opt) = |~ frac(log(2 - N_T/N))(-log(1 - N_T/N)) ~|`.

Byte-Aligned Codes

• If we want to improve compression and decompression speeds, it makes sense to look at compression methods that are more geared to the way microprocessors in today's computers work.
• Thus, it make sense to look at codes such that the split between code words falls on byte or word boundaries.
• One of the simplest of these kinds of codes is vByte. In vByte, we use the top bit of a byte to say if the thing we are encoding continues to the next byte (1) or not (0).
• We then use the remaining seven bits of each byte in our code as payload. And we encode from low order bit of the bit string we are encoding to high-order bit.
• You can also have word-aligned codes which work on multiples of 32 or 64 bits.
• One example of these is Simple-9 proposed by Anh and Moffat (2005). In this coding scheme, the first 4 bits of a 32 bits word server as a selector. The remaining 28 bits are split into chunks according to the particular selector.
• For example, if the top four bits were 0000, then 28 1 bit numbers would be encoded in the word; if the selector was 0001, 14, 2 bit numbers could be encoded and so on.