Outline

• Arithmetic Coding
• Quiz
• Gap Compression

Motivating Arithmetic Coding

• Last week, we were talking about using compression algorithms to try to shrink the size of our inverted index.
• We began by looking at general compression techniques and in particular Huffman encoding as a general form of text compression.
• We now consider a second kind of general text compression technique.
• Consider the following probability distribution for a two symbol alphabet `S = {a, b}`: `Pr(a) = 0.8`, `Pr(b) = 0.2`.
• Shannon's Theorem says symbol sequences generated according to this distribution cannot be encoded using less than `0.7219` bits/symbol on average.
• However, a Huffman code will need to use `1` bit per symbol, `39%` more than the lower bound.
• Arithmetic codes will allow as to get closer to the lower bound by dispensing with the idea that each symbol gets a separate codeword.
• As an example, we could improve upon the Huffman code for single symbols by taking pairs of symbols and making the Huffman code for them. In our example, we code "aa", "ab", "ba", and "bb" using a Huffman code. This actually reduces the the bits per symbol to `0.78`.

Arithmetic Coding

• Consider a sequence of `k` symbols from the set `S={sigma_1, ... sigma_n}`:
  `langle s_1, s_2 ..., s_k rangle in S^k`.
• Each such sequence has a certain probability associated with it: `prod_(i=1)^kPr(s_i)`.
• The sum over all such sequence is 1.
• We can think of a sequence `x` as an interval `[x_1, x_2)`, with `0 le x_1 < x_2 < 1` and `x_2 - x_1 = Pr(x)`.
• These sequences are arranged as subintervals of the interval `[0, 1)`, in lex order of the respective symbol sequence and form a partitioning of `[0, 1)`.
  

More Arithmetic Coding

• We can code a message by encoding the associated interval `I` instead of the message itself.
• It actually suffice to encode a smaller interval `I'` such that `I' subseteq I` and such that `I'` have the form `I' = [x, x+2^(-q))` with `x = sum_(i=1)^q a_i cdot 2^(-i)` where (`a_i in {0, 1}`).
• Such an interval is called a binary interval.
• We encode a binary interval as just the bit sequence `langle a_1, a_2, ... a_q rangle`.
• For example, `0`, encodes the interval `[0, 0.5)`, the bit sequence `01` encodes `[0.25, 0.5)`.
• These two steps: (1) Transforming a message into an equivalent interval; (2) Encoding a binary interval within `I` as a simple bit sequence -- is called arithmetic coding.
• If `I=[y, y +p)`, the interval `I' = [x, x+2^(-q))` we want to use is the one of smallest `q` such that
  `y le x < x + 2^(-q) le y +p.`
• The smallest `q` for which this works is `q= |~ - log (p) ~| + 1`.

Redux

• If probabilities of each k sequence are of finite precision, one can imagine using an integer to store some maximum denominator and then avoid using floating points and their associated error when doing arithmetic coding.
• Witten et al. (1987) have shown how to do this in a systematic way.
• Huffman tends to be faster to encode and decode.
• Both of these algorithms are used in more sophisticated algorithms such as gzip (Ziv Lempel, 1977) and bzip (Burrows and Wheeler, 1994).
• Straight text compression as we have seen might be used to compress the downloaded pages of a search engine or snippet-ready summaries generated from them.

Finishing Up General Text Compression

• Up till now we have been looking at general purpose text-compression algorithms.
• Such algorithms might be suitable for compressing the raw text downloaded from a collection of web sites.
• For example, the file formats that the internet archive uses for web downloads, arc and warc, consist of a file which contains a sequence of objects each of which is compressed using gzip.
• We have looked at Huffman and Arithmetic coding, gzip and bzip use more sophisticated generalization of these techniques.
• `k`th order versions of Huffman coding use a separate dictionary for each `k-1` previous character contexts and then code the `k`th character.
• Length-limited Huffman codes start with a canonical Huffman tree and do manipulations to it to produce a prefix-free code in which no codeword for a symbol is longer than some value.
• The book has a slide comparing 0th, 1st, 2nd order variant of Huffman, Arithmetic coding versus each other and versus gzip and bzip2 for Shakespeare, TREC45, and Gov2.
• Because Huffman has to store the code table, it performed the worst, followed by arithmetic coding followed by gzip, followed by bzip2 (2.5 times better than Huffman, and between 4.5 and 5 times compression over the original text).
• There is a Huffman encoding scheme called LLRUN, which is sometimes used to do posting list compression.

Compressing Posting Lists: `Delta`-values

• A posting list consists of a sequence of integers giving the doc id's of the document that contained a particular word.
• So it looks like a sequence of increasing integers such as: `L = langle 3, 7, 11, 23, 29, 37, 41 ... rangle`
• In the worst case you might imagine having to use a whole int to store a particular posting.
• On the other hand if we look at the difference between adjacent pairs, the numbers involved are usually quite small (but all greater than 0): `Delta(L) = \langle 3, 4, 4, 12, 6, 8, 4... rangle `.
• These are called `Delta`-values. One of the games of index compression is to understand the distribution of `Delta`-values so that one can find a good compression scheme for them.
• It should be noted that if one has a positional index rather than a schema independent index it also makes sense to compute the delta values for the positions within a document.
• Two common classes of codes used to do this are nonparametric and parametric codes: The former don't take into account the distribution when encoding, the latter do. Usually, the former are faster, the latter compress better.

Nonparametric Gap Compression

• The simplest nonparametric code on the positive integers is the unary code: code `k` as k-1 0's followed by a 1.
• This code is optimal if the `Delta`-values follow a geometric distribution of the form: `Pr[Delta = k] = 2^(-k)`. This is usually not the case.
• Elias (1975) proposed the `gamma` code which we have already mentioned.
• To code `k` we use `lfloor log_2(k) rfloor` 0's followed k written in binary.
• So `|\gamma(k)| = 2 lfloor log_2(k) rfloor + 1` bits.
• The `gamma`-code is good for compressing gaps of size less than 32. It is optimal when the gaps are distributed according to `1/(2k^2)`.
• If we encode `k` by first encoding its length with a `gamma` code followed by its binary representation we get a `delta`-code. It performs better for larger gaps and is optimal for distributions of the form `1/(2k(\log k)^2)`.
• One could imagine encoding `log^star k` using a unary code, followed by binary codes for each of the `m`-fold iterates of `log` between `log^star k` and `0` to get a code for `k`. This is essentially the `omega` code for `k`.

Quiz

Which of the following is true?

1. Given a symbol source `S`, emitting symbols from an alphabet `A` according to a probability distribution `P_A`, a sequence of symbols cannot be compressed to consume less than its entropy.
2. To code a string using a `gamma`-code we first build a Huffman tree.
3. The trec_eval program makes use of a sub-module to automatically judge relevance of documents without human involvement.

Parametric Gap Compression

• Unlike nonparametric codes, parametric ones take into account the specific characteristics of the list to be compressed.
• Parametric codes fall into two categories: global, which choose a single parameter value that is used for all lists in the index; and local, which use a different value for each list.
• We will mainly consider local methods on the next few slides...
• One example of a global techniques is to to group gaps into buckets, `B_j = [2^j, 2^(j+1))`.
• We use Huffman coding to encode the gaps seen and store the encoding table once. Then an individual gap is encoded as its Huffman code for its bucket, followed by its binary representation.
• This scheme is called LLRUN.
• It uses a 0th order language model. The book also talks about variants that use higher-order models.

Geometric Distributions and Posting Lists

• Suppose the `Delta`-values follow a geometric distribution: `Pr[Delta = k] = (1 - p)^(k-1)p`.
• This is not unreasonable: If our collection has N documents and term T appears in `N_T` of them. One would expect that the probability `p` that `T` appears in a given document chosen at random is something like `N_T/N`.
• So if all documents are independent of each other, the probability of seeing a gap of size `k` between two subsequent occurrences of `T` is:
  `Pr[Delta =k] = (1 - frac(N_T)(N))^(k-1)cdot frac(N_t)(N)`
• The diagram below show this distribution:
  

Golomb/Rice Codes

• Notice that most of the gaps fall around a common length. In the picture in the last slide this was between length 6 and 8.
• Call this common length the modulus.
• One way to encode the gaps is to: (1) determine an appropriate modulus, (2) split each `Delta`-value `k` into two components: a quotient q(k) and a remainder r(k) where:
  `q(k) = lfloor (k-1)/M rfloor, r(k) = (k -1)mod M`
  Encode `k` by writing `q(k)+1` in unary followed by `r(k)` as a `lfloor log(M) rfloor` bit or `|~ log(M) ~|` bit number.
• For general `M` the above code is called a Golomb code after S. Golomb (1966). If `M` is a power of `2` then is is called a Rice code after the paper that popularized this kind of coding R. Rice (1971).
• Rice codes tends to be only slightly less efficient in terms of compression, but run faster.

Finding the Modulus

• Recall a code `C` is optimal with respect to a given probability distribution `M` if the relationship
  `|C(sigma_1)| = |C(sigma_2)|+ 1`
  for two symbols `sigma_1` and `sigma_2` implies `M(sigma_1) = 1/2 cdotM(sigma_2)`.
• We know that the Golomb codeword for integer `k+M` is `1` bit longer than the codeword for the integer `k`.
• Therefore, the optimal parameter value for the modulus (call this modulus `M^star`) is: `Pr[Delta = k + M^star] = 1/2Pr[Delta = k] iff (1 - N_T/N)^(k+M^star - 1) = 1/2 (1 - N_T/N)^(k-1)`
  (divide the common `N_t/N` factor on of both sides)
  `iff M^star = frac(- log(2))(log(1 - N_T/N))`.
• This is not usually an integer.
• For Rice coding we need to choose between `M = 2^(lfloor log_2 M^star rfloor)` and `M = 2^(|~ log_2 M^star ~|)`.
• For Golomb coding we choose between `\lfloor M^star rfloor` and `|~ M^star ~|`.
• Gallagher and van Voorhis (1975) have shown that the optimal choice is:
  `M_(opt) = |~ frac(log(2 - N_T/N))(-log(1 - N_T/N)) ~|`.

Byte-Aligned Codes

• If we want to improve compression and decompression speeds, it makes sense to look at compression methods that are more geared to the way microprocessors in today's computers work.
• Thus, it make sense to look at codes such that the split between code words falls on byte or word boundaries.
• One of the simplest of these kinds of codes is vByte. In vByte, we use the top bit of a byte to say if the thing we are encoding continues to the next byte (1) or not (0).
• We then use the remaining seven bits of each byte in our code as payload. And we encode from low order bit of the bit string we are encoding to high-order bit.
• You can also have word-aligned codes which work on multiples of 32 or 64 bits.
• One example of these is Simple-9 proposed by Anh and Moffat (2005). In this coding scheme, the first 4 bits of a 32 bits word server as a selector. The remaining 28 bits are split into chunks according to the particular selector.
• For example, if the top four bits were 0000, then 28 1 bit numbers would be encoded in the word; if the selector was 0001, 14, 2 bit numbers could be encoded and so on.