Outline

• trec_eval
• In-Class Exercise
• Index Compression

trec_eval

• We've mentioned trec_eval a couple of times in class.
• trec_eval is used to make it easier to compute precision, recall, and MAP scores and is descended from the SMART system developed at Cornell in the 1960s.
• I would like to spend a minute now to demo it.
• Recall trec_eval is software available as NIST.
• It is a C program which can be compiled from the command-line using the standard:

  make
  sudo make install
  

• This should compile the program and put the binary in /home/smart/bin/
• If you want to change the install location, you need to edit the Makefile, changing the BIN variable.
• For Windows, you either need to install Cygwin or edit the Makefile so that it works with nmake and cl (assuming you have Visual Studio or VS Express).

trec_eval command-line arguments

• trec_eval is run from the command-line with a command like: trec_eval [-h] [-q] [-m] trec_rel_file trec_top_file
• The -h option is used to print help.
• The -q option says whether to give statistics on each query. Without it only aggregate statistics for the whole file are generated.
• The -m option will cause trec_eval to generate statistics for the supplied measure as opposed to the official measures used in TREC. For example, -m all_qrels generates any statistic that trec_eval is capable of; -m official generates just the official ones.
• The trec_rel_file is a file consisting of a sequence of lines giving judgements on the relevance of documents (qrels).
• The trec_top_file consists of the results from your program -- in an appropriate format.

trec_rel_file and trec_top_file

• The trec_rel_file should consist of a sequence of lines in the format:

  qid  iter  docno  rel
  

  The columns are space-delimited. qid is a query integer, and the lines are assumed to be sorted in this column, iter is a iteration number which we won't use (set to 0), docno is a document id string, and rel is 1 or 0 and means the docno was a relevant result for the query.
• Recall qid would correspond to a num of a topic in a topics file that would be given by TREC for a competition and the title of the given topic would give the actual query.
• An example line might look like:

  251 0 AP880215-0135 0
  

• The trec_rel_file would be human judged after maybe the pooling step that we described earlier in the semester.
• The trec_top_file contains information about what a particular IR system computed on a set of queries. It should consist of a sequence of lines in the format:

  query_id iter docno rank sim run_id
  

  Columns are again space delimited. query_id is again the query this result is for, iter we again set to 0, docno is the doc id string of a document returned, rank is its rank 0-1000 -- required but ignored, sim is a float similarity score -- this is not ignored and should be higher for more relevant results, run_id is a string to be printed out along with the results.
• An example line might look like:

  351   0  FR940104-0-00001  1   102.38   my-test
  

Example trec_rel_file

351 0 FR940104-0-00001 0
351 0 FR940104-0-00002 1
351 0 FR940104-0-00003 1
351 0 FR940104-0-00004 1
351 0 FR940104-0-00005 1
351 0 FR940104-0-00006 1
351 0 FR940104-0-00007 1
351 0 FR940104-0-00008 1
351 0 FR940104-0-00009 1
351 0 FR940104-0-00010 1
351 0 FR940104-0-00011 0
351 0 FR940104-0-00012 1

Example trec_top_file

351   0  FR940104-0-00001  1   102.38   run-nam2
351   0  FR940104-0-00002  1   101.38   run-nam2
351   0  FR940104-0-00003  1   91.38   run-nam2
351   0  FR940104-0-00004  1   81.38   run-nam2
351   0  FR940104-0-00005  1   71.38   run-nam2
351   0  FR940104-0-00006  1   61.38   run-nam2
351   0  FR940104-0-00007  1   51.38   run-nam2
351   0  FR940104-0-00008  1   41.38   run-nam2
351   0  FR940104-0-00009  1   31.38   run-nam2
351   0  FR940104-0-00010  1   22.38   run-nam2
351   0  FR940104-0-00011  1   21.38   run-nam2
351   0  FR940104-0-00012  1   11.38   run-nam2

Example trec_eval Output

runid                 	all	run-nam2
num_q                 	all	1
num_ret               	all	12
num_rel               	all	10
num_rel_ret           	all	10
map                   	all	0.7904
gm_map                	all	0.7904
Rprec                 	all	0.9000
bpref                 	all	0.4500
recip_rank            	all	0.5000
iprec_at_recall_0.00  	all	0.9000
iprec_at_recall_0.10  	all	0.9000
iprec_at_recall_0.20  	all	0.9000
iprec_at_recall_0.30  	all	0.9000
iprec_at_recall_0.40  	all	0.9000
iprec_at_recall_0.50  	all	0.9000
iprec_at_recall_0.60  	all	0.9000
iprec_at_recall_0.70  	all	0.9000
iprec_at_recall_0.80  	all	0.9000
iprec_at_recall_0.90  	all	0.9000
iprec_at_recall_1.00  	all	0.8333
P_5                   	all	0.8000
P_10                  	all	0.9000
P_15                  	all	0.6667
P_20                  	all	0.5000
P_30                  	all	0.3333
P_100                 	all	0.1000
P_200                 	all	0.0500
P_500                 	all	0.0200
P_1000                	all	0.0100

In-Class Exercise

1. Download the trec_eval software and compile it.
2. Pick the first three rows from the example trec_rel_file I gave that were marked as relevant and mark them as not relevant, save the file.
3. Pick the first two rows from the example trec_top_file I gave and swap their sim scores, save the file.
4. Run trec_eval on the results.

Post each of the file contents and the output of trec_eval to the Oct 24 In-Class Exercise Thread.

Index Compression

• As we mentioned an inverted index can be quite large, especially if we have a full positional index.
• The book gives an example which shows for the three text collections considered by the book: the Shakespeare Corpus (139%), TREC45 (122%) and GOV2 (77%), the corresponding uncompressed inverted index is typically larger than the original content. GOV2 would have been bigger if Javascript and tags were not included in download size.
• Since reducing the size of the index will both mean that we can index larger collections and as it will also speed-up the query retrieval from such collections we will now look at different techniques for compressing inverted indexes.
• We begin by considering how compression algorithms work in general.

General-Purpose Data Compression

• A data compression algorithm takes a chunk of data A and transforms it into another chunk of data B that is hopefully smaller that A.
• Such an algorithm consists of two components: the encoder which takes A and produces B as above; and the decoder which takes B and outputs some C related to the original A.
• A compression method can be either lossless or lossy.
• If it is lossless than C in the above should be an exact copy of A; if it is lossy than C might be a "good enough" approximation of A (For example, JPEG encoder does a lossy compression of image, the decoder produces an image that is similar to the original.)
• For index compression, we are mainly interested in lossless compression.

Symbolwise Data Compression

• Given a chunk of data A, we will usually be concerned with the information contained in A rather than the precise bit string used for A. This is called the message, and we will denote it by M.
• This message is often represented as a sequence of symbols from some set called an alphabet. I.e.,
  `M = langle sigma_1, ... sigma_n rangle, sigma_i in S`
• Data compression methods based on messages of symbols are called symbol-wise or statistical techniques.
• Such techniques rely on the facts: (1) Not all symbols in M appear with the same frequency. By encoding more frequent symbols using fewer bits than less frequent ones we can save space. (2) The `i`th symbol in a sequence `langle sigma_1, ... sigma_n rangle` often depends on the previous symbols `langle sigma_1, ... sigma_(i-1) rangle` so we can save space in writing `sigma_i` by taking this into account. For example, we might code `qu` as one symbol as `q` is almost always followed by a `u`.
• ASIDE: You can think of this as an n-gram approach to compression, people also consider Markov Model approaches (probabilistic finite automata) and probabilistic context free grammar approaches (Liv Zempel, Sequitur).

Modeling and Coding

• Symbolwise compression methods often work in two phases: modeling and coding.
• In the the modeling phase a probability distribution `M` is computed that maps symbols to their probability of occurrence.
• In the coding phase the symbols in the message are re-encoded according to a code C.
• A code is a map from symbols to codewords: `sigma |-> C(sigma)`
• Methods can be further classified as static, semi-static, adaptive depending on the degree to which the model `M` is independent from the particular message being compressed.
• You can also have first-order, second-order, and so on models depending whether the probability distribution `M` uses conditional probabilities.

Compression Models and Codes

• Compression models and codes are connected: Any model M has a code (or family of codes) associated with it that minimizes the average codeword length for a sequence of symbols generated according to M. Similarly, any code has an associated probability distribution for which the code is optimal.
• Consider the zeroth-order model M:
  M(a) = 0.5, M(b) = 0.25, M(c) = 0.125, and M(d) = 0.125.
• A code that is optimal with respect to M has the following property:
  |C(a)|=1, |C(b)| = 2, |C(c)| = 3, and |C(d)| = 3.
• An example of such a code is: C(a) = 0, C(b) = 11, C(c) = 100, and C(d) = 101.
• Using this code, C(aababacd) would be encoded as 00110110100101.
• Notice this code has the prefix-free property: no codeword is an initial substring (a prefix) of any other codeword.
• This makes it possible to uniquely decode the strings we output. For example if we have set C(b) = 10, then it would have been a prefix of C(c) = 100 and it would be impossible to decode the string 100 as it could mean either ba or c.

`gamma`-codes

• As another example of a code, let's briefly get ahead of ourselves and consider `gamma` codes
• Given a sequence of offsets in a posting list: 100000, 100005, 100011,... one might have a sequence of quite large numbers which would take many bits to store.
• However, the gap between successive numbers is small but always at least 1: 5, 6... Writing the first number and then the gaps saves a lot of space (gap-compression).
• To compress these numbers it suffices to be able to encode only numbers greater than or equal to 1.
• To compress these small numbers: We write the (number of bits - 1) we want in unary with 0's, followed by a 1, followed by the number in binary.
• So initially number 5 would be encoded as 001 101, 16 would be encoded as 00001 10000.
• Notice the high-order bit of the binary code for the number is redundant given that we known the length of the number, so we can drop this bit to get the actual encoding. For 5, 001 101 becomes 001 01 and for 16 one gets 00001 0000.

More on Prefix Codes

• A prefix code C can be thought as a binary tree in which each leaf node corresponds to a symbol `sigma`.
• The labels (0 for left, 1 for right) encountered along the path from the root to a leaf correspond to the symbol's codeword `C(sigma)`.
• Without prefix freeness, some symbol would be associated with an internal node in this kind of tree.
• Suppose for all of our symbols `M(sigma_i) = 2^(-lambda_i)` for some `lambda_i in NN`.
• Because M is a probability distribution, we have
  `sum_(i=1)^(n)M(sigma_i) = sum_(i=1)^(n)2^(-lambda_i) = 1`.
• Let's try to find an optimal code tree for this distribution.

Making an optimal code tree

• Every node in our tree must be either a leaf node (and have a codeword associated with it) or an internal node with exactly two children. i.e., A proper binary tree.
• If the tree had an internal node with only one child, we could remove the internal node and shorten its descendant codewords by 1.
• Let `L = {L_1, ..., L_n}` be the set of leaf nodes in a proper binary tree. Then: `sum_(i=1)^n2^(-d(L_i)) = 1`, where `d(L_i)` is the depth of node `L_i`.
• Thus, it makes sense to try to choose our codewords to symbols in such a way that: `|C(sigma_i)| = d(L_i) = lambda_i = - log(M(sigma_i))`.
• The resulting tree would represent an optimal code for the probability distribution `M` because the average number of bits per symbol used by `C` if we encode a sequence of symbols generated according to `M` would be: `sum_(i=1)^(n)Pr[sigma_i]cdot |C(sigma_i)| = -sum_(i=1)^(n) M(sigma_i) cdot log(M(sigma_i))`
  because `|C(sigma_i)| = - log(M(S_i))`.

Making an optimal code tree

• This would be optimal because of the following theorem from Shannon (1949)
  Source Coding Theorem. Given a symbol source S, emitting symbols from an alphabet A according to a probability distribution `P_A`, a sequence of symbols cannot be compressed to consume less than
  `H(A) = -sum P_A(sigma) cdot log(P_A(sigma))`
  bits per symbol on average. Here H(A) is called the entropy of the symbol source S.