Outline

• Concordance Lists
• Quiz
• trec_eval

Introduction

• Today, we are going to begin by looking at lightweight structures
• We have already given an ADT for inverted indices and shown this could be used for phrase search as well as to compute simple relationships like this phrase is contained between these tags.
• We now generalize this second query capability to region algebras.
• Region algebras provide operators and functions for combining and manipulating text intervals in support of lightweight structure.
• They represent an intermediate point between basic document retrieval and the complexity of full XML retrieval.

Concordance Lists

• Region algebras typically work with sets of text intervals.
• Any such interval can be expressed as a pair `[u, v]` indicating its start and end point.
• We restrict our attention to sets of such intervals such that no interval in the set may have another interval from the set nested within it.
• We call such a set of intervals a generalized concordance list or GC-list.
• The name refers to paper-based concordances, alphabetical listings of words in a document along with the context in which they appeared.
• Given two intervals `[u, v]` and `[u', v']` we write `[u, v] subset[u', v']` to indicate the first interval is a strict sub-interval of the latter.
• As examples, {`[5,9]`, `[8, 12]`, `[15,20]`} is a GC-list, although `[5,9]` and `[8, 12]` overlap, but {`[1, 10]`, `[5, 9]`, `[8,12]`, `[15, 20]`} is not because `[5, 9] subset [1, 10]`.
• To make a GC-list out of a list of intervals we define the function `G(S)`:
  `G(S) = {a | a in S mbox( and ) \neg exists b in S mbox( such that ) b subset a }`

Quiz

Which of the following is true?

1. `TF_(BM25)(t,d)` is upper bounded by `b + 1`.
2. The MaxScore heuristic allows one to remove terms from the term heap when doing query processing with heaps.
3. Accumulator pruning was used in our document-at-a-time query processing algorithms.

Properties of GC-lists

• Since intervals cannot nest, no two intervals in a GC-list may start at the same position.
• Similarly, no two intervals may end at the same position.
• Thus, the intervals in a GC-list can be ordered by either their start or end positions.
• The size of a GC-list can be bounded by taking the least start position and the biggest end position. and substracting
• A posting list can be viewed as a GC-list by treating each entry as an interval of length one.
• For example the posting list for the word "first" in the Shakespeare corpus might be: "first" = `{` `[2205, 2205], [2268, 2268]` ... `[1271487, 1271487] }`
• Similarly, the results of a phrase query such as "first witch" could be viewed as a GC-list.

Operators

• We next look at some common binary operators on GC-lists:
  1. Contained in/Not Contained in: `A <| B` (`<|` is supposed to be a triangle. For not contained in put a line through the triangle ). `A` contained in `B` means each interval in A is a sub-interval of some interval in `B`.
  2. Both of: `A Delta B` is defined to be `G(S)` where `S` consists of intervals `c` which contain an interval of `A` and an interval of `B`.
  3. One of: `A nabla B` is defined to be `G(S)` where `S` consists of intervals `c` which contain an interval of `A` or an interval of `B`.
  4. Before: `A ... B` is defined to be `G(S)` where `S` consists of intervals `c` which contains start points of intervals in `A` and endpoints of intervals in `B` where the interval in `A` ends before the interval in `B` begins.
• In the above `c`'s come from the underlying schema independent universe, say intervals in Shakespeare.
• In addition to these binary operators, we have two projection operators `pi_1(A)` and `pi_2(A)` which given a concordance list `A` return `{[u,u] | exists v mbox( with ) [u,v] in A}` and `{[v,v] | exists u mbox( with ) [u,v] in A}`
• Lastly, we define `[i] = {[u, v] | v -u +1 = i}`, the GC-lists that includes all intervals of a fixed size.

Examples

• `pi_1({` `[5, 9], [8, 12], [15,20]}`) = `{ [5,5]` `[8,8], [15,15]}`
• [10] = `{ ...,` `[101, 110], [102, 111], ...` `}`.
• Lines spoken by any witch, might be expressed as:
  ( "<LINE>" ... "</LINE>") <| (( "<SPEECH>" ... "</SPEECH>")
  |> (( "<SPEAKER>" ... "</SPEAKER>") |> "witch"))

Implementation

• The implementation of our region algebra generalizes the algorithms for phrase search, proximity ranking and Boolean Queries.
• Consider the problem of evaluating `A ... B`. An interval from the resultant GC-list starts with an interval from `A` and ends with an interval from `B`.
• If `[u, v]` is the first interval from `A` and `[u', v']` is the first interval from `B` with `u' > v`, then `v'` must be the end of the end of the first interval of `A ...B`.
• We index into `B` to find the first interval with `u' > v`
• The last interval from `A` that ends before `u'` starts the first interval of `A...B`. We index into `A` to find the last interval `[u'', v'']` where `v'' < u'`. We keep going in this fashion.
• This suggests defining operators similar to the docLeft and docRight operations we had for Boolean queries.

More Implementation

trec_eval

• We've mentioned trec_eval a couple of times in class.
• trec_eval is used to make it easier to compute precision, recall, and MAP scores and is descended from the SMART system developed at Cornell in the 1960s.
• For HW3, you will use trec_eval to compare cosine rank versus BM25.
• I would like to spend a minute now to demo it.
• Recall trec_eval is software available as NIST.
• It is a C program which can be compiled from the command-line using the standard:

  make
  sudo make install
  

• This should compile the program and put the binary in /usr/local/bin
• If you want to change the location, you need to edit the Makefile, changing the BIN variable.
• For Windows, you either need to install Cygwin or edit the Makefile so that it works with nmake and cl (assuming you have Visual Studio or VS Express).

trec_eval command-line arguments

• trec_eval is run from the command-line with a command like: trec_eval [-h] [-q] [-m] trec_rel_file trec_top_file
• The -h option is used to print help.
• The -q option says whether to give statistics on each query. Without it only aggregate statistics for the whole file are generated.
• The -m option will cause trec_eval to generate statistics for the supplied measure as opposed to the official measures used in TREC. For example, -m all_qrels generates any statistic that trec_eval is capable of; -m official generates just the official ones.
• The trec_rel_file is a file consisting of a sequence of lines giving judgements on the relevance of documents (qrels).
• The trec_top_file consists of the results from your program -- in an appropriate format.

trec_rel_file and trec_top_file

• The trec_rel_file should consist of a sequence of lines in the format:

  qid  iter  docno  rel
  

  The columns are space-delimited. qid is a query integer, and the lines are assumed to be sorted in this column, iter is a iteration number which we won't use (set to 0), docno is a document id string, and rel is 1 or 0 and means the docno was a relevant result for the query.
• Recall qid would correspond to a num of a topic in a topics file that would be given by TREC for a competition and the title of the given topic would give the actual query.
• An example line might look like:

  251 0 AP880215-0135 0
  

• The trec_top_file should consist of a sequence of lines in the format:

  query_id iter docno rank sim run_id
  

  Columns are again space delimited. query_id is again the query this result is for, iter we again set to 0, docno is the doc id string of a document returned, rank is its rank 0-1000 -- require but ignored, sim is a float similarity score -- this is not ignored and should be higher for more relevant results, run_id is a string to be printed out along with the results.
• An example line might look like:

  351   0  FR940104-0-00001  1   102.38   my-test
  

Example trec_rel_file

351 0 FR940104-0-00001 0
351 0 FR940104-0-00002 1
351 0 FR940104-0-00003 1
351 0 FR940104-0-00004 1
351 0 FR940104-0-00005 1
351 0 FR940104-0-00006 1
351 0 FR940104-0-00007 1
351 0 FR940104-0-00008 1
351 0 FR940104-0-00009 1
351 0 FR940104-0-00010 1
351 0 FR940104-0-00011 0
351 0 FR940104-0-00012 1

Example trec_top_file

351   0  FR940104-0-00001  1   102.38   run-nam2
351   0  FR940104-0-00002  1   101.38   run-nam2
351   0  FR940104-0-00003  1   91.38   run-nam2
351   0  FR940104-0-00004  1   81.38   run-nam2
351   0  FR940104-0-00005  1   71.38   run-nam2
351   0  FR940104-0-00006  1   61.38   run-nam2
351   0  FR940104-0-00007  1   51.38   run-nam2
351   0  FR940104-0-00008  1   41.38   run-nam2
351   0  FR940104-0-00009  1   31.38   run-nam2
351   0  FR940104-0-00010  1   22.38   run-nam2
351   0  FR940104-0-00011  1   21.38   run-nam2
351   0  FR940104-0-00012  1   11.38   run-nam2

Example trec_eval Output

runid                 	all	run-nam2
num_q                 	all	1
num_ret               	all	12
num_rel               	all	10
num_rel_ret           	all	10
map                   	all	0.7904
gm_map                	all	0.7904
Rprec                 	all	0.9000
bpref                 	all	0.4500
recip_rank            	all	0.5000
iprec_at_recall_0.00  	all	0.9000
iprec_at_recall_0.10  	all	0.9000
iprec_at_recall_0.20  	all	0.9000
iprec_at_recall_0.30  	all	0.9000
iprec_at_recall_0.40  	all	0.9000
iprec_at_recall_0.50  	all	0.9000
iprec_at_recall_0.60  	all	0.9000
iprec_at_recall_0.70  	all	0.9000
iprec_at_recall_0.80  	all	0.9000
iprec_at_recall_0.90  	all	0.9000
iprec_at_recall_1.00  	all	0.8333
P_5                   	all	0.8000
P_10                  	all	0.9000
P_15                  	all	0.6667
P_20                  	all	0.5000
P_30                  	all	0.3333
P_100                 	all	0.1000
P_200                 	all	0.0500
P_500                 	all	0.0200
P_1000                	all	0.0100